system

Abstract

A system is provided.SOLUTION: A system comprising: terminal means having a display device worn by a user; recognition means for recognizing and analyzing a voice of the user; server means for selecting and providing learning content; VR scenario generation means for displaying the learning content provided from the server means via display means and enabling the user to perform a dialogue based on the voice; and feedback means for providing feedback to the user based on a result of the recognition means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When it comes to foreign language learning and studying abroad, time and financial constraints, as well as a lack of opportunities to learn native pronunciation, are major challenges. These challenges are particularly pronounced for working adults who find it difficult to take long vacations, and for students who have difficulty raising the funds to study abroad. Furthermore, traditional language learning methods lack the opportunity for real-life conversation experience and native pronunciation practice, resulting in a lack of practical skills. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a system including a terminal means having a display device worn by a user, a recognition means for recognizing and analyzing the user's voice, a server means for selecting and providing learning content, a VR scenario generation means for displaying the learning content provided by the server means on a display means and allowing the user to engage in voice-based dialogue, and a feedback means for providing feedback to the user based on the results of the recognition means. This system allows users to have a practical language experience in a VR space, providing an opportunity to learn native pronunciation while eliminating the problems of cost and time.

[0006] "User" refers to an individual who operates and utilizes the System.

[0007] A "display device" is a device for displaying visual information, and is typically worn by a user.

[0008] The term "terminal means" refers to an electronic device that is directly operated by a user, and includes a display device.

[0009] "A recognition means for recognizing and analyzing speech" refers to a device or software that has the function of capturing a user's speech, analyzing it, and making a linguistic evaluation.

[0010] "Server means" refers to a computer system that works in conjunction with terminal means via a network to select and provide learning content.

[0011] "Learning content" refers to the learning materials and scenarios that allow a user to learn a language.

[0012] "Display means" refers to a device for visually displaying information provided by the server means.

[0013] "VR scenario generation means" refers to a device or software that has the function of generating specific situations or scenarios within a virtual reality space and allowing users to interactively experience them.

[0014] "Feedback means" refers to a device or software that has the function of providing the user with the analysis results of the recognition means and instruction content visually or audibly. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system according to the present invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the sensation of actually studying abroad. Specific embodiments for carrying out the present invention are described below.

[0037] System configuration overview

[0038] User:

[0039] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0040] Device:

[0041] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server.

[0042] server:

[0043] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation.

[0044] Explanation of program processing procedures

[0045] First-time setup

[0046] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0047] User: Enter your account information and submit.

[0048] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[0049] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0050] User: Record a sample voice and send it.

[0051] Server: Receives audio samples and performs initial pronunciation assessment.

[0052] Pronunciation evaluation and content provision

[0053] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0054] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0055] Experience the learning scenario

[0056] User: Selects a learning scenario (e.g., shopping).

[0057] Terminal: Generates and displays a VR space according to the selected scenario. A virtual character appears and speaks to the user.

[0058] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0059] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0060] Terminal: Receives feedback and provides it to the user visually or audibly.

[0061] Specific examples

[0062] Example: Shopping Scenario

[0063] 1. User: Select the "Shopping" scenario and enter the VR space.

[0064] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0065] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0066] 4. User: Says "Excuse me, I'm looking for an apple."

[0067] 5. Terminal: Records the user's voice and sends it to the server.

[0068] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0069] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[0070] 8. User: Follow the instructions and continue the experience.

[0071] By using the system of the present invention, users can efficiently learn a language through a realistic study abroad experience at home. Furthermore, repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[0072] The processing flow will be explained below.

[0073] Step 1:

[0074] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[0075] Step 2:

[0076] Device: Prompt the user for account information on the initial setup screen.

[0077] Step 3:

[0078] User: Enter your account information and submit.

[0079] Step 4:

[0080] Server: Receives and authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.).

[0081] Step 5:

[0082] On the device: The initial microphone setup screen for speech recognition is displayed, prompting the user to record a sample voice.

[0083] Step 6:

[0084] User: Record a sample voice and send it.

[0085] Step 7:

[0086] Server: Receives audio samples and performs initial pronunciation assessment.

[0087] Step 8:

[0088] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0089] Step 9:

[0090] Device: Starts downloading the learning content provided by the server.

[0091] Step 10:

[0092] Device: Once the content has finished downloading, it will be displayed in a list and the user will be prompted to make a selection.

[0093] Step 11:

[0094] User: Select a learning scenario (e.g., shopping).

[0095] Step 12:

[0096] Device: Generates and displays a VR space according to the selected scenario.

[0097] Step 13:

[0098] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[0099] Step 14:

[0100] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[0101] Step 15:

[0102] User: Says "Excuse me, I'm looking for an apple."

[0103] Step 16:

[0104] Device: Records the user's voice and sends it to the server.

[0105] Step 17:

[0106] Server: Analyzes the received voice data and evaluates the pronunciation.

[0107] Step 18:

[0108] Server: If the pronunciation of "ringo" is correct, generate feedback as "Excellent."

[0109] Step 19:

[0110] Terminal: A store clerk character will say, "It's here," and at the same time display the evaluation results.

[0111] Step 20:

[0112] User: Follow the instructions and continue the experience.

[0113] Step 21:

[0114] Server: Records learning progress data and prepares suggestions for the next learning session.

[0115] Step 22:

[0116] On your device: Show the user suggestions for their next study session.

[0117] Step 23:

[0118] User: Check out upcoming sessions and decide whether to book or start immediately.

[0119] Step 24:

[0120] Terminal: Prepares for the next session and notifies the user.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In conventional language learning systems, users have few opportunities to practice conversations in a real language environment, which means that their pronunciation and communication skills are not fully improved. The present invention aims to solve this problem by providing a system that allows users to efficiently learn a language through a realistic study abroad experience from the comfort of their own home.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes terminal means having a display device worn by the user, recognition means for recognizing and analyzing the user's voice, server means for selecting and providing learning content, VR scenario generation means for displaying the learning content provided by the server means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback to the user based on the results of the recognition means, means for recording the user's pronunciation as voice data and transmitting it to the server, and means for selecting and downloading appropriate learning content based on the analysis results from the server. This allows the user to practice pronunciation while experiencing realistic dialogue scenes, thereby achieving effective language learning.

[0126] "User" refers to a person who actually uses the system and wears a VR device to learn a language.

[0127] The term "terminal means" refers to an electronic device including a display device worn by a user, which exchanges information between the user and the server.

[0128] "Recognition means" refers to a device or software that has the function of recognizing and analyzing the user's voice and has the role of evaluating the voice data.

[0129] "Server means" refers to a computer system that has the function of selecting and providing learning content and analyzing user voice data.

[0130] "VR scenario generation means" refers to a device or software that generates a virtual reality scenario based on learning content provided by the server means and enables the user to interact in that environment.

[0131] The "feedback means" refers to a device or software that has the function of providing the user with visual or audio feedback on the quality of their pronunciation based on the results of the recognition means.

[0132] "Means for recording as voice data" refers to a device or software that has the function of recording the user's pronunciation and transmitting the voice data to a server.

[0133] "Means for downloading" refers to a device or software that has the function of downloading learning content provided by a server and displaying it to the user.

[0134] The system related to this invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the experience of actually studying abroad.

[0135] System configuration overview

[0136] User:

[0137] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0138] Device:

[0139] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server. For example, a display device such as Oculus Quest 2 is used.

[0140] server:

[0141] This is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Google Cloud Speech-to-Text API and other services are used for speech analysis.

[0142] Program Description

[0143] The user wearing the VR device launches a dedicated application and enters their account information to begin the initial setup. The server authenticates the user's account information and instructs them to configure the microphone used for voice recognition and record a sample voice. When the user records and sends the sample voice, the server analyzes the voice, performs an initial pronunciation evaluation, and provides learning content.

[0144] When a user selects a learning scenario, the device generates a corresponding VR space in which a virtual character speaks to the user. The user interacts with the character and practices pronunciation. After practice, the audio data is sent to the server for analysis and feedback. Feedback is provided to the user visually or audibly via the device.

[0145] Specific examples

[0146] Example of a shopping scenario

[0147] 1. User: Select the "Shopping" scenario and enter the VR space.

[0148] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0149] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0150] 4. User: Says "Excuse me, I'm looking for an apple."

[0151] 5. Terminal: Records the user's voice and sends it to the server.

[0152] 6. Server: Analyzes the audio and if the pronunciation of "apple" is correct, rates it as "Excellent."

[0153] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[0154] 8. User: Follow the instructions and continue the experience.

[0155] This system allows users to study a language efficiently while experiencing a realistic study abroad experience from home. Repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[0156] Prompt Sentence Examples

[0157] "Users experiencing virtual study abroad interact with a virtual store clerk in a shopping scenario and receive pronunciation evaluation and feedback."

[0158] The user puts on the VR device and selects the "Shopping" scenario. The device generates a VR space inside the store, and a store clerk character appears and asks, "What are you looking for?" The user says, "Excuse me, I'm looking for an apple," and the device records the speech and sends it to the server. The server analyzes the speech, and if the pronunciation is correct, it rates it as "Excellent," and the device displays the result. The user then continues the next dialogue.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] The user puts on the VR device and launches the dedicated application.

[0162] Input: The user puts on the VR device and launches the application.

[0163] Output: The application starts and the initial setup screen is displayed.

[0164] Specific operation: The user wears a head-mounted display (e.g., Oculus Quest 2) and uses the controller to select and launch the dedicated application. When the application is launched, the device displays the initial setup screen.

[0165] Step 2:

[0166] The user enters and submits their account information.

[0167] Input: Enter your account information in the text boxes displayed on the initial setup screen.

[0168] Output: The entered account information is sent to the server.

[0169] Specific operation: The user enters account information such as email address, password, and username in the text boxes provided and presses the "Send" button. The device then sends this information to the server.

[0170] Step 3:

[0171] The server authenticates the account information and registers the initial settings.

[0172] Input: Account information sent from the device

[0173] Output: Authentication result and initial setting registration completion message

[0174] Specific operation: The server checks the received account information against the database and performs authentication. If authentication is successful, it registers initial information such as the user's learning goals and language level, and returns the results to the terminal.

[0175] Step 4:

[0176] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[0177] Input: Authentication result and initial setting information returned from the server

[0178] Output: Sample audio recording screen display

[0179] Specific operation: After receiving the authentication result, the device displays a screen prompting the user to set up voice recognition, asking them to say a sample voice such as "Hello."

[0180] Step 5:

[0181] User records and submits a sample voice

[0182] Input: A sample voice utterance from the user

[0183] Output: Recorded audio data is sent to the server

[0184] Specific operation: The user speaks a sample voice according to the instructions on the device. The device records the voice and sends it to the server as voice data.

[0185] Step 6:

[0186] The server analyzes the sample audio and makes an initial pronunciation assessment

[0187] Input: Sample audio data sent from the device

[0188] Output: Initial pronunciation assessment result and a message that the learning content is ready to be provided

[0189] Specific operation: The server uses a speech analysis algorithm such as the Google Cloud Speech-to-Text API to analyze the sample speech and evaluate the user's pronunciation level. Based on the evaluation results, it prepares appropriate learning content and sends a message to the device indicating that it is ready to be provided.

[0190] Step 7:

[0191] The device downloads the learning content and displays it in a list.

[0192] Input: Message from the server that learning content is ready to be provided

[0193] Output: Download and list of learning content

[0194] Specific operation: The device receives instructions from the server, downloads the suggested learning content, displays a list of downloaded learning content, and prompts the user to select one.

[0195] Step 8:

[0196] The user selects a learning scenario

[0197] Input: User selects from a list of learning content

[0198] Output: The selected learning scenario is displayed on the device.

[0199] Specific operation: The user selects a learning scenario, such as "Shopping," from the list of learning content displayed on the device and presses the "Start" button. The selected scenario is reflected on the device.

[0200] Step 9:

[0201] The device generates and displays the VR space.

[0202] Input: Information about the learning scenario selected by the user

[0203] Output: Display of VR space and virtual characters based on the selected scenario

[0204] Specific operation: The device generates and displays an appropriate VR space (e.g., a shopping mall) based on the information of the selected scenario, and a virtual character (e.g., a store clerk) also appears.

[0205] Step 10:

[0206] Users interact with virtual characters and practice their pronunciation

[0207] Input: Question from virtual character and user's response utterance

[0208] Output: Recorded user voice data

[0209] Specific operation: The virtual character asks a question such as "What are you looking for?", and the user responds with a voice such as "Excuse me, I'm looking for an apple." The device records this voice.

[0210] Step 11:

[0211] The device sends the recorded audio data to the server.

[0212] Input: User pronunciation recording

[0213] Output: Recorded audio data is sent to the server

[0214] Specific operation: The device compresses the recorded user voice data and sends it to a server via the Internet.

[0215] Step 12:

[0216] The server analyzes the audio data and generates pronunciation evaluation and feedback.

[0217] Input: Audio data sent from the device

[0218] Output: Pronunciation evaluation results and feedback information

[0219] Specific operation: The server analyzes the audio data using a speech analysis algorithm such as the Google Cloud Speech-to-Text API, evaluates the accuracy of the pronunciation, and generates feedback such as "Excellent" based on the evaluation result and sends it to the device.

[0220] Step 13:

[0221] The device displays and provides feedback to the user

[0222] Input: Feedback information sent from the server

[0223] Output: Visual and audio feedback

[0224] Specific operation: Based on the received feedback, the device displays or speaks an evaluation result such as "Excellent" to the user. The user confirms this feedback and continues with the next learning scenario.

[0225] (Application example 1)

[0226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] Conventional language learning systems have limited opportunities for users to experience language usage in everyday life, making it difficult to improve practical language skills. Furthermore, feedback is limited, making it difficult for users to self-correct. In particular, pronunciation evaluation and correction are not performed in real time, making efficient learning difficult.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0229] In this invention, the server includes a means for selecting and providing learning content, a means for evaluating a user's pronunciation, and a means for providing feedback. This allows users to learn a language in a virtual space where they select a shopping scenario and engage in interactive learning, creating a situation similar to real life. Furthermore, real-time pronunciation evaluation and feedback allow users to efficiently self-correct and improve their pronunciation.

[0230] The "terminal means" is an electronic device including a display device worn by a user, and has the function of exchanging information between the user and the server.

[0231] The "recognition means" has the function of recognizing and analyzing the user's voice.

[0232] The "server means" is a computer system that has the function of selecting and providing learning content.

[0233] The "VR scenario generation means" has the function of displaying the learning content provided by the server means via the display means and generating a virtual reality environment in which the user can engage in voice-based dialogue.

[0234] The "feedback means" has a function of providing the user with pronunciation evaluation results and correction instructions based on the results of the recognition means.

[0235] A "shopping scenario" is a scenario that provides a virtual shopping experience that a user can choose from for language learning.

[0236] "Interactive learning" is a method in which a user progresses through learning by interacting with a virtual character.

[0237] A "virtual space" is a virtual environment that users can experience through a VR device.

[0238] This system, which is comprised of three elements: a user, a terminal, and a server, provides an environment in which users can study a language while experiencing the sensation of actually studying abroad using virtual reality (VR). This system allows users to study a language in a virtual space where they can select a shopping scenario and engage in interactive learning, creating a situation that is close to real life.

[0239] System configuration overview

[0240] 1. User: The final user of the system, who wears the VR device and experiences the language learning content.

[0241] 2. Terminal: An electronic device including a VR device worn by the user that exchanges information between the user and the server. The terminal has the function of recording the user's pronunciation and sending it to the server.

[0242] 3. Server: A computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Specifically, it performs speech recognition and evaluation using the Python requests and speech_recognition libraries. Feedback is provided using the pyttsx3 library.

[0243] Processing flow

[0244] First-time setup

[0245] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0246] User: Enter your account information and submit.

[0247] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[0248] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0249] User: Record a sample voice and send it.

[0250] Server: Receives audio samples and performs initial pronunciation assessment.

[0251] Pronunciation evaluation and content provision

[0252] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0253] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0254] Experience the learning scenario

[0255] User: Select a learning scenario (e.g., shopping).

[0256] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[0257] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0258] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0259] Terminal: Receives feedback and provides it to the user visually or audibly.

[0260] Specific examples

[0261] Example: Shopping Scenario

[0262] 1. User: Select the "Shopping" scenario and enter the VR space.

[0263] 2. Terminal: Generates a virtual space inside a shopping mall or store, and virtual characters (store clerks) appear.

[0264] 3. Terminal: A virtual character asks, "What are you looking for?"

[0265] 4. User: Says "Excuse me, I'm looking for an apple."

[0266] 5. Terminal: Records the user's voice and sends it to the server.

[0267] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0268] 7. Terminal: A virtual character will say, "It's here," and simultaneously display the evaluation results.

[0269] 8. User: Follow the instructions and continue the experience.

[0270] Prompt Sentence Examples

[0271] If the user says "Excuse me, I'm looking for an apple," the prompt is:

[0272] text

[0273] Evaluate the pronunciation of "Excuse me, I'm looking for an apple" in Japanese and generate an appropriate feedback message.

[0274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0275] Step 1:

[0276] First-time setup

[0277] Device: Put on the VR device and launch the dedicated application. The initial setup screen will prompt the user to enter their account information.

[0278] User: Enter your account information (e.g., username, email address, etc.) and submit.

[0279] Terminal input: Account information. Output: Sending account information to the server.

[0280] Server: Receives the entered account information and verifies it against the database. After authentication, it registers the user's initial settings (learning purpose, language level, etc.), configures the microphone used for voice recognition, and asks for a sample voice recording.

[0281] Server input: Account information. Output: Authentication result and instructions to register initial setting information.

[0282] Step 2:

[0283] Voice Recognition Settings

[0284] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0285] User: Record a sample voice and send it.

[0286] Input: Microphone settings, sample audio. Output: Send sample audio data to the server.

[0287] Server: Analyzes the received sample voice data, performs an initial pronunciation evaluation, and stores the evaluation results in a database.

[0288] Server input: Sample speech data. Output: Initial pronunciation evaluation results.

[0289] Step 3:

[0290] Selection and provision of learning content

[0291] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0292] Server input: User's speech data. Output: Pronunciation evaluation results and appropriate learning content.

[0293] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0294] Terminal input: Instruction to download learning content. Output: List of learning content.

[0295] Step 4:

[0296] Experience the learning scenario

[0297] User: Select a learning scenario (e.g., shopping).

[0298] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[0299] Terminal input: Selection of learning scenario. Output: Generation and display of virtual space.

[0300] Step 5:

[0301] Pronunciation practice and feedback

[0302] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0303] Input on the device: User's pronunciation. Output: Sending voice data to the server.

[0304] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0305] Server input: User's speech data. Output: Pronunciation evaluation and feedback.

[0306] Terminal: Receives feedback and provides it to the user visually or audibly.

[0307] Terminal input: Feedback data. Output: Display feedback or audio playback.

[0308] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0309] The system according to the present invention is composed of four main elements: a user, a terminal, a server, and an emotion engine. This system uses virtual reality (VR) to provide a language learning environment in which users can experience studying abroad in a virtual world. Furthermore, by recognizing the user's emotions and adapting the learning content and feedback, a more effective learning experience can be achieved. Specific embodiments for implementing the present invention are described below.

[0310] System configuration overview

[0311] User:

[0312] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0313] Device:

[0314] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and emotional data and sending it to the server.

[0315] server:

[0316] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation and emotions.

[0317] Emotion Engine:

[0318] It has an algorithm that recognizes the user's emotions, analyzes the user's voice and facial expression data, and determines their emotional state. It also has the ability to adapt learning content and feedback according to the user's emotional state.

[0319] Explanation of program processing procedures

[0320] First-time setup

[0321] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0322] User: Enter your account information and submit.

[0323] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and prompts the user to record a sample voice.

[0324] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0325] User: Record a sample voice and send it.

[0326] Server: Receives audio samples and performs initial pronunciation assessment.

[0327] Pronunciation evaluation and content provision

[0328] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0329] Device: Starts downloading the learning content provided by the server.

[0330] On your device: Once the downloads are complete, they will be displayed in a list and you will be prompted to make a selection.

[0331] Experience the learning scenario

[0332] User: Selects a learning scenario (e.g., shopping).

[0333] Device: Generates and displays a VR space according to the selected scenario.

[0334] Terminal: A virtual character appears and speaks to the user.

[0335] Emotion engine: Analyzes the user's voice and facial expression data in real time to determine the user's emotional state.

[0336] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[0337] User: Says "Excuse me, I'm looking for an apple."

[0338] Terminal: Records the user's voice and facial expression data and sends them to the server.

[0339] Server: Analyzes the received voice data and evaluates the pronunciation.

[0340] Emotion engine: Analyzes the received emotion data and evaluates the user's emotional state.

[0341] Server: Generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[0342] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The feedback will also be adjusted according to the user's emotional state.

[0343] Specific examples

[0344] Example: Shopping Scenario

[0345] 1. User: Select the "Shopping" scenario and enter the VR space.

[0346] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0347] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0348] 4. User: Says "Excuse me, I'm looking for an apple."

[0349] 5. Terminal: Records the user's voice and facial expression data and sends them to the server.

[0350] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0351] 7. Emotion Engine: Analyzes the user's facial expression data and evaluates their emotional state. For example, if the user is tired, it will provide gentle feedback.

[0352] 8. Terminal: A store clerk character will say "It's here" and simultaneously display the evaluation result. The feedback will also be adjusted according to the customer's emotional state.

[0353] 9. User: Follow the instructions and continue the experience.

[0354] By using the system of the present invention, users can efficiently learn a language while enjoying a realistic study abroad experience from the comfort of their own home. Furthermore, the emotion engine makes the learning experience more personalized, improving user motivation and learning effectiveness.

[0355] The processing flow will be explained below.

[0356] Step 1:

[0357] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[0358] Step 2:

[0359] Device: Prompt the user for account information on the initial setup screen.

[0360] Step 3:

[0361] User: Enter your account information and submit.

[0362] Step 4:

[0363] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and instructs the user to record a sample voice.

[0364] Step 5:

[0365] Device: Prompt the user to record a sample audio on the initial setup screen.

[0366] Step 6:

[0367] User: Follow the instructions to record and submit a sample audio.

[0368] Step 7:

[0369] Server: Receives audio samples and performs initial pronunciation assessment.

[0370] Step 8:

[0371] Server: Analyzes the received voice data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0372] Step 9:

[0373] Device: Starts downloading the learning content provided by the server.

[0374] Step 10:

[0375] On the device: Once the download is complete, the contents will be displayed in a list and the user will be prompted to make a selection.

[0376] Step 11:

[0377] User: Select a learning scenario (e.g., shopping).

[0378] Step 12:

[0379] Device: Generates and displays a VR space according to the selected scenario.

[0380] Step 13:

[0381] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[0382] Step 14:

[0383] Terminal: The character asks "What are you looking for?" and waits for voice input.

[0384] Step 15:

[0385] User: Says "Excuse me, I'm looking for an apple."

[0386] Step 16:

[0387] Terminal: Records the user's voice and facial expression data and sends them to the server and emotion engine.

[0388] Step 17:

[0389] Server: Analyzes the received voice data and evaluates the pronunciation.

[0390] Step 18:

[0391] Emotion engine: Analyzes received facial expression data and evaluates the user's emotional state.

[0392] Step 19:

[0393] Server: Generates feedback based on pronunciation evaluation results and emotional state.

[0394] Step 20:

[0395] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The system also adjusts the feedback content according to the user's emotional state. For example, if the user is nervous, it will provide a comment to help them relax.

[0396] Step 21:

[0397] User: Follow the instructions and continue the experience.

[0398] Step 22:

[0399] Server: Records learning progress and prepares suggestions for the next lesson.

[0400] Step 23:

[0401] On your device: Display suggestions for your next study session and prompt the user to choose their next step.

[0402] Step 24:

[0403] User: Review the proposal and decide to book your next session or start immediately.

[0404] Step 25:

[0405] Terminal: Prepares for the next session and notifies the user.

[0406] This series of processes allows users to study language efficiently in a relaxed state. The emotional engine individually optimizes the user's learning experience, which is expected to improve motivation and learning effectiveness.

[0407] Example 2

[0408] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Existing language learning systems provide uniform pronunciation evaluation and feedback to users, which can lead to insufficient learning outcomes for individual users. Furthermore, feedback and content adaptation do not take into account the user's emotional state, which can lead to a decrease in motivation and learning effectiveness. Furthermore, there is a lack of effective ways to provide real-life dialogue scenarios in VR environments, making it difficult to achieve a realistic learning experience.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0411] In this invention, the server includes a means for selecting and providing learning content, an emotion evaluation means for evaluating the user's emotional state, and an adaptation means for adapting the learning content and feedback. This allows for personalized responses that take into account the user's individual pronunciation evaluation and emotional state, improving learning effectiveness and motivation. Furthermore, realistic dialogue scenarios can be provided in a VR environment, providing a more realistic language learning experience.

[0412] The "terminal means" is a device that has a display device worn by a user and exchanges information with the server and other system elements.

[0413] A "recognition means" is a device or software that has the function of recognizing and analyzing a user's voice.

[0414] The "server means" is a computer system that selects and provides learning content and manages data in cooperation with the recognition means and other system elements.

[0415] The "VR scenario generation means" is a device or software that displays the learning content provided by the server means via the display means and generates a virtual reality scenario in which the user can engage in voice-based dialogue.

[0416] The "emotion evaluation means" is a device or software that has the function of evaluating the user's emotional state based on the results of the recognition means.

[0417] The "adaptation means" is a device or software that has the function of adapting the learning content or feedback based on the evaluation results of the emotion evaluation means.

[0418] The "feedback means" is a device or software that has a function of providing feedback to the user based on the results of the recognition means and emotion evaluation means.

[0419] The system according to the present invention is a system comprising a terminal means having a display device worn by the user, a recognition means for recognizing and analyzing the user's voice, a server means for selecting and providing learning content, a VR scenario generation means for generating a virtual reality (VR) scenario, an emotion evaluation means for evaluating the user's emotional state, an adaptation means for adapting the learning content and feedback, and a means for providing feedback.

[0420] Hardware and software used

[0421] 1. Terminal means: The display device worn by the user is a VR device such as a head-mounted display (HMD). This device is equipped with a high-performance graphics card and sensors to provide an interactive VR environment. Specifically, HMDs include "Oculus Rift" and "HTC Vive."

[0422] 2. Recognition method: Software for recognizing and analyzing the user's voice can use the voice recognition libraries "Google Speech-to-Text API" and "Microsoft Azure Speech Service."

[0423] 3. Server: The server is responsible for selecting and providing the learning content. The server is equipped with a high-performance processor and storage, and runs a database and network management system. For example, a cloud server from Amazon Web Services (AWS) can be used.

[0424] 4. VR scenario generation means: To generate a VR scenario that displays the learning content provided by the server and allows the user to interact based on voice, the game engines "Unity" and "Unreal Engine" can be used.

[0425] 5. Emotion assessment method: To assess the user's emotional state, we use emotion recognition software "Affectiva SDK" and "Microsoft Azure Emotion API" that analyze the user's voice and facial expression data.

[0426] 6. Adaptation: Software that adapts learning and feedback in real time uses algorithms running on servers that utilize machine learning and natural language processing (NLP) techniques.

[0427] 7. Feedback methods: Feedback can be provided using Text-to-Speech (TTS) systems or display pop-up technology to present feedback to the user visually or audibly.

[0428] Specific examples

[0429] Example scenario: Shopping experience

[0430] 1. User: Put on the VR device, launch the application, set up an account for the first time, and record a sample voice.

[0431] 2. Device: Once the initial setup is complete, a list of learning content provided by the server will be displayed.

[0432] 3. User: Select the "Shopping" scenario and enter the VR space.

[0433] 4. Terminal: A VR space of a shopping mall is generated, and virtual characters (store clerks) appear.

[0434] 5. Terminal: The store clerk character asks, "Welcome, what are you looking for?"

[0435] 6. User: Says, "Excuse me, I'm looking for an apple."

[0436] 7. Terminal: Records voice and facial expression data and sends it to the server.

[0437] 8. Server: Analyzes the audio and evaluates the pronunciation.

[0438] 9. Emotion Engine: Analyzes facial expression data and evaluates emotional state, for example, recognizing that the user is stressed.

[0439] 10. Server: Generates feedback based on the evaluation results and emotional state and sends it to the device.

[0440] 11. Terminal: The virtual character will say "It's here" and display the evaluation result. The evaluation result will be "Excellent," and depending on the user's emotional state, a positive message such as "Your pronunciation is really good!" will be displayed.

[0441] Prompt Sentence Examples

[0442] In a shopping scenario, describe the process flow for when a user asks a store clerk if they are looking for an apple.

[0443] This system allows users to enjoy a highly realistic learning experience and learn languages ​​more effectively through individually optimized feedback and learning content. Personalized feedback based on the user's emotional state also increases learning motivation.

[0444] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0445] Step 1:

[0446] The user puts on the VR device and launches the dedicated application.

[0447] Input: The user powers on the VR device and selects the application icon.

[0448] Specific operation: The user wears the HMD device on their head and launches an application using the controller in their hand.

[0449] Output: The application's initial setup screen is displayed, prompting the user to enter their account information.

[0450] Step 2:

[0451] The device displays the initial setup screen and prompts the user to enter their account information.

[0452] Input: The application launches and the initial setup screen appears, presenting the user with input fields such as "Name," "Email address," and "Password."

[0453] Specific behavior: The initial setup form will be displayed in the VR space.

[0454] Output: A screen is prepared for the user to enter their account information.

[0455] Step 3:

[0456] The user enters and submits their account information.

[0457] Input: The user enters the required information into the designated fields using a keyboard or voice input.

[0458] Specific behavior: The user clicks the "Submit" button.

[0459] Output: The entered account information is sent to the server.

[0460] Step 4:

[0461] The server receives the account information and performs authentication.

[0462] Input: Account information submitted by the user.

[0463] Specific operation: The server connects to the database and verifies the input information to perform authentication.

[0464] Output: The authentication result is generated (success or failure). If authentication is successful, the user's initial setup is completed.

[0465] Step 5:

[0466] The server displays a screen to guide the user through the speech recognition test and asks the user to record a sample voice.

[0467] Input: Account authentication successful, user's initial setup data.

[0468] Specific operation: The server sends a voice recognition setting screen to the device and instructs it to "record a sample voice."

[0469] Output: The voice recognition settings screen will be displayed on the device.

[0470] Step 6:

[0471] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[0472] Input: Speech recognition setting screen sent from the server.

[0473] Specific operation: Audio recording instructions will be displayed in the VR space.

[0474] Output: The user is ready to start recording.

[0475] Step 7:

[0476] User records and submits a sample voice

[0477] Input: The user reads the prompted sentence (e.g., "I want to learn Japanese").

[0478] Specific behavior: The user clicks the record button and speaks the specified sentence.

[0479] Output: The recorded audio data is stored on the device and sent to the server.

[0480] Step 8:

[0481] The server receives the sample audio and performs an initial pronunciation assessment

[0482] Input: Audio data sent by the user.

[0483] What it does: It uses a speech analysis algorithm to analyze the speech data and evaluate the accuracy of the pronunciation.

[0484] Output: The initial pronunciation evaluation result is generated.

[0485] Step 9:

[0486] The server analyzes the voice data and evaluates the user's pronunciation level.

[0487] Input: User speech data and an existing pronunciation reference model.

[0488] What it does: Uses machine learning algorithms to compare and evaluate your voice data.

[0489] Output: The user's pronunciation level evaluation result is generated.

[0490] Step 10:

[0491] The server selects appropriate learning content based on the evaluation results.

[0492] Input: Pronunciation level assessment results and user learning objectives.

[0493] Specific actions: Select appropriate learning materials from a database of learning content.

[0494] Output: Selected learning content data is generated.

[0495] Step 11:

[0496] The server sends the selected learning content to the device.

[0497] Input: Selected learning content data.

[0498] Specific operation: Divide the learning content data into packets and send them to the terminal.

[0499] Output: The device receives the learning content data.

[0500] Step 12:

[0501] The device downloads the learning content and displays it in a list.

[0502] Input: Learning content data sent from the server.

[0503] Specific behavior: Downloads content and displays a progress bar. Once completed, displays a list of learning content.

[0504] Output: A list of learning content will be displayed on the screen.

[0505] Step 13:

[0506] The user selects a learning scenario (e.g., shopping)

[0507] Input: A list of learning content.

[0508] Specific behavior: The user selects the desired scenario from a list. Example: "Shopping".

[0509] Output: The selected learning scenario is recorded on the device.

[0510] Step 14:

[0511] The device generates and displays a VR space according to the selected scenario.

[0512] Input: Data for the scenario selected by the user.

[0513] Specific operation: Use "Unity" or "Unreal Engine" to render and display the corresponding VR space.

[0514] Output: The VR space is displayed in the user's field of view.

[0515] Step 15:

[0516] The device displays a virtual character and starts a conversation

[0517] Input: The learning scenario selected by the user.

[0518] What it does: A virtual character appears in the user's field of vision, plays a preset audio file, and asks, "What are you looking for?"

[0519] Output: A virtual character begins to interact.

[0520] Step 16:

[0521] The emotion engine analyzes the user's voice and facial expression data in real time to determine their emotional state.

[0522] Input: Real-time voice and facial expression data from the user.

[0523] Specific operations: Emotion analysis is performed using the "Affectiva SDK" and "Microsoft Azure Emotion API."

[0524] Output: The user's emotional state assessment result is generated.

[0525] Step 17:

[0526] The device records the user's voice and facial expression data and sends it to the server.

[0527] Input: Speech and facial expressions made by the user.

[0528] Specific operation: Captures voice and facial expression data and sends it to the server.

[0529] Output: The recorded data is sent to the server.

[0530] Step 18:

[0531] The server analyzes the received voice data and evaluates the pronunciation

[0532] Input: User's voice data.

[0533] What it does: Uses a speech recognition algorithm to assess the accuracy of the user's pronunciation.

[0534] Output: Pronunciation evaluation results are generated.

[0535] Step 19:

[0536] The emotion engine analyzes the received emotion data and evaluates the user's emotional state.

[0537] Input: User's facial expression data.

[0538] What it does: Uses an emotion analysis algorithm to assess emotional state.

[0539] Output: The emotion evaluation result is generated.

[0540] Step 20:

[0541] The server generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[0542] Input: Pronunciation evaluation results and emotion evaluation results.

[0543] Specific operations: Generate pronunciation feedback and responses according to the emotional state and send them to the device.

[0544] Output: Feedback data is sent to the device.

[0545] Step 21:

[0546] The device controls the virtual character and provides feedback to the user.

[0547] Input: Feedback data sent by the server.

[0548] Specific behavior: A virtual character will say "It's here" and display the evaluation result. Feedback such as "Excellent" or a positive message will be displayed.

[0549] Output: A virtual character will provide guidance and feedback.

[0550] (Application example 2)

[0551] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0552] Conventional language learning systems have struggled to effectively assess users' pronunciation and comprehension and provide a personalized learning experience. Furthermore, they were unable to provide feedback based on the user's emotional state, resulting in insufficient learning effectiveness. Furthermore, it was difficult to replicate the experience of actual in-store customer service, making it difficult to improve practical language skills.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes terminal means having a visual display device worn by the user, voice recognition means for recognizing and analyzing the user's voice, information processing device means for selecting and providing learning content, virtual space generation means for displaying the learning content provided by the information processing device means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback based on the results of the voice recognition means and the user's emotional state, and emotion analysis means for analyzing the user's emotions. This allows the user to receive personalized feedback based on the emotion analysis, enabling a practical language learning experience within a virtual store.

[0554] A "visual display device" is a device worn by a user to receive visual information and provide a virtual reality experience.

[0555] "Terminal means" refers to electronic equipment used by a user, including a visual display device, for communicating with a server to exchange information.

[0556] The "voice recognition means" is a device that has the function of recording the user's voice and analyzing that voice.

[0557] The "information processing device means" is a computer system such as a server for generating and providing learning content.

[0558] "Virtual space generating means" is a device that has the function of generating a virtual reality environment and allowing a user to interact within it.

[0559] A "feedback means" is a device that has the function of providing visual or audio feedback to a user.

[0560] "Emotion analysis means" refers to an algorithm or device that analyzes the user's voice and facial expression data and determines their emotional state.

[0561] "User" refers to the final user who uses this system to study.

[0562] The present invention provides a system that allows users to experience practical language learning in a virtual store, which includes a terminal means including a visual display device, a voice recognition means, an information processing means, a virtual space generation means, a feedback means, and an emotion analysis means.

[0563] First, the user wears a visual display device (e.g., smart glasses). The terminal generates a virtual space of a virtual store via the visual display device and displays it to the user. This virtual space generation means allows the user to experience shopping in an actual store.

[0564] The user's voice data is collected and analyzed by a voice recognition means, which may be implemented by a microphone mounted on a computer or smart glasses, and the voice commands issued by the user are collected in real time and transmitted to a server.

[0565] The server analyzes the voice data and evaluates the user's pronunciation level. At this time, an emotion analysis tool is also used based on the user's facial expressions and voice data to determine the user's emotional state. Emotion analysis uses a generative AI model incorporating facial expression recognition technology and voice emotion analysis technology.

[0566] Based on the analyzed voice data and emotion data, the information processing device generates learning content and recommendations suitable for the user. This information is transmitted to the terminal device and provided to the user through the VR space.

[0567] The feedback provided to the user is displayed visually or audibly by a feedback means, which provides personalized recommendations according to the quality of the user's pronunciation and the user's emotional state.

[0568] As a concrete example, consider the case where a user enters a virtual store in a virtual space and issues a voice command such as, "Excuse me, I'm looking for an apple." The voice is analyzed, and if the pronunciation is correct, the word "Excellent" is displayed. If the user has a relaxed expression, a virtual store clerk character praises the user's excellent pronunciation. Conversely, if the user appears tired, feedback is given in a gentler tone.

[0569] An example of a prompt sentence is, "Analyze the following user's voice data and facial expression data, and indicate the user's pronunciation evaluation and emotional state. Also, output recommendations to be provided to the user in relation to the shopping scenario. Voice data: Voice data file Facial expression data: 'smile: 0.8, anger: 0.2'."

[0570] In this way, providing real-time feedback based on the user's performance and emotional state can lead to a more effective and personalized learning experience.

[0571] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0572] Step 1:

[0573] The user puts on the visual display device and starts up the terminal.

[0574] Input: The user performs an action to start the terminal.

[0575] Output: The initial device setup screen will be displayed.

[0576] Specific operation: The terminal displays the initial screen of the virtual store via the visual display device.

[0577] Step 2:

[0578] The user enters their account information and sends it to the server.

[0579] Input: The user enters their account information and presses the "Submit" button.

[0580] Output: The server receives the account information and initializes the user profile.

[0581] Specific operation: The server performs initial settings for the user based on the received account information and sends a setup completion message to the terminal.

[0582] Step 3:

[0583] A voice sample of the user is collected and sent to a server.

[0584] Input: The user records a sample voice on the voice settings screen and presses the "Submit" button.

[0585] Output: The server receives the sample speech data and performs an initial pronunciation assessment.

[0586] How it works: The device uses a microphone to record the user's voice and sends the data to the server, which then analyzes the received voice and evaluates the pronunciation level.

[0587] Step 4:

[0588] The user's voice data and facial expression data are analyzed to evaluate their emotions.

[0589] Input: Voice data collected by speech recognition means and facial expression data collected by camera function.

[0590] Output: The server stores the user's pronunciation rating and emotion rating as analysis results.

[0591] Specific operation: The server uses a generative AI model to analyze voice data and evaluate pronunciation. At the same time, it analyzes facial expression data and evaluates the user's emotional state.

[0592] Step 5:

[0593] Generate learning content and recommendations and send them to the device.

[0594] Input: Speech and emotion data analyzed by the server.

[0595] Output: Appropriate learning content and recommendations sent to the device.

[0596] Specific operation: The server uses the information processing device means to generate learning content and feedback that is optimal for the user's learning progress and emotional state, and transmits them to the terminal.

[0597] Step 6:

[0598] The device generates a virtual space and displays the learning content.

[0599] Input: Learning content sent from the server.

[0600] Output: The learning scenario in the virtual space that is displayed to the user.

[0601] Specific operation: The terminal uses the virtual space generation means to generate a scenario for a virtual store based on the learning content received from the server, and displays it to the user via a visual display device.

[0602] Step 7:

[0603] The user interacts within the virtual space and continues the experience.

[0604] Input: An action that the user gives voice instructions for.

[0605] Output: New collected speech and facial expression data.

[0606] Specific operation: The device uses a microphone and camera to collect the user's voice and facial expression data and transmits it back to the server.

[0607] Step 8:

[0608] Provide visual or audio feedback to the user.

[0609] Input: User's voice and facial expression data reanalyzed by the server.

[0610] Output: Feedback displayed on the terminal.

[0611] Specific operation: Based on the data reanalyzed by the server, feedback that is optimal for the user's pronunciation and emotional state is generated and provided to the user via the device.

[0612] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0613] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0614] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0615] [Second embodiment]

[0616] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0617] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0618] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0619] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0620] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0621] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0622] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0623] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0624] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0625] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0626] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0627] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0628] The system according to the present invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the sensation of actually studying abroad. Specific embodiments for carrying out the present invention are described below.

[0629] System configuration overview

[0630] User:

[0631] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0632] Device:

[0633] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server.

[0634] server:

[0635] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation.

[0636] Explanation of program processing procedures

[0637] First-time setup

[0638] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0639] User: Enter your account information and submit.

[0640] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[0641] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0642] User: Record a sample voice and send it.

[0643] Server: Receives audio samples and performs initial pronunciation assessment.

[0644] Pronunciation evaluation and content provision

[0645] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0646] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0647] Experience the learning scenario

[0648] User: Selects a learning scenario (e.g., shopping).

[0649] Terminal: Generates and displays a VR space according to the selected scenario. A virtual character appears and speaks to the user.

[0650] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0651] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0652] Terminal: Receives feedback and provides it to the user visually or audibly.

[0653] Specific examples

[0654] Example: Shopping Scenario

[0655] 1. User: Select the "Shopping" scenario and enter the VR space.

[0656] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0657] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0658] 4. User: Says "Excuse me, I'm looking for an apple."

[0659] 5. Terminal: Records the user's voice and sends it to the server.

[0660] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0661] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[0662] 8. User: Follow the instructions and continue the experience.

[0663] By using the system of the present invention, users can efficiently learn a language through a realistic study abroad experience at home. Furthermore, repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[0664] The processing flow will be explained below.

[0665] Step 1:

[0666] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[0667] Step 2:

[0668] Device: Prompt the user for account information on the initial setup screen.

[0669] Step 3:

[0670] User: Enter your account information and submit.

[0671] Step 4:

[0672] Server: Receives and authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.).

[0673] Step 5:

[0674] On the device: The initial microphone setup screen for speech recognition is displayed, prompting the user to record a sample voice.

[0675] Step 6:

[0676] User: Record a sample voice and send it.

[0677] Step 7:

[0678] Server: Receives audio samples and performs initial pronunciation assessment.

[0679] Step 8:

[0680] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0681] Step 9:

[0682] Device: Starts downloading the learning content provided by the server.

[0683] Step 10:

[0684] Device: Once the content has finished downloading, it will be displayed in a list and the user will be prompted to make a selection.

[0685] Step 11:

[0686] User: Select a learning scenario (e.g., shopping).

[0687] Step 12:

[0688] Device: Generates and displays a VR space according to the selected scenario.

[0689] Step 13:

[0690] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[0691] Step 14:

[0692] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[0693] Step 15:

[0694] User: Says "Excuse me, I'm looking for an apple."

[0695] Step 16:

[0696] Device: Records the user's voice and sends it to the server.

[0697] Step 17:

[0698] Server: Analyzes the received voice data and evaluates the pronunciation.

[0699] Step 18:

[0700] Server: If the pronunciation of "ringo" is correct, generate feedback as "Excellent."

[0701] Step 19:

[0702] Terminal: A store clerk character will say, "It's here," and at the same time display the evaluation results.

[0703] Step 20:

[0704] User: Follow the instructions and continue the experience.

[0705] Step 21:

[0706] Server: Records learning progress data and prepares suggestions for the next learning session.

[0707] Step 22:

[0708] On your device: Show the user suggestions for their next study session.

[0709] Step 23:

[0710] User: Check out upcoming sessions and decide whether to book or start immediately.

[0711] Step 24:

[0712] Terminal: Prepares for the next session and notifies the user.

[0713] Example 1

[0714] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0715] In conventional language learning systems, users have few opportunities to practice conversations in a real language environment, which means that their pronunciation and communication skills are not fully improved. The present invention aims to solve this problem by providing a system that allows users to efficiently learn a language through a realistic study abroad experience from the comfort of their own home.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0717] In this invention, the server includes terminal means having a display device worn by the user, recognition means for recognizing and analyzing the user's voice, server means for selecting and providing learning content, VR scenario generation means for displaying the learning content provided by the server means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback to the user based on the results of the recognition means, means for recording the user's pronunciation as voice data and transmitting it to the server, and means for selecting and downloading appropriate learning content based on the analysis results from the server. This allows the user to practice pronunciation while experiencing realistic dialogue scenes, thereby achieving effective language learning.

[0718] "User" refers to a person who actually uses the system and wears a VR device to learn a language.

[0719] The term "terminal means" refers to an electronic device including a display device worn by a user, which exchanges information between the user and the server.

[0720] "Recognition means" refers to a device or software that has the function of recognizing and analyzing the user's voice and has the role of evaluating the voice data.

[0721] "Server means" refers to a computer system that has the function of selecting and providing learning content and analyzing user voice data.

[0722] "VR scenario generation means" refers to a device or software that generates a virtual reality scenario based on learning content provided by the server means and enables the user to interact in that environment.

[0723] The "feedback means" refers to a device or software that has the function of providing the user with visual or audio feedback on the quality of their pronunciation based on the results of the recognition means.

[0724] "Means for recording as voice data" refers to a device or software that has the function of recording the user's pronunciation and transmitting the voice data to a server.

[0725] "Means for downloading" refers to a device or software that has the function of downloading learning content provided by a server and displaying it to the user.

[0726] The system related to this invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the experience of actually studying abroad.

[0727] System configuration overview

[0728] User:

[0729] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0730] Device:

[0731] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server. For example, a display device such as Oculus Quest 2 is used.

[0732] server:

[0733] This is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Google Cloud Speech-to-Text API and other services are used for speech analysis.

[0734] Program Description

[0735] The user wearing the VR device launches a dedicated application and enters their account information to begin the initial setup. The server authenticates the user's account information and instructs them to configure the microphone used for voice recognition and record a sample voice. When the user records and sends the sample voice, the server analyzes the voice, performs an initial pronunciation evaluation, and provides learning content.

[0736] When a user selects a learning scenario, the device generates a corresponding VR space in which a virtual character speaks to the user. The user interacts with the character and practices pronunciation. After practice, the audio data is sent to the server for analysis and feedback. Feedback is provided to the user visually or audibly via the device.

[0737] Specific examples

[0738] Example of a shopping scenario

[0739] 1. User: Select the "Shopping" scenario and enter the VR space.

[0740] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0741] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0742] 4. User: Says "Excuse me, I'm looking for an apple."

[0743] 5. Terminal: Records the user's voice and sends it to the server.

[0744] 6. Server: Analyzes the audio and if the pronunciation of "apple" is correct, rates it as "Excellent."

[0745] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[0746] 8. User: Follow the instructions and continue the experience.

[0747] This system allows users to study a language efficiently while experiencing a realistic study abroad experience from home. Repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[0748] Prompt Sentence Examples

[0749] "Users experiencing virtual study abroad interact with a virtual store clerk in a shopping scenario and receive pronunciation evaluation and feedback."

[0750] The user puts on the VR device and selects the "Shopping" scenario. The device generates a VR space inside the store, and a store clerk character appears and asks, "What are you looking for?" The user says, "Excuse me, I'm looking for an apple," and the device records the speech and sends it to the server. The server analyzes the speech, and if the pronunciation is correct, it rates it as "Excellent," and the device displays the result. The user then continues the next dialogue.

[0751] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0752] Step 1:

[0753] The user puts on the VR device and launches the dedicated application.

[0754] Input: The user puts on the VR device and launches the application.

[0755] Output: The application starts and the initial setup screen is displayed.

[0756] Specific operation: The user wears a head-mounted display (e.g., Oculus Quest 2) and uses the controller to select and launch the dedicated application. When the application is launched, the device displays the initial setup screen.

[0757] Step 2:

[0758] The user enters and submits their account information.

[0759] Input: Enter your account information in the text boxes displayed on the initial setup screen.

[0760] Output: The entered account information is sent to the server.

[0761] Specific operation: The user enters account information such as email address, password, and username in the text boxes provided and presses the "Send" button. The device then sends this information to the server.

[0762] Step 3:

[0763] The server authenticates the account information and registers the initial settings.

[0764] Input: Account information sent from the device

[0765] Output: Authentication result and initial setting registration completion message

[0766] Specific operation: The server checks the received account information against the database and performs authentication. If authentication is successful, it registers initial information such as the user's learning goals and language level, and returns the results to the terminal.

[0767] Step 4:

[0768] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[0769] Input: Authentication result and initial setting information returned from the server

[0770] Output: Sample audio recording screen display

[0771] Specific operation: After receiving the authentication result, the device displays a screen prompting the user to set up voice recognition, asking them to say a sample voice such as "Hello."

[0772] Step 5:

[0773] User records and submits a sample voice

[0774] Input: A sample voice utterance from the user

[0775] Output: Recorded audio data is sent to the server

[0776] Specific operation: The user speaks a sample voice according to the instructions on the device. The device records the voice and sends it to the server as voice data.

[0777] Step 6:

[0778] The server analyzes the sample audio and makes an initial pronunciation assessment

[0779] Input: Sample audio data sent from the device

[0780] Output: Initial pronunciation assessment result and a message that the learning content is ready to be provided

[0781] Specific operation: The server uses a speech analysis algorithm such as the Google Cloud Speech-to-Text API to analyze the sample speech and evaluate the user's pronunciation level. Based on the evaluation results, it prepares appropriate learning content and sends a message to the device indicating that it is ready to be provided.

[0782] Step 7:

[0783] The device downloads the learning content and displays it in a list.

[0784] Input: Message from the server that learning content is ready to be provided

[0785] Output: Download and list of learning content

[0786] Specific operation: The device receives instructions from the server, downloads the suggested learning content, displays a list of downloaded learning content, and prompts the user to select one.

[0787] Step 8:

[0788] The user selects a learning scenario

[0789] Input: User selects from a list of learning content

[0790] Output: The selected learning scenario is displayed on the device.

[0791] Specific operation: The user selects a learning scenario, such as "Shopping," from the list of learning content displayed on the device and presses the "Start" button. The selected scenario is reflected on the device.

[0792] Step 9:

[0793] The device generates and displays the VR space.

[0794] Input: Information about the learning scenario selected by the user

[0795] Output: Display of VR space and virtual characters based on the selected scenario

[0796] Specific operation: The device generates and displays an appropriate VR space (e.g., a shopping mall) based on the information of the selected scenario, and a virtual character (e.g., a store clerk) also appears.

[0797] Step 10:

[0798] Users interact with virtual characters and practice their pronunciation

[0799] Input: Question from virtual character and user's response utterance

[0800] Output: Recorded user voice data

[0801] Specific operation: The virtual character asks a question such as "What are you looking for?", and the user responds with a voice such as "Excuse me, I'm looking for an apple." The device records this voice.

[0802] Step 11:

[0803] The device sends the recorded audio data to the server.

[0804] Input: User pronunciation recording

[0805] Output: Recorded audio data is sent to the server

[0806] Specific operation: The device compresses the recorded user voice data and sends it to a server via the Internet.

[0807] Step 12:

[0808] The server analyzes the audio data and generates pronunciation evaluation and feedback.

[0809] Input: Audio data sent from the device

[0810] Output: Pronunciation evaluation results and feedback information

[0811] Specific operation: The server analyzes the audio data using a speech analysis algorithm such as the Google Cloud Speech-to-Text API, evaluates the accuracy of the pronunciation, and generates feedback such as "Excellent" based on the evaluation result and sends it to the device.

[0812] Step 13:

[0813] The device displays and provides feedback to the user

[0814] Input: Feedback information sent from the server

[0815] Output: Visual and audio feedback

[0816] Specific operation: Based on the received feedback, the device displays or speaks an evaluation result such as "Excellent" to the user. The user confirms this feedback and continues with the next learning scenario.

[0817] (Application example 1)

[0818] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Conventional language learning systems have limited opportunities for users to experience language usage in everyday life, making it difficult to improve practical language skills. Furthermore, feedback is limited, making it difficult for users to self-correct. In particular, pronunciation evaluation and correction are not performed in real time, making efficient learning difficult.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0821] In this invention, the server includes a means for selecting and providing learning content, a means for evaluating a user's pronunciation, and a means for providing feedback. This allows users to learn a language in a virtual space where they select a shopping scenario and engage in interactive learning, creating a situation similar to real life. Furthermore, real-time pronunciation evaluation and feedback allow users to efficiently self-correct and improve their pronunciation.

[0822] The "terminal means" is an electronic device including a display device worn by a user, and has the function of exchanging information between the user and the server.

[0823] The "recognition means" has the function of recognizing and analyzing the user's voice.

[0824] The "server means" is a computer system that has the function of selecting and providing learning content.

[0825] The "VR scenario generation means" has the function of displaying the learning content provided by the server means via the display means and generating a virtual reality environment in which the user can engage in voice-based dialogue.

[0826] The "feedback means" has a function of providing the user with pronunciation evaluation results and correction instructions based on the results of the recognition means.

[0827] A "shopping scenario" is a scenario that provides a virtual shopping experience that a user can choose from for language learning.

[0828] "Interactive learning" is a method in which a user progresses through learning by interacting with a virtual character.

[0829] A "virtual space" is a virtual environment that users can experience through a VR device.

[0830] This system, which is comprised of three elements: a user, a terminal, and a server, provides an environment in which users can study a language while experiencing the sensation of actually studying abroad using virtual reality (VR). This system allows users to study a language in a virtual space where they can select a shopping scenario and engage in interactive learning, creating a situation that is close to real life.

[0831] System configuration overview

[0832] 1. User: The final user of the system, who wears the VR device and experiences the language learning content.

[0833] 2. Terminal: An electronic device including a VR device worn by the user that exchanges information between the user and the server. The terminal has the function of recording the user's pronunciation and sending it to the server.

[0834] 3. Server: A computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Specifically, it performs speech recognition and evaluation using the Python requests and speech_recognition libraries. Feedback is provided using the pyttsx3 library.

[0835] Processing flow

[0836] First-time setup

[0837] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0838] User: Enter your account information and submit.

[0839] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[0840] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0841] User: Record a sample voice and send it.

[0842] Server: Receives audio samples and performs initial pronunciation assessment.

[0843] Pronunciation evaluation and content provision

[0844] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0845] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0846] Experience the learning scenario

[0847] User: Select a learning scenario (e.g., shopping).

[0848] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[0849] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0850] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0851] Terminal: Receives feedback and provides it to the user visually or audibly.

[0852] Specific examples

[0853] Example: Shopping Scenario

[0854] 1. User: Select the "Shopping" scenario and enter the VR space.

[0855] 2. Terminal: Generates a virtual space inside a shopping mall or store, and virtual characters (store clerks) appear.

[0856] 3. Terminal: A virtual character asks, "What are you looking for?"

[0857] 4. User: Says "Excuse me, I'm looking for an apple."

[0858] 5. Terminal: Records the user's voice and sends it to the server.

[0859] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0860] 7. Terminal: A virtual character will say, "It's here," and simultaneously display the evaluation results.

[0861] 8. User: Follow the instructions and continue the experience.

[0862] Prompt Sentence Examples

[0863] If the user says "Excuse me, I'm looking for an apple," the prompt is:

[0864] text

[0865] Evaluate the pronunciation of "Excuse me, I'm looking for an apple" in Japanese and generate an appropriate feedback message.

[0866] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0867] Step 1:

[0868] First-time setup

[0869] Device: Put on the VR device and launch the dedicated application. The initial setup screen will prompt the user to enter their account information.

[0870] User: Enter your account information (e.g., username, email address, etc.) and submit.

[0871] Terminal input: Account information. Output: Sending account information to the server.

[0872] Server: Receives the entered account information and verifies it against the database. After authentication, it registers the user's initial settings (learning purpose, language level, etc.), configures the microphone used for voice recognition, and asks for a sample voice recording.

[0873] Server input: Account information. Output: Authentication result and instructions to register initial setting information.

[0874] Step 2:

[0875] Voice Recognition Settings

[0876] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0877] User: Record a sample voice and send it.

[0878] Input: Microphone settings, sample audio. Output: Send sample audio data to the server.

[0879] Server: Analyzes the received sample voice data, performs an initial pronunciation evaluation, and stores the evaluation results in a database.

[0880] Server input: Sample speech data. Output: Initial pronunciation evaluation results.

[0881] Step 3:

[0882] Selection and provision of learning content

[0883] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0884] Server input: User's speech data. Output: Pronunciation evaluation results and appropriate learning content.

[0885] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[0886] Terminal input: Instruction to download learning content. Output: List of learning content.

[0887] Step 4:

[0888] Experience the learning scenario

[0889] User: Select a learning scenario (e.g., shopping).

[0890] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[0891] Terminal input: Selection of learning scenario. Output: Generation and display of virtual space.

[0892] Step 5:

[0893] Pronunciation practice and feedback

[0894] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[0895] Input on the device: User's pronunciation. Output: Sending voice data to the server.

[0896] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[0897] Server input: User's speech data. Output: Pronunciation evaluation and feedback.

[0898] Terminal: Receives feedback and provides it to the user visually or audibly.

[0899] Terminal input: Feedback data. Output: Display feedback or audio playback.

[0900] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0901] The system according to the present invention is composed of four main elements: a user, a terminal, a server, and an emotion engine. This system uses virtual reality (VR) to provide a language learning environment in which users can experience studying abroad in a virtual world. Furthermore, by recognizing the user's emotions and adapting the learning content and feedback, a more effective learning experience can be achieved. Specific embodiments for implementing the present invention are described below.

[0902] System configuration overview

[0903] User:

[0904] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[0905] Device:

[0906] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and emotional data and sending it to the server.

[0907] server:

[0908] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation and emotions.

[0909] Emotion Engine:

[0910] It has an algorithm that recognizes the user's emotions, analyzes the user's voice and facial expression data, and determines their emotional state. It also has the ability to adapt learning content and feedback according to the user's emotional state.

[0911] Explanation of program processing procedures

[0912] First-time setup

[0913] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[0914] User: Enter your account information and submit.

[0915] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and prompts the user to record a sample voice.

[0916] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[0917] User: Record a sample voice and send it.

[0918] Server: Receives audio samples and performs initial pronunciation assessment.

[0919] Pronunciation evaluation and content provision

[0920] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0921] Device: Starts downloading the learning content provided by the server.

[0922] On your device: Once the downloads are complete, they will be displayed in a list and you will be prompted to make a selection.

[0923] Experience the learning scenario

[0924] User: Selects a learning scenario (e.g., shopping).

[0925] Device: Generates and displays a VR space according to the selected scenario.

[0926] Terminal: A virtual character appears and speaks to the user.

[0927] Emotion engine: Analyzes the user's voice and facial expression data in real time to determine the user's emotional state.

[0928] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[0929] User: Says "Excuse me, I'm looking for an apple."

[0930] Terminal: Records the user's voice and facial expression data and sends them to the server.

[0931] Server: Analyzes the received voice data and evaluates the pronunciation.

[0932] Emotion engine: Analyzes the received emotion data and evaluates the user's emotional state.

[0933] Server: Generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[0934] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The feedback will also be adjusted according to the user's emotional state.

[0935] Specific examples

[0936] Example: Shopping Scenario

[0937] 1. User: Select the "Shopping" scenario and enter the VR space.

[0938] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[0939] 3. Terminal: The store clerk character asks, "What are you looking for?"

[0940] 4. User: Says "Excuse me, I'm looking for an apple."

[0941] 5. Terminal: Records the user's voice and facial expression data and sends them to the server.

[0942] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[0943] 7. Emotion Engine: Analyzes the user's facial expression data and evaluates their emotional state. For example, if the user is tired, it will provide gentle feedback.

[0944] 8. Terminal: A store clerk character will say "It's here" and simultaneously display the evaluation result. The feedback will also be adjusted according to the customer's emotional state.

[0945] 9. User: Follow the instructions and continue the experience.

[0946] By using the system of the present invention, users can efficiently learn a language while enjoying a realistic study abroad experience from the comfort of their own home. Furthermore, the emotion engine makes the learning experience more personalized, improving user motivation and learning effectiveness.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[0950] Step 2:

[0951] Device: Prompt the user for account information on the initial setup screen.

[0952] Step 3:

[0953] User: Enter your account information and submit.

[0954] Step 4:

[0955] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and instructs the user to record a sample voice.

[0956] Step 5:

[0957] Device: Prompt the user to record a sample audio on the initial setup screen.

[0958] Step 6:

[0959] User: Follow the instructions to record and submit a sample audio.

[0960] Step 7:

[0961] Server: Receives audio samples and performs initial pronunciation assessment.

[0962] Step 8:

[0963] Server: Analyzes the received voice data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[0964] Step 9:

[0965] Device: Starts downloading the learning content provided by the server.

[0966] Step 10:

[0967] On the device: Once the download is complete, the contents will be displayed in a list and the user will be prompted to make a selection.

[0968] Step 11:

[0969] User: Select a learning scenario (e.g., shopping).

[0970] Step 12:

[0971] Device: Generates and displays a VR space according to the selected scenario.

[0972] Step 13:

[0973] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[0974] Step 14:

[0975] Terminal: The character asks "What are you looking for?" and waits for voice input.

[0976] Step 15:

[0977] User: Says "Excuse me, I'm looking for an apple."

[0978] Step 16:

[0979] Terminal: Records the user's voice and facial expression data and sends them to the server and emotion engine.

[0980] Step 17:

[0981] Server: Analyzes the received voice data and evaluates the pronunciation.

[0982] Step 18:

[0983] Emotion engine: Analyzes received facial expression data and evaluates the user's emotional state.

[0984] Step 19:

[0985] Server: Generates feedback based on pronunciation evaluation results and emotional state.

[0986] Step 20:

[0987] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The system also adjusts the feedback content according to the user's emotional state. For example, if the user is nervous, it will provide a comment to help them relax.

[0988] Step 21:

[0989] User: Follow the instructions and continue the experience.

[0990] Step 22:

[0991] Server: Records learning progress and prepares suggestions for the next lesson.

[0992] Step 23:

[0993] On your device: Display suggestions for your next study session and prompt the user to choose their next step.

[0994] Step 24:

[0995] User: Review the proposal and decide to book your next session or start immediately.

[0996] Step 25:

[0997] Terminal: Prepares for the next session and notifies the user.

[0998] This series of processes allows users to study language efficiently in a relaxed state. The emotional engine individually optimizes the user's learning experience, which is expected to improve motivation and learning effectiveness.

[0999] Example 2

[1000] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] Existing language learning systems provide uniform pronunciation evaluation and feedback to users, which can lead to insufficient learning outcomes for individual users. Furthermore, feedback and content adaptation do not take into account the user's emotional state, which can lead to a decrease in motivation and learning effectiveness. Furthermore, there is a lack of effective ways to provide real-life dialogue scenarios in VR environments, making it difficult to achieve a realistic learning experience.

[1002] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1003] In this invention, the server includes a means for selecting and providing learning content, an emotion evaluation means for evaluating the user's emotional state, and an adaptation means for adapting the learning content and feedback. This allows for personalized responses that take into account the user's individual pronunciation evaluation and emotional state, improving learning effectiveness and motivation. Furthermore, realistic dialogue scenarios can be provided in a VR environment, providing a more realistic language learning experience.

[1004] The "terminal means" is a device that has a display device worn by a user and exchanges information with the server and other system elements.

[1005] A "recognition means" is a device or software that has the function of recognizing and analyzing a user's voice.

[1006] The "server means" is a computer system that selects and provides learning content and manages data in cooperation with the recognition means and other system elements.

[1007] The "VR scenario generation means" is a device or software that displays the learning content provided by the server means via the display means and generates a virtual reality scenario in which the user can engage in voice-based dialogue.

[1008] The "emotion evaluation means" is a device or software that has the function of evaluating the user's emotional state based on the results of the recognition means.

[1009] The "adaptation means" is a device or software that has the function of adapting the learning content or feedback based on the evaluation results of the emotion evaluation means.

[1010] The "feedback means" is a device or software that has a function of providing feedback to the user based on the results of the recognition means and emotion evaluation means.

[1011] The system according to the present invention is a system comprising a terminal means having a display device worn by the user, a recognition means for recognizing and analyzing the user's voice, a server means for selecting and providing learning content, a VR scenario generation means for generating a virtual reality (VR) scenario, an emotion evaluation means for evaluating the user's emotional state, an adaptation means for adapting the learning content and feedback, and a means for providing feedback.

[1012] Hardware and software used

[1013] 1. Terminal means: The display device worn by the user is a VR device such as a head-mounted display (HMD). This device is equipped with a high-performance graphics card and sensors to provide an interactive VR environment. Specifically, HMDs include "Oculus Rift" and "HTC Vive."

[1014] 2. Recognition method: Software for recognizing and analyzing the user's voice can use the voice recognition libraries "Google Speech-to-Text API" and "Microsoft Azure Speech Service."

[1015] 3. Server: The server is responsible for selecting and providing the learning content. The server is equipped with a high-performance processor and storage, and runs a database and network management system. For example, a cloud server from Amazon Web Services (AWS) can be used.

[1016] 4. VR scenario generation means: To generate a VR scenario that displays the learning content provided by the server and allows the user to interact based on voice, the game engines "Unity" and "Unreal Engine" can be used.

[1017] 5. Emotion assessment method: To assess the user's emotional state, we use emotion recognition software "Affectiva SDK" and "Microsoft Azure Emotion API" that analyze the user's voice and facial expression data.

[1018] 6. Adaptation: Software that adapts learning and feedback in real time uses algorithms running on servers that utilize machine learning and natural language processing (NLP) techniques.

[1019] 7. Feedback methods: Feedback can be provided using Text-to-Speech (TTS) systems or display pop-up technology to present feedback to the user visually or audibly.

[1020] Specific examples

[1021] Example scenario: Shopping experience

[1022] 1. User: Put on the VR device, launch the application, set up an account for the first time, and record a sample voice.

[1023] 2. Device: Once the initial setup is complete, a list of learning content provided by the server will be displayed.

[1024] 3. User: Select the "Shopping" scenario and enter the VR space.

[1025] 4. Terminal: A VR space of a shopping mall is generated, and virtual characters (store clerks) appear.

[1026] 5. Terminal: The store clerk character asks, "Welcome, what are you looking for?"

[1027] 6. User: Says, "Excuse me, I'm looking for an apple."

[1028] 7. Terminal: Records voice and facial expression data and sends it to the server.

[1029] 8. Server: Analyzes the audio and evaluates the pronunciation.

[1030] 9. Emotion Engine: Analyzes facial expression data and evaluates emotional state, for example, recognizing that the user is stressed.

[1031] 10. Server: Generates feedback based on the evaluation results and emotional state and sends it to the device.

[1032] 11. Terminal: The virtual character will say "It's here" and display the evaluation result. The evaluation result will be "Excellent," and depending on the user's emotional state, a positive message such as "Your pronunciation is really good!" will be displayed.

[1033] Prompt Sentence Examples

[1034] In a shopping scenario, describe the process flow for when a user asks a store clerk if they are looking for an apple.

[1035] This system allows users to enjoy a highly realistic learning experience and learn languages ​​more effectively through individually optimized feedback and learning content. Personalized feedback based on the user's emotional state also increases learning motivation.

[1036] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1037] Step 1:

[1038] The user puts on the VR device and launches the dedicated application.

[1039] Input: The user powers on the VR device and selects the application icon.

[1040] Specific operation: The user wears the HMD device on their head and launches an application using the controller in their hand.

[1041] Output: The application's initial setup screen is displayed, prompting the user to enter their account information.

[1042] Step 2:

[1043] The device displays the initial setup screen and prompts the user to enter their account information.

[1044] Input: The application launches and the initial setup screen appears, presenting the user with input fields such as "Name," "Email address," and "Password."

[1045] Specific behavior: The initial setup form will be displayed in the VR space.

[1046] Output: A screen is prepared for the user to enter their account information.

[1047] Step 3:

[1048] The user enters and submits their account information.

[1049] Input: The user enters the required information into the designated fields using a keyboard or voice input.

[1050] Specific behavior: The user clicks the "Submit" button.

[1051] Output: The entered account information is sent to the server.

[1052] Step 4:

[1053] The server receives the account information and performs authentication.

[1054] Input: Account information submitted by the user.

[1055] Specific operation: The server connects to the database and verifies the input information to perform authentication.

[1056] Output: The authentication result is generated (success or failure). If authentication is successful, the user's initial setup is completed.

[1057] Step 5:

[1058] The server displays a screen to guide the user through the speech recognition test and asks the user to record a sample voice.

[1059] Input: Account authentication successful, user's initial setup data.

[1060] Specific operation: The server sends a voice recognition setting screen to the device and instructs it to "record a sample voice."

[1061] Output: The voice recognition settings screen will be displayed on the device.

[1062] Step 6:

[1063] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[1064] Input: Speech recognition setting screen sent from the server.

[1065] Specific operation: Audio recording instructions will be displayed in the VR space.

[1066] Output: The user is ready to start recording.

[1067] Step 7:

[1068] User records and submits a sample voice

[1069] Input: The user reads the prompted sentence (e.g., "I want to learn Japanese").

[1070] Specific behavior: The user clicks the record button and speaks the specified sentence.

[1071] Output: The recorded audio data is stored on the device and sent to the server.

[1072] Step 8:

[1073] The server receives the sample audio and performs an initial pronunciation assessment

[1074] Input: Audio data sent by the user.

[1075] What it does: It uses a speech analysis algorithm to analyze the speech data and evaluate the accuracy of the pronunciation.

[1076] Output: The initial pronunciation evaluation result is generated.

[1077] Step 9:

[1078] The server analyzes the voice data and evaluates the user's pronunciation level.

[1079] Input: User speech data and an existing pronunciation reference model.

[1080] What it does: Uses machine learning algorithms to compare and evaluate your voice data.

[1081] Output: The user's pronunciation level evaluation result is generated.

[1082] Step 10:

[1083] The server selects appropriate learning content based on the evaluation results.

[1084] Input: Pronunciation level assessment results and user learning objectives.

[1085] Specific actions: Select appropriate learning materials from a database of learning content.

[1086] Output: Selected learning content data is generated.

[1087] Step 11:

[1088] The server sends the selected learning content to the device.

[1089] Input: Selected learning content data.

[1090] Specific operation: Divide the learning content data into packets and send them to the terminal.

[1091] Output: The device receives the learning content data.

[1092] Step 12:

[1093] The device downloads the learning content and displays it in a list.

[1094] Input: Learning content data sent from the server.

[1095] Specific behavior: Downloads content and displays a progress bar. Once completed, displays a list of learning content.

[1096] Output: A list of learning content will be displayed on the screen.

[1097] Step 13:

[1098] The user selects a learning scenario (e.g., shopping)

[1099] Input: A list of learning content.

[1100] Specific behavior: The user selects the desired scenario from a list. Example: "Shopping".

[1101] Output: The selected learning scenario is recorded on the device.

[1102] Step 14:

[1103] The device generates and displays a VR space according to the selected scenario.

[1104] Input: Data for the scenario selected by the user.

[1105] Specific operation: Use "Unity" or "Unreal Engine" to render and display the corresponding VR space.

[1106] Output: The VR space is displayed in the user's field of view.

[1107] Step 15:

[1108] The device displays a virtual character and starts a conversation

[1109] Input: The learning scenario selected by the user.

[1110] What it does: A virtual character appears in the user's field of vision, plays a preset audio file, and asks, "What are you looking for?"

[1111] Output: A virtual character begins to interact.

[1112] Step 16:

[1113] The emotion engine analyzes the user's voice and facial expression data in real time to determine their emotional state.

[1114] Input: Real-time voice and facial expression data from the user.

[1115] Specific operations: Emotion analysis is performed using the "Affectiva SDK" and "Microsoft Azure Emotion API."

[1116] Output: The user's emotional state assessment result is generated.

[1117] Step 17:

[1118] The device records the user's voice and facial expression data and sends it to the server.

[1119] Input: Speech and facial expressions made by the user.

[1120] Specific operation: Captures voice and facial expression data and sends it to the server.

[1121] Output: The recorded data is sent to the server.

[1122] Step 18:

[1123] The server analyzes the received voice data and evaluates the pronunciation

[1124] Input: User's voice data.

[1125] What it does: Uses a speech recognition algorithm to assess the accuracy of the user's pronunciation.

[1126] Output: Pronunciation evaluation results are generated.

[1127] Step 19:

[1128] The emotion engine analyzes the received emotion data and evaluates the user's emotional state.

[1129] Input: User's facial expression data.

[1130] What it does: Uses an emotion analysis algorithm to assess emotional state.

[1131] Output: The emotion evaluation result is generated.

[1132] Step 20:

[1133] The server generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[1134] Input: Pronunciation evaluation results and emotion evaluation results.

[1135] Specific operations: Generate pronunciation feedback and responses according to the emotional state and send them to the device.

[1136] Output: Feedback data is sent to the device.

[1137] Step 21:

[1138] The device controls the virtual character and provides feedback to the user.

[1139] Input: Feedback data sent by the server.

[1140] Specific behavior: A virtual character will say "It's here" and display the evaluation result. Feedback such as "Excellent" or a positive message will be displayed.

[1141] Output: A virtual character will provide guidance and feedback.

[1142] (Application example 2)

[1143] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1144] Conventional language learning systems have struggled to effectively assess users' pronunciation and comprehension and provide a personalized learning experience. Furthermore, they were unable to provide feedback based on the user's emotional state, resulting in insufficient learning effectiveness. Furthermore, it was difficult to replicate the experience of actual in-store customer service, making it difficult to improve practical language skills.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes terminal means having a visual display device worn by the user, voice recognition means for recognizing and analyzing the user's voice, information processing device means for selecting and providing learning content, virtual space generation means for displaying the learning content provided by the information processing device means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback based on the results of the voice recognition means and the user's emotional state, and emotion analysis means for analyzing the user's emotions. This allows the user to receive personalized feedback based on the emotion analysis, enabling a practical language learning experience within a virtual store.

[1146] A "visual display device" is a device worn by a user to receive visual information and provide a virtual reality experience.

[1147] "Terminal means" refers to electronic equipment used by a user, including a visual display device, for communicating with a server to exchange information.

[1148] The "voice recognition means" is a device that has the function of recording the user's voice and analyzing that voice.

[1149] The "information processing device means" is a computer system such as a server for generating and providing learning content.

[1150] "Virtual space generating means" is a device that has the function of generating a virtual reality environment and allowing a user to interact within it.

[1151] A "feedback means" is a device that has the function of providing visual or audio feedback to a user.

[1152] "Emotion analysis means" refers to an algorithm or device that analyzes the user's voice and facial expression data and determines their emotional state.

[1153] "User" refers to the final user who uses this system to study.

[1154] The present invention provides a system that allows users to experience practical language learning in a virtual store, which includes a terminal means including a visual display device, a voice recognition means, an information processing means, a virtual space generation means, a feedback means, and an emotion analysis means.

[1155] First, the user wears a visual display device (e.g., smart glasses). The terminal generates a virtual space of a virtual store via the visual display device and displays it to the user. This virtual space generation means allows the user to experience shopping in an actual store.

[1156] The user's voice data is collected and analyzed by a voice recognition means, which may be implemented by a microphone mounted on a computer or smart glasses, and the voice commands issued by the user are collected in real time and transmitted to a server.

[1157] The server analyzes the voice data and evaluates the user's pronunciation level. At this time, an emotion analysis tool is also used based on the user's facial expressions and voice data to determine the user's emotional state. Emotion analysis uses a generative AI model incorporating facial expression recognition technology and voice emotion analysis technology.

[1158] Based on the analyzed voice data and emotion data, the information processing device generates learning content and recommendations suitable for the user. This information is transmitted to the terminal device and provided to the user through the VR space.

[1159] The feedback provided to the user is displayed visually or audibly by a feedback means, which provides personalized recommendations according to the quality of the user's pronunciation and the user's emotional state.

[1160] As a concrete example, consider the case where a user enters a virtual store in a virtual space and issues a voice command such as, "Excuse me, I'm looking for an apple." The voice is analyzed, and if the pronunciation is correct, the word "Excellent" is displayed. If the user has a relaxed expression, a virtual store clerk character praises the user's excellent pronunciation. Conversely, if the user appears tired, feedback is given in a gentler tone.

[1161] An example of a prompt sentence is, "Analyze the following user's voice data and facial expression data, and indicate the user's pronunciation evaluation and emotional state. Also, output recommendations to be provided to the user in relation to the shopping scenario. Voice data: Voice data file Facial expression data: 'smile: 0.8, anger: 0.2'."

[1162] In this way, providing real-time feedback based on the user's performance and emotional state can lead to a more effective and personalized learning experience.

[1163] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1164] Step 1:

[1165] The user puts on the visual display device and starts up the terminal.

[1166] Input: The user performs an action to start the terminal.

[1167] Output: The initial device setup screen will be displayed.

[1168] Specific operation: The terminal displays the initial screen of the virtual store via the visual display device.

[1169] Step 2:

[1170] The user enters their account information and sends it to the server.

[1171] Input: The user enters their account information and presses the "Submit" button.

[1172] Output: The server receives the account information and initializes the user profile.

[1173] Specific operation: The server performs initial settings for the user based on the received account information and sends a setup completion message to the terminal.

[1174] Step 3:

[1175] A voice sample of the user is collected and sent to a server.

[1176] Input: The user records a sample voice on the voice settings screen and presses the "Submit" button.

[1177] Output: The server receives the sample speech data and performs an initial pronunciation assessment.

[1178] How it works: The device uses a microphone to record the user's voice and sends the data to the server, which then analyzes the received voice and evaluates the pronunciation level.

[1179] Step 4:

[1180] The user's voice data and facial expression data are analyzed to evaluate their emotions.

[1181] Input: Voice data collected by speech recognition means and facial expression data collected by camera function.

[1182] Output: The server stores the user's pronunciation rating and emotion rating as analysis results.

[1183] Specific operation: The server uses a generative AI model to analyze voice data and evaluate pronunciation. At the same time, it analyzes facial expression data and evaluates the user's emotional state.

[1184] Step 5:

[1185] Generate learning content and recommendations and send them to the device.

[1186] Input: Speech and emotion data analyzed by the server.

[1187] Output: Appropriate learning content and recommendations sent to the device.

[1188] Specific operation: The server uses the information processing device means to generate learning content and feedback that is optimal for the user's learning progress and emotional state, and transmits them to the terminal.

[1189] Step 6:

[1190] The device generates a virtual space and displays the learning content.

[1191] Input: Learning content sent from the server.

[1192] Output: The learning scenario in the virtual space that is displayed to the user.

[1193] Specific operation: The terminal uses the virtual space generation means to generate a scenario for a virtual store based on the learning content received from the server, and displays it to the user via a visual display device.

[1194] Step 7:

[1195] The user interacts within the virtual space and continues the experience.

[1196] Input: An action that the user gives voice instructions for.

[1197] Output: New collected speech and facial expression data.

[1198] Specific operation: The device uses a microphone and camera to collect the user's voice and facial expression data and transmits it back to the server.

[1199] Step 8:

[1200] Provide visual or audio feedback to the user.

[1201] Input: User's voice and facial expression data reanalyzed by the server.

[1202] Output: Feedback displayed on the terminal.

[1203] Specific operation: Based on the data reanalyzed by the server, feedback that is optimal for the user's pronunciation and emotional state is generated and provided to the user via the device.

[1204] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1205] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1206] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1207] [Third embodiment]

[1208] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1209] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1210] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1211] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1212] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1213] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1214] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1215] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1216] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1217] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1218] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1219] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1220] The system according to the present invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the sensation of actually studying abroad. Specific embodiments for carrying out the present invention are described below.

[1221] System configuration overview

[1222] User:

[1223] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[1224] Device:

[1225] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server.

[1226] server:

[1227] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation.

[1228] Explanation of program processing procedures

[1229] First-time setup

[1230] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[1231] User: Enter your account information and submit.

[1232] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[1233] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[1234] User: Record a sample voice and send it.

[1235] Server: Receives audio samples and performs initial pronunciation assessment.

[1236] Pronunciation evaluation and content provision

[1237] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1238] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[1239] Experience the learning scenario

[1240] User: Selects a learning scenario (e.g., shopping).

[1241] Terminal: Generates and displays a VR space according to the selected scenario. A virtual character appears and speaks to the user.

[1242] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[1243] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[1244] Terminal: Receives feedback and provides it to the user visually or audibly.

[1245] Specific examples

[1246] Example: Shopping Scenario

[1247] 1. User: Select the "Shopping" scenario and enter the VR space.

[1248] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[1249] 3. Terminal: The store clerk character asks, "What are you looking for?"

[1250] 4. User: Says "Excuse me, I'm looking for an apple."

[1251] 5. Terminal: Records the user's voice and sends it to the server.

[1252] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[1253] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[1254] 8. User: Follow the instructions and continue the experience.

[1255] By using the system of the present invention, users can efficiently learn a language through a realistic study abroad experience at home. Furthermore, repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[1256] The processing flow will be explained below.

[1257] Step 1:

[1258] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[1259] Step 2:

[1260] Device: Prompt the user for account information on the initial setup screen.

[1261] Step 3:

[1262] User: Enter your account information and submit.

[1263] Step 4:

[1264] Server: Receives and authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.).

[1265] Step 5:

[1266] On the device: The initial microphone setup screen for speech recognition is displayed, prompting the user to record a sample voice.

[1267] Step 6:

[1268] User: Record a sample voice and send it.

[1269] Step 7:

[1270] Server: Receives audio samples and performs initial pronunciation assessment.

[1271] Step 8:

[1272] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1273] Step 9:

[1274] Device: Starts downloading the learning content provided by the server.

[1275] Step 10:

[1276] Device: Once the content has finished downloading, it will be displayed in a list and the user will be prompted to make a selection.

[1277] Step 11:

[1278] User: Select a learning scenario (e.g., shopping).

[1279] Step 12:

[1280] Device: Generates and displays a VR space according to the selected scenario.

[1281] Step 13:

[1282] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[1283] Step 14:

[1284] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[1285] Step 15:

[1286] User: Says "Excuse me, I'm looking for an apple."

[1287] Step 16:

[1288] Device: Records the user's voice and sends it to the server.

[1289] Step 17:

[1290] Server: Analyzes the received voice data and evaluates the pronunciation.

[1291] Step 18:

[1292] Server: If the pronunciation of "ringo" is correct, generate feedback as "Excellent."

[1293] Step 19:

[1294] Terminal: A store clerk character will say, "It's here," and at the same time display the evaluation results.

[1295] Step 20:

[1296] User: Follow the instructions and continue the experience.

[1297] Step 21:

[1298] Server: Records learning progress data and prepares suggestions for the next learning session.

[1299] Step 22:

[1300] On your device: Show the user suggestions for their next study session.

[1301] Step 23:

[1302] User: Check out upcoming sessions and decide whether to book or start immediately.

[1303] Step 24:

[1304] Terminal: Prepares for the next session and notifies the user.

[1305] Example 1

[1306] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1307] In conventional language learning systems, users have few opportunities to practice conversations in a real language environment, which means that their pronunciation and communication skills are not fully improved. The present invention aims to solve this problem by providing a system that allows users to efficiently learn a language through a realistic study abroad experience from the comfort of their own home.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1309] In this invention, the server includes terminal means having a display device worn by the user, recognition means for recognizing and analyzing the user's voice, server means for selecting and providing learning content, VR scenario generation means for displaying the learning content provided by the server means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback to the user based on the results of the recognition means, means for recording the user's pronunciation as voice data and transmitting it to the server, and means for selecting and downloading appropriate learning content based on the analysis results from the server. This allows the user to practice pronunciation while experiencing realistic dialogue scenes, thereby achieving effective language learning.

[1310] "User" refers to a person who actually uses the system and wears a VR device to learn a language.

[1311] The term "terminal means" refers to an electronic device including a display device worn by a user, which exchanges information between the user and the server.

[1312] "Recognition means" refers to a device or software that has the function of recognizing and analyzing the user's voice and has the role of evaluating the voice data.

[1313] "Server means" refers to a computer system that has the function of selecting and providing learning content and analyzing user voice data.

[1314] "VR scenario generation means" refers to a device or software that generates a virtual reality scenario based on learning content provided by the server means and enables the user to interact in that environment.

[1315] The "feedback means" refers to a device or software that has the function of providing the user with visual or audio feedback on the quality of their pronunciation based on the results of the recognition means.

[1316] "Means for recording as voice data" refers to a device or software that has the function of recording the user's pronunciation and transmitting the voice data to a server.

[1317] "Means for downloading" refers to a device or software that has the function of downloading learning content provided by a server and displaying it to the user.

[1318] The system related to this invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the experience of actually studying abroad.

[1319] System configuration overview

[1320] User:

[1321] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[1322] Device:

[1323] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server. For example, a display device such as Oculus Quest 2 is used.

[1324] server:

[1325] This is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Google Cloud Speech-to-Text API and other services are used for speech analysis.

[1326] Program Description

[1327] The user wearing the VR device launches a dedicated application and enters their account information to begin the initial setup. The server authenticates the user's account information and instructs them to configure the microphone used for voice recognition and record a sample voice. When the user records and sends the sample voice, the server analyzes the voice, performs an initial pronunciation evaluation, and provides learning content.

[1328] When a user selects a learning scenario, the device generates a corresponding VR space in which a virtual character speaks to the user. The user interacts with the character and practices pronunciation. After practice, the audio data is sent to the server for analysis and feedback. Feedback is provided to the user visually or audibly via the device.

[1329] Specific examples

[1330] Example of a shopping scenario

[1331] 1. User: Select the "Shopping" scenario and enter the VR space.

[1332] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[1333] 3. Terminal: The store clerk character asks, "What are you looking for?"

[1334] 4. User: Says "Excuse me, I'm looking for an apple."

[1335] 5. Terminal: Records the user's voice and sends it to the server.

[1336] 6. Server: Analyzes the audio and if the pronunciation of "apple" is correct, rates it as "Excellent."

[1337] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[1338] 8. User: Follow the instructions and continue the experience.

[1339] This system allows users to study a language efficiently while experiencing a realistic study abroad experience from home. Repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[1340] Prompt Sentence Examples

[1341] "Users experiencing virtual study abroad interact with a virtual store clerk in a shopping scenario and receive pronunciation evaluation and feedback."

[1342] The user puts on the VR device and selects the "Shopping" scenario. The device generates a VR space inside the store, and a store clerk character appears and asks, "What are you looking for?" The user says, "Excuse me, I'm looking for an apple," and the device records the speech and sends it to the server. The server analyzes the speech, and if the pronunciation is correct, it rates it as "Excellent," and the device displays the result. The user then continues the next dialogue.

[1343] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1344] Step 1:

[1345] The user puts on the VR device and launches the dedicated application.

[1346] Input: The user puts on the VR device and launches the application.

[1347] Output: The application starts and the initial setup screen is displayed.

[1348] Specific operation: The user wears a head-mounted display (e.g., Oculus Quest 2) and uses the controller to select and launch the dedicated application. When the application is launched, the device displays the initial setup screen.

[1349] Step 2:

[1350] The user enters and submits their account information.

[1351] Input: Enter your account information in the text boxes displayed on the initial setup screen.

[1352] Output: The entered account information is sent to the server.

[1353] Specific operation: The user enters account information such as email address, password, and username in the text boxes provided and presses the "Send" button. The device then sends this information to the server.

[1354] Step 3:

[1355] The server authenticates the account information and registers the initial settings.

[1356] Input: Account information sent from the device

[1357] Output: Authentication result and initial setting registration completion message

[1358] Specific operation: The server checks the received account information against the database and performs authentication. If authentication is successful, it registers initial information such as the user's learning goals and language level, and returns the results to the terminal.

[1359] Step 4:

[1360] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[1361] Input: Authentication result and initial setting information returned from the server

[1362] Output: Sample audio recording screen display

[1363] Specific operation: After receiving the authentication result, the device displays a screen prompting the user to set up voice recognition, asking them to say a sample voice such as "Hello."

[1364] Step 5:

[1365] User records and submits a sample voice

[1366] Input: A sample voice utterance from the user

[1367] Output: Recorded audio data is sent to the server

[1368] Specific operation: The user speaks a sample voice according to the instructions on the device. The device records the voice and sends it to the server as voice data.

[1369] Step 6:

[1370] The server analyzes the sample audio and makes an initial pronunciation assessment

[1371] Input: Sample audio data sent from the device

[1372] Output: Initial pronunciation assessment result and a message that the learning content is ready to be provided

[1373] Specific operation: The server uses a speech analysis algorithm such as the Google Cloud Speech-to-Text API to analyze the sample speech and evaluate the user's pronunciation level. Based on the evaluation results, it prepares appropriate learning content and sends a message to the device indicating that it is ready to be provided.

[1374] Step 7:

[1375] The device downloads the learning content and displays it in a list.

[1376] Input: Message from the server that learning content is ready to be provided

[1377] Output: Download and list of learning content

[1378] Specific operation: The device receives instructions from the server, downloads the suggested learning content, displays a list of downloaded learning content, and prompts the user to select one.

[1379] Step 8:

[1380] The user selects a learning scenario

[1381] Input: User selects from a list of learning content

[1382] Output: The selected learning scenario is displayed on the device.

[1383] Specific operation: The user selects a learning scenario, such as "Shopping," from the list of learning content displayed on the device and presses the "Start" button. The selected scenario is reflected on the device.

[1384] Step 9:

[1385] The device generates and displays the VR space.

[1386] Input: Information about the learning scenario selected by the user

[1387] Output: Display of VR space and virtual characters based on the selected scenario

[1388] Specific operation: The device generates and displays an appropriate VR space (e.g., a shopping mall) based on the information of the selected scenario, and a virtual character (e.g., a store clerk) also appears.

[1389] Step 10:

[1390] Users interact with virtual characters and practice their pronunciation

[1391] Input: Question from virtual character and user's response utterance

[1392] Output: Recorded user voice data

[1393] Specific operation: The virtual character asks a question such as "What are you looking for?", and the user responds with a voice such as "Excuse me, I'm looking for an apple." The device records this voice.

[1394] Step 11:

[1395] The device sends the recorded audio data to the server.

[1396] Input: User pronunciation recording

[1397] Output: Recorded audio data is sent to the server

[1398] Specific operation: The device compresses the recorded user voice data and sends it to a server via the Internet.

[1399] Step 12:

[1400] The server analyzes the audio data and generates pronunciation evaluation and feedback.

[1401] Input: Audio data sent from the device

[1402] Output: Pronunciation evaluation results and feedback information

[1403] Specific operation: The server analyzes the audio data using a speech analysis algorithm such as the Google Cloud Speech-to-Text API, evaluates the accuracy of the pronunciation, and generates feedback such as "Excellent" based on the evaluation result and sends it to the device.

[1404] Step 13:

[1405] The device displays and provides feedback to the user

[1406] Input: Feedback information sent from the server

[1407] Output: Visual and audio feedback

[1408] Specific operation: Based on the received feedback, the device displays or speaks an evaluation result such as "Excellent" to the user. The user confirms this feedback and continues with the next learning scenario.

[1409] (Application example 1)

[1410] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] Conventional language learning systems have limited opportunities for users to experience language usage in everyday life, making it difficult to improve practical language skills. Furthermore, feedback is limited, making it difficult for users to self-correct. In particular, pronunciation evaluation and correction are not performed in real time, making efficient learning difficult.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1413] In this invention, the server includes a means for selecting and providing learning content, a means for evaluating a user's pronunciation, and a means for providing feedback. This allows users to learn a language in a virtual space where they select a shopping scenario and engage in interactive learning, creating a situation similar to real life. Furthermore, real-time pronunciation evaluation and feedback allow users to efficiently self-correct and improve their pronunciation.

[1414] The "terminal means" is an electronic device including a display device worn by a user, and has the function of exchanging information between the user and the server.

[1415] The "recognition means" has the function of recognizing and analyzing the user's voice.

[1416] The "server means" is a computer system that has the function of selecting and providing learning content.

[1417] The "VR scenario generation means" has the function of displaying the learning content provided by the server means via the display means and generating a virtual reality environment in which the user can engage in voice-based dialogue.

[1418] The "feedback means" has a function of providing the user with pronunciation evaluation results and correction instructions based on the results of the recognition means.

[1419] A "shopping scenario" is a scenario that provides a virtual shopping experience that a user can choose from for language learning.

[1420] "Interactive learning" is a method in which a user progresses through learning by interacting with a virtual character.

[1421] A "virtual space" is a virtual environment that users can experience through a VR device.

[1422] This system, which is comprised of three elements: a user, a terminal, and a server, provides an environment in which users can study a language while experiencing the sensation of actually studying abroad using virtual reality (VR). This system allows users to study a language in a virtual space where they can select a shopping scenario and engage in interactive learning, creating a situation that is close to real life.

[1423] System configuration overview

[1424] 1. User: The final user of the system, who wears the VR device and experiences the language learning content.

[1425] 2. Terminal: An electronic device including a VR device worn by the user that exchanges information between the user and the server. The terminal has the function of recording the user's pronunciation and sending it to the server.

[1426] 3. Server: A computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Specifically, it performs speech recognition and evaluation using the Python requests and speech_recognition libraries. Feedback is provided using the pyttsx3 library.

[1427] Processing flow

[1428] First-time setup

[1429] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[1430] User: Enter your account information and submit.

[1431] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[1432] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[1433] User: Record a sample voice and send it.

[1434] Server: Receives audio samples and performs initial pronunciation assessment.

[1435] Pronunciation evaluation and content provision

[1436] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1437] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[1438] Experience the learning scenario

[1439] User: Select a learning scenario (e.g., shopping).

[1440] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[1441] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[1442] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[1443] Terminal: Receives feedback and provides it to the user visually or audibly.

[1444] Specific examples

[1445] Example: Shopping Scenario

[1446] 1. User: Select the "Shopping" scenario and enter the VR space.

[1447] 2. Terminal: Generates a virtual space inside a shopping mall or store, and virtual characters (store clerks) appear.

[1448] 3. Terminal: A virtual character asks, "What are you looking for?"

[1449] 4. User: Says "Excuse me, I'm looking for an apple."

[1450] 5. Terminal: Records the user's voice and sends it to the server.

[1451] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[1452] 7. Terminal: A virtual character will say, "It's here," and simultaneously display the evaluation results.

[1453] 8. User: Follow the instructions and continue the experience.

[1454] Prompt Sentence Examples

[1455] If the user says "Excuse me, I'm looking for an apple," the prompt is:

[1456] text

[1457] Evaluate the pronunciation of "Excuse me, I'm looking for an apple" in Japanese and generate an appropriate feedback message.

[1458] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1459] Step 1:

[1460] First-time setup

[1461] Device: Put on the VR device and launch the dedicated application. The initial setup screen will prompt the user to enter their account information.

[1462] User: Enter your account information (e.g., username, email address, etc.) and submit.

[1463] Terminal input: Account information. Output: Sending account information to the server.

[1464] Server: Receives the entered account information and verifies it against the database. After authentication, it registers the user's initial settings (learning purpose, language level, etc.), configures the microphone used for voice recognition, and asks for a sample voice recording.

[1465] Server input: Account information. Output: Authentication result and instructions to register initial setting information.

[1466] Step 2:

[1467] Voice Recognition Settings

[1468] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[1469] User: Record a sample voice and send it.

[1470] Input: Microphone settings, sample audio. Output: Send sample audio data to the server.

[1471] Server: Analyzes the received sample voice data, performs an initial pronunciation evaluation, and stores the evaluation results in a database.

[1472] Server input: Sample speech data. Output: Initial pronunciation evaluation results.

[1473] Step 3:

[1474] Selection and provision of learning content

[1475] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1476] Server input: User's speech data. Output: Pronunciation evaluation results and appropriate learning content.

[1477] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[1478] Terminal input: Instruction to download learning content. Output: List of learning content.

[1479] Step 4:

[1480] Experience the learning scenario

[1481] User: Select a learning scenario (e.g., shopping).

[1482] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[1483] Terminal input: Selection of learning scenario. Output: Generation and display of virtual space.

[1484] Step 5:

[1485] Pronunciation practice and feedback

[1486] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[1487] Input on the device: User's pronunciation. Output: Sending voice data to the server.

[1488] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[1489] Server input: User's speech data. Output: Pronunciation evaluation and feedback.

[1490] Terminal: Receives feedback and provides it to the user visually or audibly.

[1491] Terminal input: Feedback data. Output: Display feedback or audio playback.

[1492] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1493] The system according to the present invention is composed of four main elements: a user, a terminal, a server, and an emotion engine. This system uses virtual reality (VR) to provide a language learning environment in which users can experience studying abroad in a virtual world. Furthermore, by recognizing the user's emotions and adapting the learning content and feedback, a more effective learning experience can be achieved. Specific embodiments for implementing the present invention are described below.

[1494] System configuration overview

[1495] User:

[1496] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[1497] Device:

[1498] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and emotional data and sending it to the server.

[1499] server:

[1500] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation and emotions.

[1501] Emotion Engine:

[1502] It has an algorithm that recognizes the user's emotions, analyzes the user's voice and facial expression data, and determines their emotional state. It also has the ability to adapt learning content and feedback according to the user's emotional state.

[1503] Explanation of program processing procedures

[1504] First-time setup

[1505] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[1506] User: Enter your account information and submit.

[1507] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and prompts the user to record a sample voice.

[1508] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[1509] User: Record a sample voice and send it.

[1510] Server: Receives audio samples and performs initial pronunciation assessment.

[1511] Pronunciation evaluation and content provision

[1512] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1513] Device: Starts downloading the learning content provided by the server.

[1514] On your device: Once the downloads are complete, they will be displayed in a list and you will be prompted to make a selection.

[1515] Experience the learning scenario

[1516] User: Selects a learning scenario (e.g., shopping).

[1517] Device: Generates and displays a VR space according to the selected scenario.

[1518] Terminal: A virtual character appears and speaks to the user.

[1519] Emotion engine: Analyzes the user's voice and facial expression data in real time to determine the user's emotional state.

[1520] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[1521] User: Says "Excuse me, I'm looking for an apple."

[1522] Terminal: Records the user's voice and facial expression data and sends them to the server.

[1523] Server: Analyzes the received voice data and evaluates the pronunciation.

[1524] Emotion engine: Analyzes the received emotion data and evaluates the user's emotional state.

[1525] Server: Generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[1526] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The feedback will also be adjusted according to the user's emotional state.

[1527] Specific examples

[1528] Example: Shopping Scenario

[1529] 1. User: Select the "Shopping" scenario and enter the VR space.

[1530] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[1531] 3. Terminal: The store clerk character asks, "What are you looking for?"

[1532] 4. User: Says "Excuse me, I'm looking for an apple."

[1533] 5. Terminal: Records the user's voice and facial expression data and sends them to the server.

[1534] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[1535] 7. Emotion Engine: Analyzes the user's facial expression data and evaluates their emotional state. For example, if the user is tired, it will provide gentle feedback.

[1536] 8. Terminal: A store clerk character will say "It's here" and simultaneously display the evaluation result. The feedback will also be adjusted according to the customer's emotional state.

[1537] 9. User: Follow the instructions and continue the experience.

[1538] By using the system of the present invention, users can efficiently learn a language while enjoying a realistic study abroad experience from the comfort of their own home. Furthermore, the emotion engine makes the learning experience more personalized, improving user motivation and learning effectiveness.

[1539] The processing flow will be explained below.

[1540] Step 1:

[1541] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[1542] Step 2:

[1543] Device: Prompt the user for account information on the initial setup screen.

[1544] Step 3:

[1545] User: Enter your account information and submit.

[1546] Step 4:

[1547] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and instructs the user to record a sample voice.

[1548] Step 5:

[1549] Device: Prompt the user to record a sample audio on the initial setup screen.

[1550] Step 6:

[1551] User: Follow the instructions to record and submit a sample audio.

[1552] Step 7:

[1553] Server: Receives audio samples and performs initial pronunciation assessment.

[1554] Step 8:

[1555] Server: Analyzes the received voice data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1556] Step 9:

[1557] Device: Starts downloading the learning content provided by the server.

[1558] Step 10:

[1559] On the device: Once the download is complete, the contents will be displayed in a list and the user will be prompted to make a selection.

[1560] Step 11:

[1561] User: Select a learning scenario (e.g., shopping).

[1562] Step 12:

[1563] Device: Generates and displays a VR space according to the selected scenario.

[1564] Step 13:

[1565] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[1566] Step 14:

[1567] Terminal: The character asks "What are you looking for?" and waits for voice input.

[1568] Step 15:

[1569] User: Says "Excuse me, I'm looking for an apple."

[1570] Step 16:

[1571] Terminal: Records the user's voice and facial expression data and sends them to the server and emotion engine.

[1572] Step 17:

[1573] Server: Analyzes the received voice data and evaluates the pronunciation.

[1574] Step 18:

[1575] Emotion engine: Analyzes received facial expression data and evaluates the user's emotional state.

[1576] Step 19:

[1577] Server: Generates feedback based on pronunciation evaluation results and emotional state.

[1578] Step 20:

[1579] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The system also adjusts the feedback content according to the user's emotional state. For example, if the user is nervous, it will provide a comment to help them relax.

[1580] Step 21:

[1581] User: Follow the instructions and continue the experience.

[1582] Step 22:

[1583] Server: Records learning progress and prepares suggestions for the next lesson.

[1584] Step 23:

[1585] On your device: Display suggestions for your next study session and prompt the user to choose their next step.

[1586] Step 24:

[1587] User: Review the proposal and decide to book your next session or start immediately.

[1588] Step 25:

[1589] Terminal: Prepares for the next session and notifies the user.

[1590] This series of processes allows users to study language efficiently in a relaxed state. The emotional engine individually optimizes the user's learning experience, which is expected to improve motivation and learning effectiveness.

[1591] Example 2

[1592] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1593] Existing language learning systems provide uniform pronunciation evaluation and feedback to users, which can lead to insufficient learning outcomes for individual users. Furthermore, feedback and content adaptation do not take into account the user's emotional state, which can lead to a decrease in motivation and learning effectiveness. Furthermore, there is a lack of effective ways to provide real-life dialogue scenarios in VR environments, making it difficult to achieve a realistic learning experience.

[1594] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1595] In this invention, the server includes a means for selecting and providing learning content, an emotion evaluation means for evaluating the user's emotional state, and an adaptation means for adapting the learning content and feedback. This allows for personalized responses that take into account the user's individual pronunciation evaluation and emotional state, improving learning effectiveness and motivation. Furthermore, realistic dialogue scenarios can be provided in a VR environment, providing a more realistic language learning experience.

[1596] The "terminal means" is a device that has a display device worn by a user and exchanges information with the server and other system elements.

[1597] A "recognition means" is a device or software that has the function of recognizing and analyzing a user's voice.

[1598] The "server means" is a computer system that selects and provides learning content and manages data in cooperation with the recognition means and other system elements.

[1599] The "VR scenario generation means" is a device or software that displays the learning content provided by the server means via the display means and generates a virtual reality scenario in which the user can engage in voice-based dialogue.

[1600] The "emotion evaluation means" is a device or software that has the function of evaluating the user's emotional state based on the results of the recognition means.

[1601] The "adaptation means" is a device or software that has the function of adapting the learning content or feedback based on the evaluation results of the emotion evaluation means.

[1602] The "feedback means" is a device or software that has a function of providing feedback to the user based on the results of the recognition means and emotion evaluation means.

[1603] The system according to the present invention is a system comprising a terminal means having a display device worn by the user, a recognition means for recognizing and analyzing the user's voice, a server means for selecting and providing learning content, a VR scenario generation means for generating a virtual reality (VR) scenario, an emotion evaluation means for evaluating the user's emotional state, an adaptation means for adapting the learning content and feedback, and a means for providing feedback.

[1604] Hardware and software used

[1605] 1. Terminal means: The display device worn by the user is a VR device such as a head-mounted display (HMD). This device is equipped with a high-performance graphics card and sensors to provide an interactive VR environment. Specifically, HMDs include "Oculus Rift" and "HTC Vive."

[1606] 2. Recognition method: Software for recognizing and analyzing the user's voice can use the voice recognition libraries "Google Speech-to-Text API" and "Microsoft Azure Speech Service."

[1607] 3. Server: The server is responsible for selecting and providing the learning content. The server is equipped with a high-performance processor and storage, and runs a database and network management system. For example, a cloud server from Amazon Web Services (AWS) can be used.

[1608] 4. VR scenario generation means: To generate a VR scenario that displays the learning content provided by the server and allows the user to interact based on voice, the game engines "Unity" and "Unreal Engine" can be used.

[1609] 5. Emotion assessment method: To assess the user's emotional state, we use emotion recognition software "Affectiva SDK" and "Microsoft Azure Emotion API" that analyze the user's voice and facial expression data.

[1610] 6. Adaptation: Software that adapts learning and feedback in real time uses algorithms running on servers that utilize machine learning and natural language processing (NLP) techniques.

[1611] 7. Feedback methods: Feedback can be provided using Text-to-Speech (TTS) systems or display pop-up technology to present feedback to the user visually or audibly.

[1612] Specific examples

[1613] Example scenario: Shopping experience

[1614] 1. User: Put on the VR device, launch the application, set up an account for the first time, and record a sample voice.

[1615] 2. Device: Once the initial setup is complete, a list of learning content provided by the server will be displayed.

[1616] 3. User: Select the "Shopping" scenario and enter the VR space.

[1617] 4. Terminal: A VR space of a shopping mall is generated, and virtual characters (store clerks) appear.

[1618] 5. Terminal: The store clerk character asks, "Welcome, what are you looking for?"

[1619] 6. User: Says, "Excuse me, I'm looking for an apple."

[1620] 7. Terminal: Records voice and facial expression data and sends it to the server.

[1621] 8. Server: Analyzes the audio and evaluates the pronunciation.

[1622] 9. Emotion Engine: Analyzes facial expression data and evaluates emotional state, for example, recognizing that the user is stressed.

[1623] 10. Server: Generates feedback based on the evaluation results and emotional state and sends it to the device.

[1624] 11. Terminal: The virtual character will say "It's here" and display the evaluation result. The evaluation result will be "Excellent," and depending on the user's emotional state, a positive message such as "Your pronunciation is really good!" will be displayed.

[1625] Prompt Sentence Examples

[1626] In a shopping scenario, describe the process flow for when a user asks a store clerk if they are looking for an apple.

[1627] This system allows users to enjoy a highly realistic learning experience and learn languages ​​more effectively through individually optimized feedback and learning content. Personalized feedback based on the user's emotional state also increases learning motivation.

[1628] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1629] Step 1:

[1630] The user puts on the VR device and launches the dedicated application.

[1631] Input: The user powers on the VR device and selects the application icon.

[1632] Specific operation: The user wears the HMD device on their head and launches an application using the controller in their hand.

[1633] Output: The application's initial setup screen is displayed, prompting the user to enter their account information.

[1634] Step 2:

[1635] The device displays the initial setup screen and prompts the user to enter their account information.

[1636] Input: The application launches and the initial setup screen appears, presenting the user with input fields such as "Name," "Email address," and "Password."

[1637] Specific behavior: The initial setup form will be displayed in the VR space.

[1638] Output: A screen is prepared for the user to enter their account information.

[1639] Step 3:

[1640] The user enters and submits their account information.

[1641] Input: The user enters the required information into the designated fields using a keyboard or voice input.

[1642] Specific behavior: The user clicks the "Submit" button.

[1643] Output: The entered account information is sent to the server.

[1644] Step 4:

[1645] The server receives the account information and performs authentication.

[1646] Input: Account information submitted by the user.

[1647] Specific operation: The server connects to the database and verifies the input information to perform authentication.

[1648] Output: The authentication result is generated (success or failure). If authentication is successful, the user's initial setup is completed.

[1649] Step 5:

[1650] The server displays a screen to guide the user through the speech recognition test and asks the user to record a sample voice.

[1651] Input: Account authentication successful, user's initial setup data.

[1652] Specific operation: The server sends a voice recognition setting screen to the device and instructs it to "record a sample voice."

[1653] Output: The voice recognition settings screen will be displayed on the device.

[1654] Step 6:

[1655] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[1656] Input: Speech recognition setting screen sent from the server.

[1657] Specific operation: Audio recording instructions will be displayed in the VR space.

[1658] Output: The user is ready to start recording.

[1659] Step 7:

[1660] User records and submits a sample voice

[1661] Input: The user reads the prompted sentence (e.g., "I want to learn Japanese").

[1662] Specific behavior: The user clicks the record button and speaks the specified sentence.

[1663] Output: The recorded audio data is stored on the device and sent to the server.

[1664] Step 8:

[1665] The server receives the sample audio and performs an initial pronunciation assessment

[1666] Input: Audio data sent by the user.

[1667] What it does: It uses a speech analysis algorithm to analyze the speech data and evaluate the accuracy of the pronunciation.

[1668] Output: The initial pronunciation evaluation result is generated.

[1669] Step 9:

[1670] The server analyzes the voice data and evaluates the user's pronunciation level.

[1671] Input: User speech data and an existing pronunciation reference model.

[1672] What it does: Uses machine learning algorithms to compare and evaluate your voice data.

[1673] Output: The user's pronunciation level evaluation result is generated.

[1674] Step 10:

[1675] The server selects appropriate learning content based on the evaluation results.

[1676] Input: Pronunciation level assessment results and user learning objectives.

[1677] Specific actions: Select appropriate learning materials from a database of learning content.

[1678] Output: Selected learning content data is generated.

[1679] Step 11:

[1680] The server sends the selected learning content to the device.

[1681] Input: Selected learning content data.

[1682] Specific operation: Divide the learning content data into packets and send them to the terminal.

[1683] Output: The device receives the learning content data.

[1684] Step 12:

[1685] The device downloads the learning content and displays it in a list.

[1686] Input: Learning content data sent from the server.

[1687] Specific behavior: Downloads content and displays a progress bar. Once completed, displays a list of learning content.

[1688] Output: A list of learning content will be displayed on the screen.

[1689] Step 13:

[1690] The user selects a learning scenario (e.g., shopping)

[1691] Input: A list of learning content.

[1692] Specific behavior: The user selects the desired scenario from a list. Example: "Shopping".

[1693] Output: The selected learning scenario is recorded on the device.

[1694] Step 14:

[1695] The device generates and displays a VR space according to the selected scenario.

[1696] Input: Data for the scenario selected by the user.

[1697] Specific operation: Use "Unity" or "Unreal Engine" to render and display the corresponding VR space.

[1698] Output: The VR space is displayed in the user's field of view.

[1699] Step 15:

[1700] The device displays a virtual character and starts a conversation

[1701] Input: The learning scenario selected by the user.

[1702] What it does: A virtual character appears in the user's field of vision, plays a preset audio file, and asks, "What are you looking for?"

[1703] Output: A virtual character begins to interact.

[1704] Step 16:

[1705] The emotion engine analyzes the user's voice and facial expression data in real time to determine their emotional state.

[1706] Input: Real-time voice and facial expression data from the user.

[1707] Specific operations: Emotion analysis is performed using the "Affectiva SDK" and "Microsoft Azure Emotion API."

[1708] Output: The user's emotional state assessment result is generated.

[1709] Step 17:

[1710] The device records the user's voice and facial expression data and sends it to the server.

[1711] Input: Speech and facial expressions made by the user.

[1712] Specific operation: Captures voice and facial expression data and sends it to the server.

[1713] Output: The recorded data is sent to the server.

[1714] Step 18:

[1715] The server analyzes the received voice data and evaluates the pronunciation

[1716] Input: User's voice data.

[1717] What it does: Uses a speech recognition algorithm to assess the accuracy of the user's pronunciation.

[1718] Output: Pronunciation evaluation results are generated.

[1719] Step 19:

[1720] The emotion engine analyzes the received emotion data and evaluates the user's emotional state.

[1721] Input: User's facial expression data.

[1722] What it does: Uses an emotion analysis algorithm to assess emotional state.

[1723] Output: The emotion evaluation result is generated.

[1724] Step 20:

[1725] The server generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[1726] Input: Pronunciation evaluation results and emotion evaluation results.

[1727] Specific operations: Generate pronunciation feedback and responses according to the emotional state and send them to the device.

[1728] Output: Feedback data is sent to the device.

[1729] Step 21:

[1730] The device controls the virtual character and provides feedback to the user.

[1731] Input: Feedback data sent by the server.

[1732] Specific behavior: A virtual character will say "It's here" and display the evaluation result. Feedback such as "Excellent" or a positive message will be displayed.

[1733] Output: A virtual character will provide guidance and feedback.

[1734] (Application example 2)

[1735] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1736] Conventional language learning systems have struggled to effectively assess users' pronunciation and comprehension and provide a personalized learning experience. Furthermore, they were unable to provide feedback based on the user's emotional state, resulting in insufficient learning effectiveness. Furthermore, it was difficult to replicate the experience of actual in-store customer service, making it difficult to improve practical language skills.

[1737] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes terminal means having a visual display device worn by the user, voice recognition means for recognizing and analyzing the user's voice, information processing device means for selecting and providing learning content, virtual space generation means for displaying the learning content provided by the information processing device means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback based on the results of the voice recognition means and the user's emotional state, and emotion analysis means for analyzing the user's emotions. This allows the user to receive personalized feedback based on the emotion analysis, enabling a practical language learning experience within a virtual store.

[1738] A "visual display device" is a device worn by a user to receive visual information and provide a virtual reality experience.

[1739] "Terminal means" refers to electronic equipment used by a user, including a visual display device, for communicating with a server to exchange information.

[1740] The "voice recognition means" is a device that has the function of recording the user's voice and analyzing that voice.

[1741] The "information processing device means" is a computer system such as a server for generating and providing learning content.

[1742] "Virtual space generating means" is a device that has the function of generating a virtual reality environment and allowing a user to interact within it.

[1743] A "feedback means" is a device that has the function of providing visual or audio feedback to a user.

[1744] "Emotion analysis means" refers to an algorithm or device that analyzes the user's voice and facial expression data and determines their emotional state.

[1745] "User" refers to the final user who uses this system to study.

[1746] The present invention provides a system that allows users to experience practical language learning in a virtual store, which includes a terminal means including a visual display device, a voice recognition means, an information processing means, a virtual space generation means, a feedback means, and an emotion analysis means.

[1747] First, the user wears a visual display device (e.g., smart glasses). The terminal generates a virtual space of a virtual store via the visual display device and displays it to the user. This virtual space generation means allows the user to experience shopping in an actual store.

[1748] The user's voice data is collected and analyzed by a voice recognition means, which may be implemented by a microphone mounted on a computer or smart glasses, and the voice commands issued by the user are collected in real time and transmitted to a server.

[1749] The server analyzes the voice data and evaluates the user's pronunciation level. At this time, an emotion analysis tool is also used based on the user's facial expressions and voice data to determine the user's emotional state. Emotion analysis uses a generative AI model incorporating facial expression recognition technology and voice emotion analysis technology.

[1750] Based on the analyzed voice data and emotion data, the information processing device generates learning content and recommendations suitable for the user. This information is transmitted to the terminal device and provided to the user through the VR space.

[1751] The feedback provided to the user is displayed visually or audibly by a feedback means, which provides personalized recommendations according to the quality of the user's pronunciation and the user's emotional state.

[1752] As a concrete example, consider the case where a user enters a virtual store in a virtual space and issues a voice command such as, "Excuse me, I'm looking for an apple." The voice is analyzed, and if the pronunciation is correct, the word "Excellent" is displayed. If the user has a relaxed expression, a virtual store clerk character praises the user's excellent pronunciation. Conversely, if the user appears tired, feedback is given in a gentler tone.

[1753] An example of a prompt sentence is, "Analyze the following user's voice data and facial expression data, and indicate the user's pronunciation evaluation and emotional state. Also, output recommendations to be provided to the user in relation to the shopping scenario. Voice data: Voice data file Facial expression data: 'smile: 0.8, anger: 0.2'."

[1754] In this way, providing real-time feedback based on the user's performance and emotional state can lead to a more effective and personalized learning experience.

[1755] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1756] Step 1:

[1757] The user puts on the visual display device and starts up the terminal.

[1758] Input: The user performs an action to start the terminal.

[1759] Output: The initial device setup screen will be displayed.

[1760] Specific operation: The terminal displays the initial screen of the virtual store via the visual display device.

[1761] Step 2:

[1762] The user enters their account information and sends it to the server.

[1763] Input: The user enters their account information and presses the "Submit" button.

[1764] Output: The server receives the account information and initializes the user profile.

[1765] Specific operation: The server performs initial settings for the user based on the received account information and sends a setup completion message to the terminal.

[1766] Step 3:

[1767] A voice sample of the user is collected and sent to a server.

[1768] Input: The user records a sample voice on the voice settings screen and presses the "Submit" button.

[1769] Output: The server receives the sample speech data and performs an initial pronunciation assessment.

[1770] How it works: The device uses a microphone to record the user's voice and sends the data to the server, which then analyzes the received voice and evaluates the pronunciation level.

[1771] Step 4:

[1772] The user's voice data and facial expression data are analyzed to evaluate their emotions.

[1773] Input: Voice data collected by speech recognition means and facial expression data collected by camera function.

[1774] Output: The server stores the user's pronunciation rating and emotion rating as analysis results.

[1775] Specific operation: The server uses a generative AI model to analyze voice data and evaluate pronunciation. At the same time, it analyzes facial expression data and evaluates the user's emotional state.

[1776] Step 5:

[1777] Generate learning content and recommendations and send them to the device.

[1778] Input: Speech and emotion data analyzed by the server.

[1779] Output: Appropriate learning content and recommendations sent to the device.

[1780] Specific operation: The server uses the information processing device means to generate learning content and feedback that is optimal for the user's learning progress and emotional state, and transmits them to the terminal.

[1781] Step 6:

[1782] The device generates a virtual space and displays the learning content.

[1783] Input: Learning content sent from the server.

[1784] Output: The learning scenario in the virtual space that is displayed to the user.

[1785] Specific operation: The terminal uses the virtual space generation means to generate a scenario for a virtual store based on the learning content received from the server, and displays it to the user via a visual display device.

[1786] Step 7:

[1787] The user interacts within the virtual space and continues the experience.

[1788] Input: An action that the user gives voice instructions for.

[1789] Output: New collected speech and facial expression data.

[1790] Specific operation: The device uses a microphone and camera to collect the user's voice and facial expression data and transmits it back to the server.

[1791] Step 8:

[1792] Provide visual or audio feedback to the user.

[1793] Input: User's voice and facial expression data reanalyzed by the server.

[1794] Output: Feedback displayed on the terminal.

[1795] Specific operation: Based on the data reanalyzed by the server, feedback that is optimal for the user's pronunciation and emotional state is generated and provided to the user via the device.

[1796] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1797] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1798] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1799] [Fourth embodiment]

[1800] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1801] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1802] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1803] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1804] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1805] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1806] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1807] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1808] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1809] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1810] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1811] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1812] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1813] The system according to the present invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the sensation of actually studying abroad. Specific embodiments for carrying out the present invention are described below.

[1814] System configuration overview

[1815] User:

[1816] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[1817] Device:

[1818] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server.

[1819] server:

[1820] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation.

[1821] Explanation of program processing procedures

[1822] First-time setup

[1823] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[1824] User: Enter your account information and submit.

[1825] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[1826] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[1827] User: Record a sample voice and send it.

[1828] Server: Receives audio samples and performs initial pronunciation assessment.

[1829] Pronunciation evaluation and content provision

[1830] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1831] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[1832] Experience the learning scenario

[1833] User: Selects a learning scenario (e.g., shopping).

[1834] Terminal: Generates and displays a VR space according to the selected scenario. A virtual character appears and speaks to the user.

[1835] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[1836] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[1837] Terminal: Receives feedback and provides it to the user visually or audibly.

[1838] Specific examples

[1839] Example: Shopping Scenario

[1840] 1. User: Select the "Shopping" scenario and enter the VR space.

[1841] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[1842] 3. Terminal: The store clerk character asks, "What are you looking for?"

[1843] 4. User: Says "Excuse me, I'm looking for an apple."

[1844] 5. Terminal: Records the user's voice and sends it to the server.

[1845] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[1846] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[1847] 8. User: Follow the instructions and continue the experience.

[1848] By using the system of the present invention, users can efficiently learn a language through a realistic study abroad experience at home. Furthermore, repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[1849] The processing flow will be explained below.

[1850] Step 1:

[1851] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[1852] Step 2:

[1853] Device: Prompt the user for account information on the initial setup screen.

[1854] Step 3:

[1855] User: Enter your account information and submit.

[1856] Step 4:

[1857] Server: Receives and authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.).

[1858] Step 5:

[1859] On the device: The initial microphone setup screen for speech recognition is displayed, prompting the user to record a sample voice.

[1860] Step 6:

[1861] User: Record a sample voice and send it.

[1862] Step 7:

[1863] Server: Receives audio samples and performs initial pronunciation assessment.

[1864] Step 8:

[1865] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[1866] Step 9:

[1867] Device: Starts downloading the learning content provided by the server.

[1868] Step 10:

[1869] Device: Once the content has finished downloading, it will be displayed in a list and the user will be prompted to make a selection.

[1870] Step 11:

[1871] User: Select a learning scenario (e.g., shopping).

[1872] Step 12:

[1873] Device: Generates and displays a VR space according to the selected scenario.

[1874] Step 13:

[1875] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[1876] Step 14:

[1877] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[1878] Step 15:

[1879] User: Says "Excuse me, I'm looking for an apple."

[1880] Step 16:

[1881] Device: Records the user's voice and sends it to the server.

[1882] Step 17:

[1883] Server: Analyzes the received voice data and evaluates the pronunciation.

[1884] Step 18:

[1885] Server: If the pronunciation of "ringo" is correct, generate feedback as "Excellent."

[1886] Step 19:

[1887] Terminal: A store clerk character will say, "It's here," and at the same time display the evaluation results.

[1888] Step 20:

[1889] User: Follow the instructions and continue the experience.

[1890] Step 21:

[1891] Server: Records learning progress data and prepares suggestions for the next learning session.

[1892] Step 22:

[1893] On your device: Show the user suggestions for their next study session.

[1894] Step 23:

[1895] User: Check out upcoming sessions and decide whether to book or start immediately.

[1896] Step 24:

[1897] Terminal: Prepares for the next session and notifies the user.

[1898] Example 1

[1899] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1900] In conventional language learning systems, users have few opportunities to practice conversations in a real language environment, which means that their pronunciation and communication skills are not fully improved. The present invention aims to solve this problem by providing a system that allows users to efficiently learn a language through a realistic study abroad experience from the comfort of their own home.

[1901] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1902] In this invention, the server includes terminal means having a display device worn by the user, recognition means for recognizing and analyzing the user's voice, server means for selecting and providing learning content, VR scenario generation means for displaying the learning content provided by the server means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback to the user based on the results of the recognition means, means for recording the user's pronunciation as voice data and transmitting it to the server, and means for selecting and downloading appropriate learning content based on the analysis results from the server. This allows the user to practice pronunciation while experiencing realistic dialogue scenes, thereby achieving effective language learning.

[1903] "User" refers to a person who actually uses the system and wears a VR device to learn a language.

[1904] The term "terminal means" refers to an electronic device including a display device worn by a user, which exchanges information between the user and the server.

[1905] "Recognition means" refers to a device or software that has the function of recognizing and analyzing the user's voice and has the role of evaluating the voice data.

[1906] "Server means" refers to a computer system that has the function of selecting and providing learning content and analyzing user voice data.

[1907] "VR scenario generation means" refers to a device or software that generates a virtual reality scenario based on learning content provided by the server means and enables the user to interact in that environment.

[1908] The "feedback means" refers to a device or software that has the function of providing the user with visual or audio feedback on the quality of their pronunciation based on the results of the recognition means.

[1909] "Means for recording as voice data" refers to a device or software that has the function of recording the user's pronunciation and transmitting the voice data to a server.

[1910] "Means for downloading" refers to a device or software that has the function of downloading learning content provided by a server and displaying it to the user.

[1911] The system related to this invention is composed of three elements: a user, a terminal, and a server. This system uses virtual reality (VR) to provide an environment in which users can study a language while experiencing the experience of actually studying abroad.

[1912] System configuration overview

[1913] User:

[1914] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[1915] Device:

[1916] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and sending it to the server. For example, a display device such as Oculus Quest 2 is used.

[1917] server:

[1918] This is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Google Cloud Speech-to-Text API and other services are used for speech analysis.

[1919] Program Description

[1920] The user wearing the VR device launches a dedicated application and enters their account information to begin the initial setup. The server authenticates the user's account information and instructs them to configure the microphone used for voice recognition and record a sample voice. When the user records and sends the sample voice, the server analyzes the voice, performs an initial pronunciation evaluation, and provides learning content.

[1921] When a user selects a learning scenario, the device generates a corresponding VR space in which a virtual character speaks to the user. The user interacts with the character and practices pronunciation. After practice, the audio data is sent to the server for analysis and feedback. Feedback is provided to the user visually or audibly via the device.

[1922] Specific examples

[1923] Example of a shopping scenario

[1924] 1. User: Select the "Shopping" scenario and enter the VR space.

[1925] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[1926] 3. Terminal: The store clerk character asks, "What are you looking for?"

[1927] 4. User: Says "Excuse me, I'm looking for an apple."

[1928] 5. Terminal: Records the user's voice and sends it to the server.

[1929] 6. Server: Analyzes the audio and if the pronunciation of "apple" is correct, rates it as "Excellent."

[1930] 7. Terminal: The store clerk character will say, "It's here," and at the same time display the evaluation results.

[1931] 8. User: Follow the instructions and continue the experience.

[1932] This system allows users to study a language efficiently while experiencing a realistic study abroad experience from home. Repeated practice based on feedback also promotes the acquisition of native-like pronunciation.

[1933] Prompt Sentence Examples

[1934] "Users experiencing virtual study abroad interact with a virtual store clerk in a shopping scenario and receive pronunciation evaluation and feedback."

[1935] The user puts on the VR device and selects the "Shopping" scenario. The device generates a VR space inside the store, and a store clerk character appears and asks, "What are you looking for?" The user says, "Excuse me, I'm looking for an apple," and the device records the speech and sends it to the server. The server analyzes the speech, and if the pronunciation is correct, it rates it as "Excellent," and the device displays the result. The user then continues the next dialogue.

[1936] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1937] Step 1:

[1938] The user puts on the VR device and launches the dedicated application.

[1939] Input: The user puts on the VR device and launches the application.

[1940] Output: The application starts and the initial setup screen is displayed.

[1941] Specific operation: The user wears a head-mounted display (e.g., Oculus Quest 2) and uses the controller to select and launch the dedicated application. When the application is launched, the device displays the initial setup screen.

[1942] Step 2:

[1943] The user enters and submits their account information.

[1944] Input: Enter your account information in the text boxes displayed on the initial setup screen.

[1945] Output: The entered account information is sent to the server.

[1946] Specific operation: The user enters account information such as email address, password, and username in the text boxes provided and presses the "Send" button. The device then sends this information to the server.

[1947] Step 3:

[1948] The server authenticates the account information and registers the initial settings.

[1949] Input: Account information sent from the device

[1950] Output: Authentication result and initial setting registration completion message

[1951] Specific operation: The server checks the received account information against the database and performs authentication. If authentication is successful, it registers initial information such as the user's learning goals and language level, and returns the results to the terminal.

[1952] Step 4:

[1953] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[1954] Input: Authentication result and initial setting information returned from the server

[1955] Output: Sample audio recording screen display

[1956] Specific operation: After receiving the authentication result, the device displays a screen prompting the user to set up voice recognition, asking them to say a sample voice such as "Hello."

[1957] Step 5:

[1958] User records and submits a sample voice

[1959] Input: A sample voice utterance from the user

[1960] Output: Recorded audio data is sent to the server

[1961] Specific operation: The user speaks a sample voice according to the instructions on the device. The device records the voice and sends it to the server as voice data.

[1962] Step 6:

[1963] The server analyzes the sample audio and makes an initial pronunciation assessment

[1964] Input: Sample audio data sent from the device

[1965] Output: Initial pronunciation assessment result and a message that the learning content is ready to be provided

[1966] Specific operation: The server uses a speech analysis algorithm such as the Google Cloud Speech-to-Text API to analyze the sample speech and evaluate the user's pronunciation level. Based on the evaluation results, it prepares appropriate learning content and sends a message to the device indicating that it is ready to be provided.

[1967] Step 7:

[1968] The device downloads the learning content and displays it in a list.

[1969] Input: Message from the server that learning content is ready to be provided

[1970] Output: Download and list of learning content

[1971] Specific operation: The device receives instructions from the server, downloads the suggested learning content, displays a list of downloaded learning content, and prompts the user to select one.

[1972] Step 8:

[1973] The user selects a learning scenario

[1974] Input: User selects from a list of learning content

[1975] Output: The selected learning scenario is displayed on the device.

[1976] Specific operation: The user selects a learning scenario, such as "Shopping," from the list of learning content displayed on the device and presses the "Start" button. The selected scenario is reflected on the device.

[1977] Step 9:

[1978] The device generates and displays the VR space.

[1979] Input: Information about the learning scenario selected by the user

[1980] Output: Display of VR space and virtual characters based on the selected scenario

[1981] Specific operation: The device generates and displays an appropriate VR space (e.g., a shopping mall) based on the information of the selected scenario, and a virtual character (e.g., a store clerk) also appears.

[1982] Step 10:

[1983] Users interact with virtual characters and practice their pronunciation

[1984] Input: Question from virtual character and user's response utterance

[1985] Output: Recorded user voice data

[1986] Specific operation: The virtual character asks a question such as "What are you looking for?", and the user responds with a voice such as "Excuse me, I'm looking for an apple." The device records this voice.

[1987] Step 11:

[1988] The device sends the recorded audio data to the server.

[1989] Input: User pronunciation recording

[1990] Output: Recorded audio data is sent to the server

[1991] Specific operation: The device compresses the recorded user voice data and sends it to a server via the Internet.

[1992] Step 12:

[1993] The server analyzes the audio data and generates pronunciation evaluation and feedback.

[1994] Input: Audio data sent from the device

[1995] Output: Pronunciation evaluation results and feedback information

[1996] Specific operation: The server analyzes the audio data using a speech analysis algorithm such as the Google Cloud Speech-to-Text API, evaluates the accuracy of the pronunciation, and generates feedback such as "Excellent" based on the evaluation result and sends it to the device.

[1997] Step 13:

[1998] The device displays and provides feedback to the user

[1999] Input: Feedback information sent from the server

[2000] Output: Visual and audio feedback

[2001] Specific operation: Based on the received feedback, the device displays or speaks an evaluation result such as "Excellent" to the user. The user confirms this feedback and continues with the next learning scenario.

[2002] (Application example 1)

[2003] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2004] Conventional language learning systems have limited opportunities for users to experience language use in everyday life, making it difficult to improve practical language skills. Furthermore, feedback is limited, making it difficult for users to self-correct. In particular, pronunciation evaluation and correction are not performed in real time, making efficient learning difficult.

[2005] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2006] In this invention, the server includes a means for selecting and providing learning content, a means for evaluating a user's pronunciation, and a means for providing feedback. This allows users to learn a language in a virtual space where they select a shopping scenario and engage in interactive learning, creating a situation similar to real life. Furthermore, real-time pronunciation evaluation and feedback allow users to efficiently self-correct and improve their pronunciation.

[2007] The "terminal means" is an electronic device including a display device worn by a user, and has the function of exchanging information between the user and the server.

[2008] The "recognition means" has the function of recognizing and analyzing the user's voice.

[2009] The "server means" is a computer system that has the function of selecting and providing learning content.

[2010] The "VR scenario generation means" has the function of displaying the learning content provided by the server means via the display means and generating a virtual reality environment in which the user can engage in voice-based dialogue.

[2011] The "feedback means" has a function of providing the user with pronunciation evaluation results and correction instructions based on the results of the recognition means.

[2012] A "shopping scenario" is a scenario that provides a virtual shopping experience that a user can choose from for language learning.

[2013] "Interactive learning" is a method in which a user progresses through learning by interacting with a virtual character.

[2014] A "virtual space" is a virtual environment that users can experience through a VR device.

[2015] This system, which is comprised of three elements: a user, a terminal, and a server, provides an environment in which users can study a language while experiencing the sensation of actually studying abroad using virtual reality (VR). This system allows users to study a language in a virtual space where they can select a shopping scenario and engage in interactive learning, creating a situation that is close to real life.

[2016] System configuration overview

[2017] 1. User: The final user of the system, who wears the VR device and experiences the language learning content.

[2018] 2. Terminal: An electronic device including a VR device worn by the user that exchanges information between the user and the server. The terminal has the function of recording the user's pronunciation and sending it to the server.

[2019] 3. Server: A computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation. Specifically, it performs speech recognition and evaluation using the Python requests and speech_recognition libraries. Feedback is provided using the pyttsx3 library.

[2020] Processing flow

[2021] First-time setup

[2022] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[2023] User: Enter your account information and submit.

[2024] Server: Authenticates the account information and registers the user's initial settings (learning objectives, language level, etc.). As initial settings, it instructs the microphone settings to be used for voice recognition and the recording of a sample voice.

[2025] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[2026] User: Record a sample voice and send it.

[2027] Server: Receives audio samples and performs initial pronunciation assessment.

[2028] Pronunciation evaluation and content provision

[2029] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[2030] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[2031] Experience the learning scenario

[2032] User: Select a learning scenario (e.g., shopping).

[2033] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[2034] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[2035] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[2036] Terminal: Receives feedback and provides it to the user visually or audibly.

[2037] Specific examples

[2038] Example: Shopping Scenario

[2039] 1. User: Select the "Shopping" scenario and enter the VR space.

[2040] 2. Terminal: Generates a virtual space inside a shopping mall or store, and virtual characters (store clerks) appear.

[2041] 3. Terminal: A virtual character asks, "What are you looking for?"

[2042] 4. User: Says "Excuse me, I'm looking for an apple."

[2043] 5. Terminal: Records the user's voice and sends it to the server.

[2044] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[2045] 7. Terminal: A virtual character will say, "It's here," and simultaneously display the evaluation results.

[2046] 8. User: Follow the instructions and continue the experience.

[2047] Prompt Sentence Examples

[2048] If the user says "Excuse me, I'm looking for an apple," the prompt is:

[2049] text

[2050] Evaluate the pronunciation of "Excuse me, I'm looking for an apple" in Japanese and generate an appropriate feedback message.

[2051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2052] Step 1:

[2053] First-time setup

[2054] Device: Put on the VR device and launch the dedicated application. The initial setup screen will prompt the user to enter their account information.

[2055] User: Enter your account information (e.g., username, email address, etc.) and submit.

[2056] Terminal input: Account information. Output: Sending account information to the server.

[2057] Server: Receives the entered account information and verifies it against the database. After authentication, it registers the user's initial settings (learning purpose, language level, etc.), configures the microphone used for voice recognition, and asks for a sample voice recording.

[2058] Server input: Account information. Output: Authentication result and instructions to register initial setting information.

[2059] Step 2:

[2060] Voice Recognition Settings

[2061] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[2062] User: Record a sample voice and send it.

[2063] Input: Microphone settings, sample audio. Output: Send sample audio data to the server.

[2064] Server: Analyzes the received sample voice data, performs an initial pronunciation evaluation, and stores the evaluation results in a database.

[2065] Server input: Sample speech data. Output: Initial pronunciation evaluation results.

[2066] Step 3:

[2067] Selection and provision of learning content

[2068] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[2069] Server input: User's speech data. Output: Pronunciation evaluation results and appropriate learning content.

[2070] Device: Receives instructions to download learning content provided by the server, and executes the download. Once the download is complete, it displays a list of the content and prompts the user to select it.

[2071] Terminal input: Instruction to download learning content. Output: List of learning content.

[2072] Step 4:

[2073] Experience the learning scenario

[2074] User: Select a learning scenario (e.g., shopping).

[2075] Terminal: Generates and displays a virtual space according to the selected scenario. Virtual characters appear and talk to the user.

[2076] Terminal input: Selection of learning scenario. Output: Generation and display of virtual space.

[2077] Step 5:

[2078] Pronunciation practice and feedback

[2079] User: Interacts with the character and practices pronunciation. After pronunciation, the voice data is sent to the server.

[2080] Input on the device: User's pronunciation. Output: Sending voice data to the server.

[2081] Server: Analyzes the audio, evaluates the pronunciation, generates feedback, and sends it to the device.

[2082] Server input: User's speech data. Output: Pronunciation evaluation and feedback.

[2083] Terminal: Receives feedback and provides it to the user visually or audibly.

[2084] Terminal input: Feedback data. Output: Display feedback or audio playback.

[2085] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2086] The system according to the present invention is composed of four main elements: a user, a terminal, a server, and an emotion engine. This system uses virtual reality (VR) to provide a language learning environment in which users can experience studying abroad in a virtual world. Furthermore, by recognizing the user's emotions and adapting the learning content and feedback, a more effective learning experience can be achieved. Specific embodiments for implementing the present invention are described below.

[2087] System configuration overview

[2088] User:

[2089] This refers to the final user of the system, who wears the VR device and experiences the language learning content.

[2090] Device:

[2091] It is an electronic device that includes a VR device worn by the user, and exchanges information between the user and the server. The device has the function of recording the user's pronunciation and emotional data and sending it to the server.

[2092] server:

[2093] The server is a computer system that receives and analyzes data sent from the device. The server generates and provides learning content and evaluates the user's pronunciation and emotions.

[2094] Emotion Engine:

[2095] It has an algorithm that recognizes the user's emotions, analyzes the user's voice and facial expression data, and determines their emotional state. It also has the ability to adapt learning content and feedback according to the user's emotional state.

[2096] Explanation of program processing procedures

[2097] First-time setup

[2098] Device: The user puts on the VR device and launches the dedicated application. The initial setup screen appears, prompting the user to enter their account information.

[2099] User: Enter your account information and submit.

[2100] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and prompts the user to record a sample voice.

[2101] Device: Test speech recognition on the initial setup screen and prompt the user to record a sample voice.

[2102] User: Record a sample voice and send it.

[2103] Server: Receives audio samples and performs initial pronunciation assessment.

[2104] Pronunciation evaluation and content provision

[2105] Server: Analyzes the audio data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[2106] Device: Starts downloading the learning content provided by the server.

[2107] On your device: Once the downloads are complete, they will be displayed in a list and you will be prompted to make a selection.

[2108] Experience the learning scenario

[2109] User: Selects a learning scenario (e.g., shopping).

[2110] Device: Generates and displays a VR space according to the selected scenario.

[2111] Terminal: A virtual character appears and speaks to the user.

[2112] Emotion engine: Analyzes the user's voice and facial expression data in real time to determine the user's emotional state.

[2113] Terminal: The character asks "What are you looking for?" and waits for user voice input.

[2114] User: Says "Excuse me, I'm looking for an apple."

[2115] Terminal: Records the user's voice and facial expression data and sends them to the server.

[2116] Server: Analyzes the received voice data and evaluates the pronunciation.

[2117] Emotion engine: Analyzes the received emotion data and evaluates the user's emotional state.

[2118] Server: Generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[2119] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The feedback will also be adjusted according to the user's emotional state.

[2120] Specific examples

[2121] Example: Shopping Scenario

[2122] 1. User: Select the "Shopping" scenario and enter the VR space.

[2123] 2. Terminal: A VR space inside the store is generated, and a virtual character (store clerk) appears.

[2124] 3. Terminal: The store clerk character asks, "What are you looking for?"

[2125] 4. User: Says "Excuse me, I'm looking for an apple."

[2126] 5. Terminal: Records the user's voice and facial expression data and sends them to the server.

[2127] 6. Server: Analyzes the audio and evaluates the pronunciation. If the pronunciation of "apple" is correct, it is evaluated as "Excellent."

[2128] 7. Emotion Engine: Analyzes the user's facial expression data and evaluates their emotional state. For example, if the user is tired, it will provide gentle feedback.

[2129] 8. Terminal: A store clerk character will say "It's here" and simultaneously display the evaluation results. The feedback will also be adjusted according to the customer's emotional state.

[2130] 9. User: Follow the instructions and continue the experience.

[2131] By using the system of the present invention, users can efficiently learn a language while enjoying a realistic study abroad experience from the comfort of their own home. Furthermore, the emotion engine makes the learning experience more personalized, improving user motivation and learning effectiveness.

[2132] The processing flow will be explained below.

[2133] Step 1:

[2134] User: Put on the VR device and launch the dedicated application. The initial setup screen will be displayed.

[2135] Step 2:

[2136] Device: Prompt the user for account information on the initial setup screen.

[2137] Step 3:

[2138] User: Enter your account information and submit.

[2139] Step 4:

[2140] Server: Receives and authenticates the account information, and registers the user's initial settings (learning objectives, language level, etc.). As part of the initial settings, displays the microphone settings screen to be used for voice recognition, and instructs the user to record a sample voice.

[2141] Step 5:

[2142] Device: Prompt the user to record a sample audio on the initial setup screen.

[2143] Step 6:

[2144] User: Follow the instructions to record and submit a sample audio.

[2145] Step 7:

[2146] Server: Receives audio samples and performs initial pronunciation assessment.

[2147] Step 8:

[2148] Server: Analyzes the received voice data and evaluates the user's pronunciation level. Based on the evaluation results, it selects appropriate learning content.

[2149] Step 9:

[2150] Device: Starts downloading the learning content provided by the server.

[2151] Step 10:

[2152] On the device: Once the download is complete, the contents will be displayed in a list and the user will be prompted to make a selection.

[2153] Step 11:

[2154] User: Select a learning scenario (e.g., shopping).

[2155] Step 12:

[2156] Device: Generates and displays a VR space according to the selected scenario.

[2157] Step 13:

[2158] Terminal: A virtual character (e.g., a store clerk) appears and speaks to the user.

[2159] Step 14:

[2160] Terminal: The character asks "What are you looking for?" and waits for voice input.

[2161] Step 15:

[2162] User: Says "Excuse me, I'm looking for an apple."

[2163] Step 16:

[2164] Terminal: Records the user's voice and facial expression data and sends them to the server and emotion engine.

[2165] Step 17:

[2166] Server: Analyzes the received voice data and evaluates the pronunciation.

[2167] Step 18:

[2168] Emotion engine: Analyzes received facial expression data and evaluates the user's emotional state.

[2169] Step 19:

[2170] Server: Generates feedback based on pronunciation evaluation results and emotional state.

[2171] Step 20:

[2172] Terminal: A store clerk character will say, "It's here," and simultaneously display the evaluation results. The system also adjusts the feedback content according to the user's emotional state. For example, if the user is nervous, it will provide a comment to help them relax.

[2173] Step 21:

[2174] User: Follow the instructions and continue the experience.

[2175] Step 22:

[2176] Server: Records learning progress and prepares suggestions for the next lesson.

[2177] Step 23:

[2178] On your device: Display suggestions for your next study session and prompt the user to choose their next step.

[2179] Step 24:

[2180] User: Review the proposal and decide to book your next session or start immediately.

[2181] Step 25:

[2182] Terminal: Prepares for the next session and notifies the user.

[2183] This series of processes allows users to study language efficiently in a relaxed state. The emotional engine individually optimizes the user's learning experience, which is expected to improve motivation and learning effectiveness.

[2184] Example 2

[2185] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2186] Existing language learning systems provide uniform pronunciation evaluation and feedback to users, which can lead to insufficient learning outcomes for individual users. Furthermore, feedback and content adaptation do not take into account the user's emotional state, which can lead to a decrease in motivation and learning effectiveness. Furthermore, there is a lack of effective ways to provide real-life dialogue scenarios in VR environments, making it difficult to achieve a realistic learning experience.

[2187] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2188] In this invention, the server includes a means for selecting and providing learning content, an emotion evaluation means for evaluating the user's emotional state, and an adaptation means for adapting the learning content and feedback. This allows for personalized responses that take into account the user's individual pronunciation evaluation and emotional state, improving learning effectiveness and motivation. Furthermore, realistic dialogue scenarios can be provided in a VR environment, providing a more realistic language learning experience.

[2189] The "terminal means" is a device that has a display device worn by a user and exchanges information with the server and other system elements.

[2190] A "recognition means" is a device or software that has the function of recognizing and analyzing a user's voice.

[2191] The "server means" is a computer system that selects and provides learning content and manages data in cooperation with the recognition means and other system elements.

[2192] The "VR scenario generation means" is a device or software that displays the learning content provided by the server means via the display means and generates a virtual reality scenario in which the user can engage in voice-based dialogue.

[2193] The "emotion evaluation means" is a device or software that has the function of evaluating the user's emotional state based on the results of the recognition means.

[2194] The "adaptation means" is a device or software that has the function of adapting the learning content or feedback based on the evaluation results of the emotion evaluation means.

[2195] The "feedback means" is a device or software that has a function of providing feedback to the user based on the results of the recognition means and emotion evaluation means.

[2196] The system according to the present invention is a system comprising a terminal means having a display device worn by the user, a recognition means for recognizing and analyzing the user's voice, a server means for selecting and providing learning content, a VR scenario generation means for generating a virtual reality (VR) scenario, an emotion evaluation means for evaluating the user's emotional state, an adaptation means for adapting the learning content and feedback, and a means for providing feedback.

[2197] Hardware and software used

[2198] 1. Terminal means: The display device worn by the user is a VR device such as a head-mounted display (HMD). This device is equipped with a high-performance graphics card and sensors to provide an interactive VR environment. Specifically, HMDs include "Oculus Rift" and "HTC Vive."

[2199] 2. Recognition method: Software for recognizing and analyzing the user's voice can use the voice recognition libraries "Google Speech-to-Text API" and "Microsoft Azure Speech Service."

[2200] 3. Server: The server is responsible for selecting and providing the learning content. The server is equipped with a high-performance processor and storage, and runs a database and network management system. For example, a cloud server from Amazon Web Services (AWS) can be used.

[2201] 4. VR scenario generation means: To generate a VR scenario that displays the learning content provided by the server and allows the user to interact based on voice, the game engines "Unity" and "Unreal Engine" can be used.

[2202] 5. Emotion assessment method: To assess the user's emotional state, we use emotion recognition software "Affectiva SDK" and "Microsoft Azure Emotion API" that analyze the user's voice and facial expression data.

[2203] 6. Adaptation: Software that adapts learning and feedback in real time uses algorithms running on servers that utilize machine learning and natural language processing (NLP) techniques.

[2204] 7. Feedback methods: Feedback can be provided using Text-to-Speech (TTS) systems or display pop-up technology to present feedback to the user visually or audibly.

[2205] Specific examples

[2206] Example scenario: Shopping experience

[2207] 1. User: Put on the VR device, launch the application, set up an account for the first time, and record a sample voice.

[2208] 2. Device: Once the initial setup is complete, a list of learning content provided by the server will be displayed.

[2209] 3. User: Select the "Shopping" scenario and enter the VR space.

[2210] 4. Terminal: A VR space of a shopping mall is generated, and virtual characters (store clerks) appear.

[2211] 5. Terminal: The store clerk character asks, "Welcome, what are you looking for?"

[2212] 6. User: Says, "Excuse me, I'm looking for an apple."

[2213] 7. Terminal: Records voice and facial expression data and sends it to the server.

[2214] 8. Server: Analyzes the audio and evaluates the pronunciation.

[2215] 9. Emotion Engine: Analyzes facial expression data and evaluates emotional state, for example, recognizing that the user is stressed.

[2216] 10. Server: Generates feedback based on the evaluation results and emotional state and sends it to the device.

[2217] 11. Terminal: The virtual character will say "It's here" and display the evaluation result. The evaluation result will be "Excellent," and depending on the user's emotional state, a positive message such as "Your pronunciation is really good!" will be displayed.

[2218] Prompt Sentence Examples

[2219] In a shopping scenario, describe the process flow for when a user asks a store clerk if they are looking for an apple.

[2220] This system allows users to enjoy a highly realistic learning experience and learn languages ​​more effectively through individually optimized feedback and learning content. Personalized feedback based on the user's emotional state also increases learning motivation.

[2221] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2222] Step 1:

[2223] The user puts on the VR device and launches the dedicated application.

[2224] Input: The user powers on the VR device and selects the application icon.

[2225] Specific operation: The user wears the HMD device on their head and launches an application using the controller in their hand.

[2226] Output: The application's initial setup screen is displayed, prompting the user to enter their account information.

[2227] Step 2:

[2228] The device displays the initial setup screen and prompts the user to enter their account information.

[2229] Input: The application launches and the initial setup screen appears, presenting the user with input fields such as "Name," "Email address," and "Password."

[2230] Specific behavior: The initial setup form will be displayed in the VR space.

[2231] Output: A screen is prepared for the user to enter their account information.

[2232] Step 3:

[2233] The user enters and submits their account information.

[2234] Input: The user enters the required information into the designated fields using a keyboard or voice input.

[2235] Specific behavior: The user clicks the "Submit" button.

[2236] Output: The entered account information is sent to the server.

[2237] Step 4:

[2238] The server receives the account information and performs authentication.

[2239] Input: Account information submitted by the user.

[2240] Specific operation: The server connects to the database and verifies the input information to perform authentication.

[2241] Output: The authentication result is generated (success or failure). If authentication is successful, the user's initial setup is completed.

[2242] Step 5:

[2243] The server displays a screen to guide the user through the speech recognition test and asks the user to record a sample voice.

[2244] Input: Account authentication successful, user's initial setup data.

[2245] Specific operation: The server sends a voice recognition setting screen to the device and instructs it to "record a sample voice."

[2246] Output: The voice recognition settings screen will be displayed on the device.

[2247] Step 6:

[2248] The device displays the voice recognition setup screen and prompts the user to record a sample voice.

[2249] Input: Speech recognition setting screen sent from the server.

[2250] Specific operation: Audio recording instructions will be displayed in the VR space.

[2251] Output: The user is ready to start recording.

[2252] Step 7:

[2253] User records and submits a sample voice

[2254] Input: The user reads the prompted sentence (e.g., "I want to learn Japanese").

[2255] Specific behavior: The user clicks the record button and speaks the specified sentence.

[2256] Output: The recorded audio data is stored on the device and sent to the server.

[2257] Step 8:

[2258] The server receives the sample audio and performs an initial pronunciation assessment

[2259] Input: Audio data sent by the user.

[2260] What it does: It uses a speech analysis algorithm to analyze the speech data and evaluate the accuracy of the pronunciation.

[2261] Output: The initial pronunciation evaluation result is generated.

[2262] Step 9:

[2263] The server analyzes the voice data and evaluates the user's pronunciation level.

[2264] Input: User speech data and an existing pronunciation reference model.

[2265] What it does: It uses machine learning algorithms to compare and evaluate your voice data.

[2266] Output: The user's pronunciation level evaluation result is generated.

[2267] Step 10:

[2268] The server selects appropriate learning content based on the evaluation results.

[2269] Input: Pronunciation level assessment results and user learning objectives.

[2270] Specific actions: Select appropriate learning materials from a database of learning content.

[2271] Output: Selected learning content data is generated.

[2272] Step 11:

[2273] The server sends the selected learning content to the device.

[2274] Input: Selected learning content data.

[2275] Specific operation: Divide the learning content data into packets and send them to the terminal.

[2276] Output: The device receives the learning content data.

[2277] Step 12:

[2278] The device downloads the learning content and displays it in a list.

[2279] Input: Learning content data sent from the server.

[2280] Specific behavior: Downloads content and displays a progress bar. Once completed, displays a list of learning content.

[2281] Output: A list of learning content will be displayed on the screen.

[2282] Step 13:

[2283] The user selects a learning scenario (e.g., shopping)

[2284] Input: A list of learning content.

[2285] Specific behavior: The user selects the desired scenario from a list. Example: "Shopping".

[2286] Output: The selected learning scenario is recorded on the device.

[2287] Step 14:

[2288] The device generates and displays a VR space according to the selected scenario.

[2289] Input: Data for the scenario selected by the user.

[2290] Specific operation: Use "Unity" or "Unreal Engine" to render and display the corresponding VR space.

[2291] Output: The VR space is displayed in the user's field of view.

[2292] Step 15:

[2293] The device displays a virtual character and starts a conversation

[2294] Input: The learning scenario selected by the user.

[2295] What it does: A virtual character appears in the user's field of vision, plays a preset audio file, and asks, "What are you looking for?"

[2296] Output: A virtual character begins to interact.

[2297] Step 16:

[2298] The emotion engine analyzes the user's voice and facial expression data in real time to determine their emotional state.

[2299] Input: Real-time voice and facial expression data from the user.

[2300] Specific operations: Emotion analysis is performed using the "Affectiva SDK" and "Microsoft Azure Emotion API."

[2301] Output: The user's emotional state assessment result is generated.

[2302] Step 17:

[2303] The device records the user's voice and facial expression data and sends it to the server.

[2304] Input: Speech and facial expressions made by the user.

[2305] Specific operation: Captures voice and facial expression data and sends it to the server.

[2306] Output: The recorded data is sent to the server.

[2307] Step 18:

[2308] The server analyzes the received voice data and evaluates the pronunciation

[2309] Input: User's voice data.

[2310] What it does: Uses a speech recognition algorithm to assess the accuracy of the user's pronunciation.

[2311] Output: Pronunciation evaluation results are generated.

[2312] Step 19:

[2313] The emotion engine analyzes the received emotion data and evaluates the user's emotional state.

[2314] Input: User's facial expression data.

[2315] What it does: Uses an emotion analysis algorithm to assess emotional state.

[2316] Output: The emotion evaluation result is generated.

[2317] Step 20:

[2318] The server generates feedback based on the pronunciation evaluation results and emotional state and sends it to the device.

[2319] Input: Pronunciation evaluation results and emotion evaluation results.

[2320] Specific operations: Generate pronunciation feedback and responses according to the emotional state and send them to the device.

[2321] Output: Feedback data is sent to the device.

[2322] Step 21:

[2323] The device controls the virtual character and provides feedback to the user.

[2324] Input: Feedback data sent by the server.

[2325] Specific behavior: A virtual character will say "It's here" and display the evaluation result. Feedback such as "Excellent" or a positive message will be displayed.

[2326] Output: A virtual character will provide guidance and feedback.

[2327] (Application example 2)

[2328] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2329] Conventional language learning systems have struggled to effectively assess users' pronunciation and comprehension and provide a personalized learning experience. Furthermore, they were unable to provide feedback based on the user's emotional state, resulting in insufficient learning effectiveness. Furthermore, it was difficult to replicate the experience of actual in-store customer service, making it difficult to improve practical language skills.

[2330] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes terminal means having a visual display device worn by the user, voice recognition means for recognizing and analyzing the user's voice, information processing device means for selecting and providing learning content, virtual space generation means for displaying the learning content provided by the information processing device means on a display means and allowing the user to engage in voice-based dialogue, feedback means for providing feedback based on the results of the voice recognition means and the user's emotional state, and emotion analysis means for analyzing the user's emotions. This allows the user to receive personalized feedback based on the emotion analysis, enabling a practical language learning experience within a virtual store.

[2331] A "visual display device" is a device worn by a user to receive visual information and provide a virtual reality experience.

[2332] "Terminal means" refers to electronic equipment used by a user, including a visual display device, for communicating with a server to exchange information.

[2333] The "voice recognition means" is a device that has the function of recording the user's voice and analyzing that voice.

[2334] The "information processing device means" is a computer system such as a server for generating and providing learning content.

[2335] "Virtual space generating means" is a device that has the function of generating a virtual reality environment and allowing a user to interact within it.

[2336] A "feedback means" is a device that has the function of providing visual or audio feedback to a user.

[2337] "Emotion analysis means" refers to an algorithm or device that analyzes the user's voice and facial expression data and determines their emotional state.

[2338] "User" refers to the final user who uses this system to study.

[2339] The present invention provides a system that allows users to experience practical language learning in a virtual store, which includes a terminal means including a visual display device, a voice recognition means, an information processing means, a virtual space generation means, a feedback means, and an emotion analysis means.

[2340] First, the user wears a visual display device (e.g., smart glasses). The terminal generates a virtual space of a virtual store via the visual display device and displays it to the user. This virtual space generation means allows the user to experience shopping in an actual store.

[2341] The user's voice data is collected and analyzed by a voice recognition means, which may be implemented by a microphone mounted on a computer or smart glasses, and the voice commands issued by the user are collected in real time and transmitted to a server.

[2342] The server analyzes the voice data and evaluates the user's pronunciation level. At this time, an emotion analysis tool is also used based on the user's facial expressions and voice data to determine the user's emotional state. Emotion analysis uses a generative AI model incorporating facial expression recognition technology and voice emotion analysis technology.

[2343] Based on the analyzed voice data and emotion data, the information processing device generates learning content and recommendations suitable for the user. This information is transmitted to the terminal device and provided to the user through the VR space.

[2344] The feedback provided to the user is displayed visually or audibly by a feedback means, which provides personalized recommendations according to the quality of the user's pronunciation and the user's emotional state.

[2345] As a concrete example, consider the case where a user enters a virtual store in a virtual space and issues a voice command such as, "Excuse me, I'm looking for an apple." The voice is analyzed, and if the pronunciation is correct, the word "Excellent" is displayed. If the user has a relaxed expression, a virtual store clerk character praises the user's excellent pronunciation. Conversely, if the user appears tired, feedback is given in a gentler tone.

[2346] An example of a prompt sentence is, "Analyze the following user's voice data and facial expression data, and indicate the user's pronunciation evaluation and emotional state. Also, output recommendations to be provided to the user in relation to the shopping scenario. Voice data: Voice data file Facial expression data: 'smile: 0.8, anger: 0.2'."

[2347] In this way, providing real-time feedback based on the user's performance and emotional state can lead to a more effective and personalized learning experience.

[2348] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2349] Step 1:

[2350] The user puts on the visual display device and starts up the terminal.

[2351] Input: The user performs an action to start the terminal.

[2352] Output: The initial device setup screen will be displayed.

[2353] Specific operation: The terminal displays the initial screen of the virtual store via the visual display device.

[2354] Step 2:

[2355] The user enters their account information and sends it to the server.

[2356] Input: The user enters their account information and presses the "Submit" button.

[2357] Output: The server receives the account information and initializes the user profile.

[2358] Specific operation: The server performs initial settings for the user based on the received account information and sends a setup completion message to the terminal.

[2359] Step 3:

[2360] A voice sample of the user is collected and sent to a server.

[2361] Input: The user records a sample voice on the voice settings screen and presses the "Submit" button.

[2362] Output: The server receives the sample speech data and performs an initial pronunciation assessment.

[2363] How it works: The device uses a microphone to record the user's voice and sends the data to the server, which then analyzes the received voice and evaluates the pronunciation level.

[2364] Step 4:

[2365] The user's voice data and facial expression data are analyzed to evaluate their emotions.

[2366] Input: Voice data collected by speech recognition means and facial expression data collected by camera function.

[2367] Output: The server stores the user's pronunciation rating and emotion rating as analysis results.

[2368] Specific operation: The server uses a generative AI model to analyze voice data and evaluate pronunciation. At the same time, it analyzes facial expression data and evaluates the user's emotional state.

[2369] Step 5:

[2370] Generate learning content and recommendations and send them to the device.

[2371] Input: Speech and emotion data analyzed by the server.

[2372] Output: Appropriate learning content and recommendations sent to the device.

[2373] Specific operation: The server uses the information processing device means to generate learning content and feedback that is optimal for the user's learning progress and emotional state, and transmits them to the terminal.

[2374] Step 6:

[2375] The device generates a virtual space and displays the learning content.

[2376] Input: Learning content sent from the server.

[2377] Output: The learning scenario in the virtual space that is displayed to the user.

[2378] Specific operation: The terminal uses the virtual space generation means to generate a scenario for a virtual store based on the learning content received from the server, and displays it to the user via a visual display device.

[2379] Step 7:

[2380] The user interacts within the virtual space and continues the experience.

[2381] Input: An action that the user gives voice instructions for.

[2382] Output: New collected speech and facial expression data.

[2383] Specific operation: The device uses a microphone and camera to collect the user's voice and facial expression data and transmits it back to the server.

[2384] Step 8:

[2385] Provide visual or audio feedback to the user.

[2386] Input: User's voice and facial expression data reanalyzed by the server.

[2387] Output: Feedback displayed on the terminal.

[2388] Specific operation: Based on the data reanalyzed by the server, feedback that is optimal for the user's pronunciation and emotional state is generated and provided to the user via the device.

[2389] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2391] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2392] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2393] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2394] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2395] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2396] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2397] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2398] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2399] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2400] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2401] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2402] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2403] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in...

Claims

1. a terminal means having a display device worn by a user; recognition means for recognizing and analyzing a user's voice; a server means for selecting and providing learning content; a VR scenario generation means for displaying the learning content provided by the server means via a display means and enabling a user to have a dialogue based on voice; feedback means for providing feedback to a user based on a result of said recognition means; A system including:

2. 2. The system according to claim 1, wherein said recognition means has a function of evaluating the user's pronunciation level.

3. 2. The system according to claim 1, wherein the feedback means includes means for visually or audibly informing the user of the quality of pronunciation.

Images