system

Abstract

Provide a system. The method includes: a means for allowing a user to select multiple foreign languages ​​to learn; means for generating a virtual reality space based on a user's selection; means for analyzing and evaluating user utterances; A means for awarding points based on the evaluations obtained by the user; a means for using said points to reserve a simulated session with a real-time instructor; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, it is extremely difficult for individuals to efficiently acquire multiple foreign languages. This difficulty increases even more when learning multiple languages ​​in parallel over a short period of time. With traditional learning methods, individuals have limited opportunities to actually use multiple languages ​​in their daily lives, making it difficult to improve their reflexes and practical skills. Furthermore, it is difficult to receive real-time feedback and evaluation, preventing the maximum benefit from learning. Furthermore, users are unable to concretely grasp their learning progress or results, making it difficult to maintain motivation to learn. [Means for solving the problem]

[0005] The present invention provides a system that allows users to efficiently learn multiple foreign languages ​​in a short period of time. The system includes a means for allowing a user to select multiple foreign languages ​​to learn, a means for generating a virtual reality space based on the user's selection, a means for analyzing and evaluating the user's speech, a means for awarding points based on the evaluation obtained by the user, and a means for using the points to reserve a simulated session with a real-time instructor. This system provides users with an opportunity to actually use multiple foreign languages ​​in a virtual reality space, thereby improving their spontaneous learning and practical skills. Furthermore, real-time feedback and evaluation make it easier for users to concretely grasp the effects of their learning, thereby maintaining their motivation to learn.

[0006] A "user" is an individual or entity that uses a system.

[0007] A "foreign language" is a language other than the user's native language.

[0008] "Selectable" means that the user can select one or more of a plurality of options.

[0009] A "virtual reality space" is a computer-generated three-dimensional virtual environment in which users can interact with others.

[0010] "Generation" is the process by which the system creates virtual reality spaces and other data.

[0011] "Utterance" is a communication act performed by a user using voice.

[0012] "Analysis" is the process by which a system interprets input data and understands its content.

[0013] "Evaluation" is the process by which the system scores and judges the user's speech and behavior based on criteria.

[0014] "Points" are a numerical representation of a user's learning achievements and can be used for specific services or activities.

[0015] "Awarding" refers to the act of the system giving points to a user.

[0016] "Real time" means processing immediately without delay.

[0017] An "instructor" is an expert who provides education and guidance to users.

[0018] A "simulated session" is a session that is not real, but takes place in a situation similar to reality.

[0019] A "reservation" is an act of reserving in advance the right to use a service or resource at a specific date and time in the future.

[0020] "Establishing a connection" refers to a system that enables two or more devices or services to communicate.

[0021] A "learning database" is an organized collection of information and data necessary for learning.

[0022] "Learning history" is a record of the learning activities a user has performed through the system.

[0023] A "virtual character" is an artificial intelligence character generated by the system to interact with the user in a virtual reality space. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] The present invention relates to a virtual reality (VR)-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[0046] Overall structure

[0047] This system consists of a user, a terminal (a user's VR device), and a server. The user accesses the system using the VR device, and the server manages and processes the entire data.

[0048] User Interface

[0049] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0050] Virtual reality space generation

[0051] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[0052] Language Learning Activities

[0053] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0054] Newspaper article summaries

[0055] Another activity is to allow users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points.

[0056] Points and real instructors

[0057] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[0058] Specific examples

[0059] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and interact with a virtual store clerk in Japanese. After completing the interaction, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and fluency of their use of both languages.

[0060] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[0061] According to the embodiments of the present invention, a user can learn multiple foreign languages ​​in a realistic environment, enabling more efficient language acquisition compared to conventional learning methods.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user puts on the VR device, starts the system, and the login screen appears.

[0065] The terminal displays the user's login information in an input form and prompts them to enter it.

[0066] Step 2:

[0067] The user enters their username and password and clicks the login button.

[0068] The terminal receives the input and sends the login information to the server.

[0069] Step 3:

[0070] The server checks the received login information against its database and authenticates it.

[0071] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[0072] Step 4:

[0073] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[0074] Users select 3-5 languages ​​they want to learn and press the next button.

[0075] Step 5:

[0076] The terminal transmits the user's selection to the server.

[0077] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[0078] Step 6:

[0079] After selecting a language, the user will be taken to the avatar customization screen.

[0080] The device will display the avatar settings options.

[0081] Users customize their avatar's appearance and complete the setup.

[0082] The device sends the set avatar to the server.

[0083] Step 7:

[0084] The server stores the received avatar information and generates the user's avatar in the VR space.

[0085] Step 8:

[0086] The user selects the first activity (e.g., shopping) in the VR space and starts.

[0087] The terminal requests scenario data for a shopping activity from the server.

[0088] The server transmits the corresponding scenario data to the terminal.

[0089] Step 9:

[0090] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[0091] The user speaks through a microphone and answers questions posed by the virtual character.

[0092] Step 10:

[0093] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[0094] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[0095] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[0096] Step 11:

[0097] The terminal displays the points and the evaluation results to the user.

[0098] Step 12:

[0099] The user selects the "read newspaper article" activity and presses the activity start button.

[0100] The terminal transmits activity selection information to the server.

[0101] The server selects the appropriate newspaper article and sends it to the terminal.

[0102] Step 13:

[0103] The terminal displays the newspaper article on the screen for the user to view.

[0104] The user reads the article and enters a summary in the input field.

[0105] Step 14:

[0106] After the user has finished entering the summary, he or she presses the send button.

[0107] The terminal transmits the summary data to the server.

[0108] Step 15:

[0109] The server reviews the summaries, rates them for accuracy, and calculates points.

[0110] The evaluation and point calculation results are sent back to the terminal.

[0111] Step 16:

[0112] The terminal displays the evaluation results and points to the user.

[0113] Step 17:

[0114] Users can use the points they earn to book sessions with instructors.

[0115] The terminal displays the reservation request and prompts the user for input.

[0116] Step 18:

[0117] The user enters the date, time and point information and presses the reservation button.

[0118] The terminal sends a reservation request to the server.

[0119] Step 19:

[0120] The server checks the instructor's schedule against the user's request and confirms the reservation.

[0121] The server returns reservation confirmation information to the terminal.

[0122] Step 20:

[0123] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[0124] Users can directly talk with instructors to deepen their learning.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] Traditional language learning methods often involve the use of textbooks and vocabulary books, making it difficult to efficiently improve actual communication skills. Furthermore, learning multiple foreign languages ​​at once can lead to reduced learning efficiency. Furthermore, to practice actual conversations, students must travel to the location or schedule sessions with specific instructors, which can be time-consuming and costly. There is a need to solve these problems and provide a more efficient and practical language learning environment.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for allowing a user to select multiple languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's voice, means for assigning a score based on the evaluation obtained by the user, and means for booking a simulated session with a real-time educator using the score, thereby enabling a user to efficiently learn multiple languages ​​through practical language activities in the virtual reality space and to use the acquired score to participate in a session with a real-time educator.

[0130] A "user" is someone who operates the system and learns a language.

[0131] "Language" refers to multiple foreign languages ​​selected by the User for the purpose of learning and using them.

[0132] "Selection mechanisms" refers to interfaces or features that allow users to choose the language they want to learn from multiple options.

[0133] "Virtual reality space" refers to a three-dimensional virtual environment that a user can access through a VR device.

[0134] "Means of generation" refers to the software and hardware functions for constructing and displaying virtual reality spaces.

[0135] "Means for analyzing and evaluating speech" refers to algorithms or systems that convert a user's speech into text using speech recognition technology and evaluate the content of that text.

[0136] "Score" refers to a score given based on a user's learning activities and evaluation results.

[0137] "Means for awarding" refers to the system or algorithm used to assign points to users based on their evaluation results.

[0138] "Real-time educator" refers to a lecturer or trainer with whom a user interacts during learning, either through virtual reality or in person.

[0139] "Simulated Session" refers to a virtual learning session conducted in real time between a user and an educator.

[0140] "Means for Booking" refers to an interface or system that allows a user to book a session with a real-time educator using a score.

[0141] "Virtual character" refers to an animated character that is displayed for a user to interact with within a virtual reality space.

[0142] "News article" refers to information or news articles that a user can view within a virtual reality space.

[0143] "Summary" refers to text or speech that the user has written in a concise summary of the contents of a newspaper article.

[0144] "Means for evaluation" refers to the system or algorithm that analyzes the content of the summary entered by the user and determines its quality and accuracy.

[0145] MODE FOR CARRYING OUT THE INVENTION

[0146] The present invention relates to a VR-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[0147] Hardware and software used

[0148] This system consists of a user, a terminal (the user's VR device), and a server. The specific hardware and software used are as follows:

[0149] Hardware:

[0150] VR device (e.g. Oculus Quest 2)

[0151] software:

[0152] VR applications (e.g. apps developed with Unity)

[0153] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0154] Database system (e.g., MySQL (registered trademark))

[0155] User operations

[0156] The user puts on the VR device, launches a dedicated application, and logs into the system. After logging in, the user selects the language they want to learn and enters the virtual reality space. Based on the selected language, the server sends the corresponding language data to the device.

[0157] Virtual reality space generation

[0158] The device uses the Unity engine to generate a virtual reality space based on the language data received from the server. This space contains virtual characters and specific scenarios (e.g., virtual towns, cafes), and is constructed so that users can carry out activities.

[0159] Language Learning Activities

[0160] Users interact with characters in a virtual reality space. During the interaction, the user's voice is collected by the device's microphone and converted into text in real time using a speech recognition API. The converted text data is sent to a server, which evaluates the content and assigns a score. This score is used to visualize the user's evaluation and progress.

[0161] Newspaper article summaries

[0162] As another learning activity, users can read newspaper articles in a virtual reality space. After reading the article, the user summarizes the content and sends the summary data via their device to a server. The server uses a generative AI model (e.g., GPT-4®) to evaluate the quality of the summary and assign an appropriate score.

[0163] Using points and sessions with real instructors

[0164] The points earned by users can be used to book a simulated session with a real-time educator. Users use their points to book a session with a teacher through the server, and once the reservation is confirmed, a connection is established at the specified time. Users can converse directly with the teacher and receive feedback on their pronunciation and grammar.

[0165] Examples of concrete examples and prompts

[0166] For example, if a user is learning Japanese and French, they log in to the system through a VR device, select Japanese or French, and then enter the virtual reality space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. They then switch to a scenario where they order at a French cafe and place their order in French. Through this series of activities, the user naturally improves their usage frequency and fluency in both languages.

[0167] (Example of a prompt)

[0168] "If a user wants to learn Japanese and French, they log in through a VR device and go through a shopping scenario in a virtual town in Japanese and a cafe ordering scenario in French. They can then use their points to book a session with a real-time instructor to further their learning."

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] Logging in to the system

[0172] Input: The user puts on the VR device and enters their ID and password to access the system.

[0173] Specific behavior:

[0174] The user puts on the VR device, launches the dedicated application, enters their ID and password on the login screen, and presses the "Login" button.

[0175] Data processing and calculation:

[0176] The terminal sends the entered ID and password to the server, which checks the database and, if authentication is successful, sends the user information back to the terminal.

[0177] Output: A successful authentication message that allows the user to access the main menu.

[0178] Step 2:

[0179] Selecting a language to learn

[0180] Input: The user selects the language they want to learn.

[0181] Specific behavior:

[0182] The user selects the language they want to learn (e.g., Japanese, French) from the main menu and performs the selection using the VR controller.

[0183] Data processing and calculation:

[0184] The terminal transmits the selected language data to the server, and the server prepares the corresponding language data and learning content and transmits them to the terminal.

[0185] Output: The learning content is displayed according to the user's language selection.

[0186] Step 3:

[0187] Virtual reality space generation

[0188] Input: Language data and scenario information sent by the server to the device.

[0189] Specific behavior:

[0190] The device uses the Unity engine to generate a virtual reality space based on the received language data and scenario information, placing textures and objects according to the scenario and generating a user avatar.

[0191] Data processing and calculation:

[0192] The device loads and renders 3D models and textures for the virtual space based on language data and scenario information.

[0193] Output: The virtual reality space is displayed on the user's VR device.

[0194] Step 4:

[0195] Conducting language learning activities

[0196] Input: User's voice input.

[0197] Specific behavior:

[0198] The user controls an avatar and interacts with virtual store clerks and characters. During the interaction, the user's voice is collected by the device's microphone.

[0199] Data processing and calculation:

[0200] The device converts the collected voice data into text in real time using a speech recognition API, and sends the converted text data to a server, which evaluates the content and calculates a score.

[0201] Output: The server evaluates the score and gives it back to the user.

[0202] Step 5:

[0203] Newspaper article summary activity

[0204] Input: A newspaper article that the user reads in a virtual reality space.

[0205] Specific behavior:

[0206] The user moves to a designated location in the virtual reality space and reads a newspaper article. After reading the article, the user summarizes the content and inputs it into the terminal.

[0207] Data processing and calculation:

[0208] The device sends the summary data to a server, which uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and calculate a score.

[0209] Output: The server evaluates the summary and gives the score as feedback to the user.

[0210] Step 6:

[0211] Book a session with an educator in real time

[0212] Input: User-specified desired date and time and session reservation information with the educator.

[0213] Specific behavior:

[0214] The user selects the "Book a session with an instructor" option from the main menu, chooses the desired date and time and instructor, and makes the reservation using points.

[0215] Data processing and calculation:

[0216] The reservation information is sent from the terminal to the server, and the server registers the information in the reservation system. The reservation is confirmed in the instructor's schedule management system.

[0217] Output: The user is provided with a reservation confirmation message and information on how to establish a connection at the specified time.

[0218] In this way, users can efficiently learn a language through specific input, processing, and output at each step.

[0219] (Application example 1)

[0220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] Conventional language learning systems have the drawback of providing users with few opportunities to practically learn the language actually used in the local area, making it difficult to acquire the high level of language proficiency required for real-life situations. There is also concern that the learning content may become monotonous, reducing user motivation. Furthermore, there are limited means for learning while receiving real-time feedback, which often results in insufficient learning efficiency.

[0222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0223] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the user's evaluation, means for reserving a real-time simulated session with a teacher using the points, and means for the user to learn a foreign language by selecting products and completing purchase and sale procedures in the virtual space. This allows the user to efficiently learn a language in an environment similar to real-life situations.

[0224] A "user" is someone who seeks to learn a language using the virtual reality-based language learning system of the present invention.

[0225] A "means for enabling selection" is a device or software that provides an interface and functionality for allowing a user to select the language they wish to learn from multiple foreign languages.

[0226] "Means for generating virtual reality spaces" refers to technologies and systems that create a virtual reality learning environment based on the language selected by the user.

[0227] "Means for analyzing and evaluating speech" refers to technology for analyzing a user's voice data and evaluating its content, specifically a system that uses voice recognition and natural language processing.

[0228] The "means for awarding points" is a mechanism for awarding reward points to a user based on an evaluation of the user's use of language.

[0229] The "means for reserving a simulated session with a real-time instructor" is a mechanism by which a user can use the points they have earned to make a reservation for a real-time language learning session with a real instructor.

[0230] The "means of selecting products in a virtual space and carrying out the buying and selling procedure" is a simulation function that allows users to select products in a virtual reality space and learn a language through that selection.

[0231] "Means for displaying a virtual character" refers to a technique or device for displaying a character for interaction with a user in a virtual reality space.

[0232] The "means for reading newspaper articles and inputting summaries of their contents" refers to an interface and technology that allows a user to view newspaper articles in a virtual reality space and input a summary of the main points.

[0233] The "means of evaluation" is a mechanism for evaluating a user's language proficiency based on the summary and dialogue content entered by the user.

[0234] The present invention relates to a system that allows users to efficiently learn multiple foreign languages ​​using VR (virtual reality). The overall configuration of the system consists of a server, a terminal (a user's VR device), and a user. Specific embodiments of the system are described below.

[0235] Overall structure

[0236] The system begins when the user accesses the server using a VR device, selects the foreign language they want to learn, and logs in. The server manages and processes all data.

[0237] User Interface

[0238] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0239] Virtual reality space generation

[0240] The device retrieves language data from the server based on the user's selection and generates a virtual reality space containing various scenarios in which the user can perform specific tasks (e.g., product selection and purchase procedures).

[0241] Language Learning Activities

[0242] Users interact with virtual characters in a virtual reality space. During the interaction, the device analyzes the user's speech and sends the data to the server. The server evaluates the speech and awards the user points. The user can use these points to challenge themselves to more difficult activities.

[0243] Product Selection Scenario

[0244] Users learn the language through a scenario in which they select and purchase products in a virtual store. They ask questions about the products in the local language and interact with a virtual character acting as a store clerk. Through this process, users can acquire vocabulary and phrases necessary for shopping and everyday conversation.

[0245] Newspaper article summaries

[0246] Another activity involves users reading local newspaper articles in the virtual reality space and summarizing their content, which is then sent to the server, which then evaluates and awards points.

[0247] Simulated sessions with real-time instructors

[0248] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[0249] Hardware and software used

[0250] Hardware

[0251] VR devices (e.g. Oculus Rift, HTC Vive)

[0252] A server (e.g., a Flask application server)

[0253] software

[0254] Speech recognition software (e.g., the SpeechRecognition Python library)

[0255] Translation API (e.g. Google Translate API)

[0256] Specific examples

[0257] For example, if a user wants to learn Japanese and French at the same time, they would follow the following steps: First, the user logs in to the system and selects Japanese and French as the languages ​​they want to learn. Next, the user starts a shopping scenario in a virtual store and selects products while interacting with the clerk in Japanese. Next, the user switches to a scenario where they are ordering at a French cafe and places their order in French.

[0258] Example prompts for generative AI models

[0259] "Design a scenario where a user can select products in a virtual store and have a conversation with a local sales associate."

[0260] "Create an algorithm that analyzes the user's voice and rates them based on a points system."

[0261] This allows users to efficiently improve their language skills in an environment that is close to the real world, and allows them to naturally acquire language skills in actual situations.

[0262] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0263] Step 1:

[0264] The user puts on the VR device and logs in to the system.

[0265] Input: The user enters login information into the VR device.

[0266] Data processing / calculation: The terminal sends the login information to the server for authentication. The server then verifies the user's authentication information by checking it against a database.

[0267] Output: If authentication is successful, the user's account information and learning history will be loaded.

[0268] Step 2:

[0269] The user selects the foreign language they want to learn.

[0270] Input: The user selects the language they wish to learn through the system's interface.

[0271] Data processing / calculation: The device receives the user's selection and sends the information to the server. The server retrieves the data in the selected language and prepares the virtual reality space.

[0272] Output: The virtual reality space is now set up and ready to be displayed to the user.

[0273] Step 3:

[0274] The device generates a virtual reality space.

[0275] Input: Language data obtained from the server and setting information for the virtual reality space.

[0276] Data processing / calculation: The terminal generates a virtual reality space based on the data received from the server.

[0277] Output: The generated virtual reality space is displayed on the user's VR device.

[0278] Step 4:

[0279] The user selects a product in the virtual reality space and completes the transaction process.

[0280] Input: Product information provided in the virtual reality space and user selection operations.

[0281] Data processing / calculation: When a user selects a product and starts a dialogue, the device collects the user's speech as voice data and sends it to the server, which analyzes the voice data and evaluates its content.

[0282] Output: The evaluation results are given to the user as points. The user receives feedback based on their learning status.

[0283] Step 5:

[0284] A user interacts with a virtual character in a virtual reality space.

[0285] Input: User utterances and virtual character configuration information.

[0286] Data processing / calculation: The device analyzes the user's speech in real time and generates appropriate responses from the virtual character. The server continuously records the dialogue and manages it as learning data.

[0287] Output: The interaction with the virtual character progresses and the user receives continuous feedback.

[0288] Step 6:

[0289] A user reads a newspaper article and enters a summary of it.

[0290] Input: Newspaper article provided in virtual reality space and user summary input.

[0291] Data processing / calculation: When a user inputs a summary, the terminal sends the content to the server, which evaluates the summary and assigns points based on the analysis results.

[0292] Output: Points are awarded and the user confirms the evaluation results.

[0293] Step 7:

[0294] Users use points to book simulated sessions with real-time instructors.

[0295] Input: Points acquired by the user and booking actions.

[0296] Data processing / calculation: The server checks the user's point balance and displays a list of available instructors. When a user reserves a session with an instructor, the server manages the connection time and instructor information and confirms the reservation.

[0297] Output: The reservation is confirmed and the user is notified of the reservation details. When the session time arrives, the device establishes a connection with the designated instructor and begins learning in real time.

[0298] In this way, through specific actions and data flows at each step, a system is realized that allows users to efficiently learn a foreign language in a virtual reality space.

[0299] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0300] The present invention combines a VR (virtual reality)-based language learning system that enables users to efficiently learn multiple foreign languages ​​with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[0301] Overall structure

[0302] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[0303] User Interface

[0304] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0305] Virtual reality space generation

[0306] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[0307] Language Learning Activities

[0308] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0309] Emotion Engine Functions

[0310] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. This makes the user's learning experience more personalized and efficient.

[0311] Newspaper article summaries

[0312] Another activity is for users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points. During this process, the emotion engine also analyzes the user's emotions and provides appropriate feedback and adjusts the points.

[0313] Points and real instructors

[0314] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[0315] Specific examples

[0316] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. After completing the conversation, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and spontaneity of their use of both languages. In addition, the emotion engine analyzes the user's emotional state in real time and provides appropriate feedback, further improving learning efficiency.

[0317] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[0318] According to the embodiment of the present invention, users can learn multiple foreign languages ​​in a realistic environment, which allows for more efficient language acquisition than conventional learning methods. Furthermore, by effectively utilizing the emotion engine, users can enjoy an optimal learning experience that is tailored to their emotional state.

[0319] The processing flow will be explained below.

[0320] Step 1:

[0321] The user puts on the VR device, starts the system, and the login screen appears.

[0322] The terminal displays the user's login information in an input form and prompts them to enter it.

[0323] Step 2:

[0324] The user enters their username and password and clicks the login button.

[0325] The terminal receives the input and sends the login information to the server.

[0326] Step 3:

[0327] The server checks the received login information against its database and authenticates it.

[0328] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[0329] Step 4:

[0330] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[0331] Users select 3-5 languages ​​they want to learn and press the next button.

[0332] Step 5:

[0333] The terminal transmits the user's selection to the server.

[0334] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[0335] Step 6:

[0336] After selecting a language, the user will be taken to the avatar customization screen.

[0337] The device will display the avatar settings options.

[0338] Users customize their avatar's appearance and complete the setup.

[0339] The device sends the set avatar to the server.

[0340] Step 7:

[0341] The server stores the received avatar information and generates the user's avatar in the VR space.

[0342] Step 8:

[0343] The user selects the first activity (e.g., shopping) in the VR space and starts.

[0344] The terminal requests scenario data for a shopping activity from the server.

[0345] The server transmits the corresponding scenario data to the terminal.

[0346] Step 9:

[0347] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[0348] The user speaks through a microphone and answers questions posed by the virtual character.

[0349] Step 10:

[0350] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[0351] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[0352] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[0353] Step 11:

[0354] The terminal displays the points and the evaluation results to the user.

[0355] Step 12:

[0356] The user selects the "read newspaper article" activity and presses the activity start button.

[0357] The terminal transmits activity selection information to the server.

[0358] The server selects the appropriate newspaper article and sends it to the terminal.

[0359] Step 13:

[0360] The terminal displays the newspaper article on the screen for the user to view.

[0361] The user reads the article and enters a summary in the input field.

[0362] Step 14:

[0363] After the user has finished entering the summary, he or she presses the send button.

[0364] The terminal transmits the summary data to the server.

[0365] Step 15:

[0366] The server reviews the summaries, rates them for accuracy, and calculates points.

[0367] The evaluation and point calculation results are sent back to the terminal.

[0368] Step 16:

[0369] The terminal displays the evaluation results and points to the user.

[0370] Step 17:

[0371] Users can use the points they earn to book sessions with instructors.

[0372] The terminal displays the reservation request and prompts the user for input.

[0373] Step 18:

[0374] The user enters the date, time and point information and presses the reservation button.

[0375] The terminal sends a reservation request to the server.

[0376] Step 19:

[0377] The server checks the instructor's schedule against the user's request and confirms the reservation.

[0378] The server returns reservation confirmation information to the terminal.

[0379] Step 20:

[0380] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[0381] Users can directly talk with instructors to deepen their learning.

[0382] Step 21:

[0383] While the user is active in the VR space, the device uses an emotion engine to analyze the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine will detect this and send it to the server.

[0384] Step 22:

[0385] The server adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. For example, if the emotion engine determines that the user is tired, it will switch to a lighter task or display an encouraging message.

[0386] Step 23:

[0387] When a user reads a newspaper article and summarizes it, the device uses an emotion engine to analyze the user's emotional state. For example, if the user is feeling stressed, the device may take an approach such as adjusting the evaluation criteria for summarizing.

[0388] Step 24:

[0389] During a simulated session with a real-time instructor, the device uses an emotion engine to monitor the user's emotional state. For example, if the user is nervous, the emotion engine provides this information to the instructor in real time, allowing the instructor to provide appropriate feedback and support.

[0390] This provides a learning experience that takes the user's emotions into consideration, maximizing learning efficiency and effectiveness.

[0391] Example 2

[0392] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0393] Conventional language learning systems have the problem of being unable to respond to individual learning needs due to insufficient feedback on users' learning progress and emotional state. In particular, they provide uniform learning content without considering users' emotions, making it difficult to achieve efficient learning. Furthermore, they are unable to effectively evaluate users' learning outcomes and reflect the results in the learning process, making it difficult to maintain motivation to learn.

[0394] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0395] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to study, means for generating a virtual reality space based on the user's selection, means for displaying a virtual character with which the user can interact in the virtual reality space, means for analyzing and evaluating the user's speech, an emotion engine that includes an analysis of the user's emotional state and means for adjusting the difficulty level of the learning content and the feedback content based on the emotional state, means for awarding points based on the evaluation obtained by the user, and means for booking a simulated session with a real-time instructor using the points. This makes it possible to provide a personalized learning experience based on the user's learning progress and emotional state.

[0396] "User" refers to an individual or organization that uses the system to learn a foreign language.

[0397] "Virtual reality space" refers to a digital environment different from the real world that users can enter through VR devices.

[0398] "Virtual character" refers to a digital character that is displayed to interact with a user within a virtual reality space.

[0399] "Utterance" refers to the verbal expressions used by a user when interacting with virtual characters and other users in a virtual reality space.

[0400] "Emotion engine" refers to a software or hardware system that monitors a user's facial expressions and vocal tone to analyze the user's emotional state.

[0401] "Emotional state" refers to the psychological and emotional state, such as stress, fatigue, or satisfaction, that a user experiences while using a system.

[0402] "Evaluation" refers to the analysis and scoring of a user's speech and activity by the server.

[0403] "Points" refer to rewards or scores awarded based on a user's learning outcomes.

[0404] A "simulated session" refers to a virtual online learning session between a real-time instructor and a user.

[0405] The present invention combines a VR-based language learning system that enables users to efficiently learn a foreign language with an emotion engine that recognizes the user's emotions. This system consists of a user, a terminal (the user's VR device), a server, and an emotion engine. Specific embodiments of this system are described below.

[0406] User Login and Language Selection

[0407] The user puts on a VR device (e.g., Oculus Rift or HTC Vive) and connects to the provided login screen. By entering their user ID and password, they are granted access to the system. Then, the user selects the foreign language they wish to learn on the dashboard. The options include multiple foreign languages ​​(e.g., Japanese, French).

[0408] Virtual reality space generation

[0409] The device retrieves language data from the server based on the user's selection. During this process, the device sends a request to the server (e.g., a cloud service), which returns information about the user's selected language from a database (e.g., MySQL). Using the retrieved data, the device generates a virtual reality space using the Unity engine.

[0410] Interaction in virtual reality space

[0411] The user enters the generated virtual reality space and selects various scenarios (e.g., shopping at a store, ordering at a cafe). The user's avatar is displayed and the user interacts with the virtual character. The user's voice is collected by the VR device's microphone and analyzed in real time.

[0412] Speech and emotion data analysis

[0413] The device sends the user's voice data to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converts it into text. The converted text data is sent to a server, which runs a script to evaluate the content. The server then awards points based on the evaluation results and returns the information to the device.

[0414] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. For example, it uses emotion recognition technology (e.g., Affectiva's SDK) to recognize emotions from facial expressions and transmits the emotion data to the server. The server then dynamically adjusts the difficulty of the learning content and the content of the feedback based on this data.

[0415] Summarizing and evaluating newspaper articles

[0416] The user reads a newspaper article displayed in the virtual reality space and summarizes its contents. The created summary is sent to the server via the device. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summary and award points. During this process, the emotion engine monitors the user's emotions and provides appropriate feedback and adjusts the points.

[0417] Points Redemption and Instructor Sessions

[0418] Users can use their earned points to book a simulated session with a real-time instructor. Once the reservation is confirmed, the server connects the instructor and the user via video chat at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[0419] Examples of concrete examples and prompts

[0420] For example, if a user wants to learn Japanese and French in parallel, they can follow this process:

[0421] 1. The user logs into the system, selects Japanese and French, and begins a conversation in Japanese in the virtual town.

[0422] 2. Once the dialogue is complete, place your order in French at a French cafe.

[0423] 3. During this process, the emotion engine detects the user's stress, and the server automatically adjusts the learning content.

[0424] 4. The user reads the newspaper article in the virtual reality space and sends the summary to the server.

[0425] 5. The server evaluates the summary and awards points.

[0426] 6. Finally, users can use their points to book a simulated session with a real-time instructor, during which the emotion engine monitors the user's emotions and provides real-time feedback to the instructor.

[0427] Prompt Sentence Examples

[0428] "I want to learn Japanese and French at the same time in a VR space. Please simulate shopping in a store in Japanese and ordering at a cafe in French. Also, please use an emotion engine to adjust the learning feedback."

[0429] This system allows users to efficiently learn multiple languages ​​in a realistic virtual environment. It also utilizes an emotion engine to provide the optimal learning environment for each user.

[0430] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0431] Step 1:

[0432] The user puts on the VR device and enters their user ID and password on the login screen. Once the input is complete and the authentication server verifies the authentication information and permits login, the user can access the dashboard. This allows the user to proceed to the next step.

[0433] Input: User ID, Password

[0434] Output: User authentication status (login status)

[0435] Step 2:

[0436] Users select the foreign language they want to learn on the dashboard. Once the system receives the user's selection, it sends that information to the server, which retrieves data about the selected language from a database and sends it back to the device, which then loads content related to the language the user wants to learn.

[0437] Input: Selected foreign language

[0438] Output: Data related to the selected foreign language

[0439] Step 3:

[0440] The device receives the language data sent from the server and generates a virtual reality space. At this time, the device uses a game engine such as the Unity engine to create the graphics and scenario for the virtual space. This allows the user to access a virtual reality space that corresponds to the selected language.

[0441] Input: Language data sent from the server

[0442] Output: Generated virtual reality space

[0443] Step 4:

[0444] The user enters the generated virtual reality space and selects a specific scenario (e.g., shopping at a store, ordering at a cafe). Once the selection is complete, a conversation with a virtual character based on the scenario begins. The user's speech is collected by the microphone of the VR device.

[0445] Input: Select a specific scenario

[0446] Output: Start of dialogue with virtual character

[0447] Step 5:

[0448] The device analyzes the user's voice data in real time. The voice data is sent to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converted into text data. The converted text data is sent to a server, which then runs a script to evaluate the content. Points are awarded based on the evaluation results.

[0449] Input: User's voice data

[0450] Output: Evaluation results and points awarded

[0451] Step 6:

[0452] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. The emotion data is sent to a server, which then uses this data to perform calculations to adjust the difficulty of the learning content and the content of the feedback. This provides an optimal learning environment according to the user's emotional state.

[0453] Input: User's facial expressions and voice tone

[0454] Output: Adjustment of learning content and feedback

[0455] Step 7:

[0456] Users read newspaper articles displayed in a virtual reality space and create summaries. The summaries are then sent to a server via their devices. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summaries, and points are awarded based on the evaluation results.

[0457] Input: User-generated abstract

[0458] Output: Evaluation results and points awarded

[0459] Step 8:

[0460] Users can use their earned points to book a simulated session with a real-time instructor. The reservation information is sent to the server, which then connects the instructor and user via a video chat platform at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[0461] Input: Reservation information and points

[0462] Output: Connection with instructor and real-time feedback

[0463] (Application example 2)

[0464] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0465] Conventional language learning systems do not provide sufficient support for users to efficiently learn multiple foreign languages, and it is difficult to provide an optimal learning experience that reflects the user's emotional state in real time while learning.

[0466] Furthermore, the lack of a virtual store customer service simulation as a practical application environment for language learning makes it difficult to implement practical learning that takes into account real-world applications. To solve these problems, a system is needed that can grasp the user's emotional state in real time and adjust the learning content based on that, while providing a virtual experience that is close to real-life situations.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0468] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the evaluation obtained by the user, means for reserving a simulated session with a real-time instructor using the points, means for recognizing the user's emotional state and analyzing the data to optimize the learning experience, and means for the user to simulate customer service in a foreign language in a virtual store. This enables a user to efficiently acquire multiple foreign languages ​​through practical language use scenarios while receiving learning content personalized in real time according to their emotional state.

[0469] "User" refers to an individual who uses the system to learn a foreign language.

[0470] "Multiple foreign languages" refers to multiple different languages ​​that a user can select at will to learn.

[0471] "Virtual reality space" refers to a computer-generated simulated environment that a user experiences through a VR device.

[0472] "Utterance" refers to verbal communication that the user makes while studying.

[0473] "Evaluation" refers to analyzing the content of a user's utterance and converting it into a score based on its accuracy and appropriateness.

[0474] "Points" refer to scores or rewards based on evaluations that users obtain through learning activities.

[0475] "Real-time instructor" refers to a human instructor who provides direct instruction to a user through the system.

[0476] "Simulated session" refers to a simulated learning or interaction session within a virtual environment.

[0477] "Emotional state" refers to the psychological state of the user that is analyzed from facial expressions, tone of voice, and the like.

[0478] "Virtual store" refers to an environment set in virtual reality that resembles a physical store.

[0479] "Customer service simulation" refers to a learning activity in which a user interacts as a store clerk or a customer in a virtual store, simulating a scenario in which the user provides or receives service.

[0480] The present invention is a system that allows users to efficiently learn multiple foreign languages, and provides a more personalized learning experience by combining a VR-based virtual reality space with an emotion engine that analyzes the user's emotional state. A specific embodiment of this system will be described below.

[0481] Overall structure

[0482] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[0483] User Interface

[0484] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0485] Virtual reality space generation

[0486] The terminal acquires language data from the server based on the user's selection and generates a virtual reality space. Various scenarios are set in this space, and the user can perform specific tasks (e.g., customer service simulations in a virtual store) within the space. Specifically, the system provides scenarios for the user to ask for product explanations or place orders as a customer.

[0487] Language Learning Activities

[0488] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0489] Emotion Engine Functions

[0490] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. The emotion engine used can be, for example, the Affectiva SDK.

[0491] Points and real instructors

[0492] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[0493] Hardware and software used

[0494] Hardware:

[0495] VR device (e.g., Oculus Rift): Used to display the virtual space and capture user input.

[0496] Emotion engine (e.g., Affectiva SDK): Recognizes emotions by analyzing the user's facial expression data.

[0497] software:

[0498] Language processing engine (e.g., Google Cloud Translation API): Analyzes the user's speech and translates it into the foreign language of their choice.

[0499] Database: Stores and manages users' learning progress and points.

[0500] Specific examples

[0501] For example, in a scenario where a user is a salesperson in a virtual store explaining a product in French, the system would prompt the user with questions such as, "What are the features of this product?" The user would respond to the question in French, and their responses would be analyzed and evaluated. If the emotion engine detects the user becoming confused or stressed, the system would adjust its learning and provide clearer explanations or additional support.

[0502] Prompt Sentence Examples

[0503] "As a customer in a virtual store in a VR space, ask for product details in French."

[0504] "Use an emotion engine to analyze user emotions and optimize the learning experience."

[0505] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0506] Step 1: User Login and Language Selection

[0507] The user puts on the VR device and logs in to the system. The system receives the user's ID and password as input and authenticates them. If authentication is successful, the user selects one or more foreign languages ​​to learn. The selected language data is sent to the server.

[0508] Step 2: Creating a virtual reality space

[0509] The server acquires language data based on the user's selection and sends information to the terminal (VR device) to generate a virtual reality space. The terminal generates the virtual reality space and displays it to the user. At this time, scenarios and tasks within the virtual space (for example, a customer service simulation in a virtual store) are also set.

[0510] Step 3: User begins interacting with virtual character

[0511] The user begins to interact with a virtual character in a virtual reality space. The virtual character presents the user with a predefined prompt, such as "Please tell me about the features of this product." The VR device captures the user's voice and sends it to the server as audio data.

[0512] Step 4: Analyzing and evaluating user voice

[0513] The server analyzes the received voice data using a language processing engine (for example, Google Cloud Translation API). As a result of the analysis, the voice input content is converted into text data and evaluated. This evaluation is based on criteria such as accuracy, appropriateness, and pronunciation. The evaluation results are awarded to the user as points.

[0514] Step 5: Emotion analysis using the emotion engine

[0515] Facial expression data and voice tone data captured during user interaction are analyzed using an emotion engine (e.g., Affectiva SDK). The emotion engine recognizes the user's emotional state and sends the results to the server. The server uses this data to determine the user's level of fatigue or stress.

[0516] Step 6: Adjust your learning

[0517] The server then adjusts the user's learning based on the analyzed emotional data, adjusting the level of feedback and explanation provided to the user. For example, if the user is confused, the server will provide clearer explanations and additional support.

[0518] Step 7: Use points and simulate a real-time instructor session

[0519] Users use points earned in the virtual reality space to reserve a simulated session with a real-time instructor. The server checks point balances and manages reservations. Once a reservation is confirmed, a connection is established between the instructor and the user at the specified time. During the session, the emotion engine monitors the user's state and provides real-time feedback to the instructor.

[0520] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0521] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0522] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0523] [Second embodiment]

[0524] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0525] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0526] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0527] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0528] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0529] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0530] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0531] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0532] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0533] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0534] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0535] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0536] The present invention relates to a virtual reality (VR)-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[0537] Overall structure

[0538] This system consists of a user, a terminal (a user's VR device), and a server. The user accesses the system using the VR device, and the server manages and processes the entire data.

[0539] User Interface

[0540] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0541] Virtual reality space generation

[0542] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[0543] Language Learning Activities

[0544] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0545] Newspaper article summaries

[0546] Another activity is to allow users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points.

[0547] Points and real instructors

[0548] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[0549] Specific examples

[0550] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and interact with a virtual store clerk in Japanese. After completing the interaction, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and fluency of their use of both languages.

[0551] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[0552] According to the embodiments of the present invention, a user can learn multiple foreign languages ​​in a realistic environment, enabling more efficient language acquisition compared to conventional learning methods.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] The user puts on the VR device, starts the system, and the login screen appears.

[0556] The terminal displays the user's login information in an input form and prompts them to enter it.

[0557] Step 2:

[0558] The user enters their username and password and clicks the login button.

[0559] The terminal receives the input and sends the login information to the server.

[0560] Step 3:

[0561] The server checks the received login information against its database and authenticates it.

[0562] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[0563] Step 4:

[0564] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[0565] Users select 3-5 languages ​​they want to learn and press the next button.

[0566] Step 5:

[0567] The terminal transmits the user's selection to the server.

[0568] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[0569] Step 6:

[0570] After selecting a language, the user will be taken to the avatar customization screen.

[0571] The device will display the avatar settings options.

[0572] Users customize their avatar's appearance and complete the setup.

[0573] The device sends the set avatar to the server.

[0574] Step 7:

[0575] The server stores the received avatar information and generates the user's avatar in the VR space.

[0576] Step 8:

[0577] The user selects the first activity (e.g., shopping) in the VR space and starts.

[0578] The terminal requests scenario data for a shopping activity from the server.

[0579] The server transmits the corresponding scenario data to the terminal.

[0580] Step 9:

[0581] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[0582] The user speaks through a microphone and answers questions posed by the virtual character.

[0583] Step 10:

[0584] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[0585] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[0586] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[0587] Step 11:

[0588] The terminal displays the points and the evaluation results to the user.

[0589] Step 12:

[0590] The user selects the "read newspaper article" activity and presses the activity start button.

[0591] The terminal transmits activity selection information to the server.

[0592] The server selects the appropriate newspaper article and sends it to the terminal.

[0593] Step 13:

[0594] The terminal displays the newspaper article on the screen for the user to view.

[0595] The user reads the article and enters a summary in the input field.

[0596] Step 14:

[0597] After the user has finished entering the summary, he or she presses the send button.

[0598] The terminal transmits the summary data to the server.

[0599] Step 15:

[0600] The server reviews the summaries, rates them for accuracy, and calculates points.

[0601] The evaluation and point calculation results are sent back to the terminal.

[0602] Step 16:

[0603] The terminal displays the evaluation results and points to the user.

[0604] Step 17:

[0605] Users can use the points they earn to book sessions with instructors.

[0606] The terminal displays the reservation request and prompts the user for input.

[0607] Step 18:

[0608] The user enters the date, time and point information and presses the reservation button.

[0609] The terminal sends a reservation request to the server.

[0610] Step 19:

[0611] The server checks the instructor's schedule against the user's request and confirms the reservation.

[0612] The server returns reservation confirmation information to the terminal.

[0613] Step 20:

[0614] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[0615] Users can directly talk with instructors to deepen their learning.

[0616] Example 1

[0617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] Traditional language learning methods often involve the use of textbooks and vocabulary books, making it difficult to efficiently improve actual communication skills. Furthermore, learning multiple foreign languages ​​at once can lead to reduced learning efficiency. Furthermore, to practice actual conversations, students must travel to the location or schedule sessions with specific instructors, which can be time-consuming and costly. There is a need to solve these problems and provide a more efficient and practical language learning environment.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0620] In this invention, the server includes means for allowing a user to select multiple languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's voice, means for assigning a score based on the evaluation obtained by the user, and means for booking a simulated session with a real-time educator using the score, thereby enabling a user to efficiently learn multiple languages ​​through practical language activities in the virtual reality space and to use the acquired score to participate in a session with a real-time educator.

[0621] A "user" is someone who operates the system and learns a language.

[0622] "Language" refers to multiple foreign languages ​​selected by the User for the purpose of learning and using them.

[0623] "Selection mechanisms" refers to interfaces or features that allow users to choose the language they want to learn from multiple options.

[0624] "Virtual reality space" refers to a three-dimensional virtual environment that a user can access through a VR device.

[0625] "Means of generation" refers to the software and hardware functions for constructing and displaying virtual reality spaces.

[0626] "Means for analyzing and evaluating speech" refers to algorithms or systems that convert a user's speech into text using speech recognition technology and evaluate the content of that text.

[0627] "Score" refers to a score given based on a user's learning activities and evaluation results.

[0628] "Means for awarding" refers to the system or algorithm used to assign points to users based on their evaluation results.

[0629] "Real-time educator" refers to a lecturer or trainer with whom a user interacts during learning, either through virtual reality or in person.

[0630] "Simulated Session" refers to a virtual learning session conducted in real time between a user and an educator.

[0631] "Means for Booking" refers to an interface or system that allows a user to book a session with a real-time educator using a score.

[0632] "Virtual character" refers to an animated character that is displayed for a user to interact with within a virtual reality space.

[0633] "News article" refers to information or news articles that a user can view within a virtual reality space.

[0634] "Summary" refers to text or speech that the user has written in a concise summary of the contents of a newspaper article.

[0635] "Means for evaluation" refers to the system or algorithm that analyzes the content of the summary entered by the user and determines its quality and accuracy.

[0636] MODE FOR CARRYING OUT THE INVENTION

[0637] The present invention relates to a VR-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[0638] Hardware and software used

[0639] This system consists of a user, a terminal (the user's VR device), and a server. The specific hardware and software used are as follows:

[0640] Hardware:

[0641] VR device (e.g. Oculus Quest 2)

[0642] software:

[0643] VR applications (e.g. apps developed with Unity)

[0644] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0645] Database system (e.g. MySQL)

[0646] User operations

[0647] The user puts on the VR device, launches a dedicated application, and logs into the system. After logging in, the user selects the language they want to learn and enters the virtual reality space. Based on the selected language, the server sends the corresponding language data to the device.

[0648] Virtual reality space generation

[0649] The device uses the Unity engine to generate a virtual reality space based on the language data received from the server. This space contains virtual characters and specific scenarios (e.g., virtual towns, cafes), and is constructed so that users can carry out activities.

[0650] Language Learning Activities

[0651] Users interact with characters in a virtual reality space. During the interaction, the user's voice is collected by the device's microphone and converted into text in real time using a speech recognition API. The converted text data is sent to a server, which evaluates the content and assigns a score. This score is used to visualize the user's evaluation and progress.

[0652] Newspaper article summaries

[0653] As another learning activity, users can read newspaper articles in a virtual reality space. After reading the article, they summarize the content and send the summary data to a server via their device. The server uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and assign an appropriate score.

[0654] Using points and sessions with real instructors

[0655] The points earned by users can be used to book a simulated session with a real-time educator. Users use their points to book a session with a teacher through the server, and once the reservation is confirmed, a connection is established at the specified time. Users can converse directly with the teacher and receive feedback on their pronunciation and grammar.

[0656] Examples of concrete examples and prompts

[0657] For example, if a user is learning Japanese and French, they log in to the system through a VR device, select Japanese or French, and then enter the virtual reality space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. They then switch to a scenario where they order at a French cafe and place their order in French. Through this series of activities, the user naturally improves their usage frequency and fluency in both languages.

[0658] (Example of a prompt)

[0659] "If a user wants to learn Japanese and French, they log in through a VR device and go through a shopping scenario in a virtual town in Japanese and a cafe ordering scenario in French. They can then use their points to book a session with a real-time instructor to further their learning."

[0660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0661] Step 1:

[0662] Logging in to the system

[0663] Input: The user puts on the VR device and enters their ID and password to access the system.

[0664] Specific behavior:

[0665] The user puts on the VR device, launches the dedicated application, enters their ID and password on the login screen, and presses the "Login" button.

[0666] Data processing and calculation:

[0667] The terminal sends the entered ID and password to the server, which checks the database and, if authentication is successful, sends the user information back to the terminal.

[0668] Output: A successful authentication message that allows the user to access the main menu.

[0669] Step 2:

[0670] Selecting a language to learn

[0671] Input: The user selects the language they want to learn.

[0672] Specific behavior:

[0673] The user selects the language they want to learn (e.g., Japanese, French) from the main menu and performs the selection using the VR controller.

[0674] Data processing and calculation:

[0675] The terminal transmits the selected language data to the server, and the server prepares the corresponding language data and learning content and transmits them to the terminal.

[0676] Output: The learning content is displayed according to the user's language selection.

[0677] Step 3:

[0678] Virtual reality space generation

[0679] Input: Language data and scenario information sent by the server to the device.

[0680] Specific behavior:

[0681] The device uses the Unity engine to generate a virtual reality space based on the received language data and scenario information, placing textures and objects according to the scenario and generating a user avatar.

[0682] Data processing and calculation:

[0683] The device loads and renders 3D models and textures for the virtual space based on language data and scenario information.

[0684] Output: The virtual reality space is displayed on the user's VR device.

[0685] Step 4:

[0686] Conducting language learning activities

[0687] Input: User's voice input.

[0688] Specific behavior:

[0689] The user controls an avatar and interacts with virtual store clerks and characters. During the interaction, the user's voice is collected by the device's microphone.

[0690] Data processing and calculation:

[0691] The device converts the collected voice data into text in real time using a speech recognition API, and sends the converted text data to a server, which evaluates the content and calculates a score.

[0692] Output: The server evaluates the score and gives it back to the user.

[0693] Step 5:

[0694] Newspaper article summary activity

[0695] Input: A newspaper article that the user reads in a virtual reality space.

[0696] Specific behavior:

[0697] The user moves to a designated location in the virtual reality space and reads a newspaper article. After reading the article, the user summarizes the content and inputs it into the terminal.

[0698] Data processing and calculation:

[0699] The device sends the summary data to a server, which uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and calculate a score.

[0700] Output: The server evaluates the summary and gives the score as feedback to the user.

[0701] Step 6:

[0702] Book a session with an educator in real time

[0703] Input: User-specified desired date and time and session reservation information with the educator.

[0704] Specific behavior:

[0705] The user selects the "Book a session with an instructor" option from the main menu, chooses the desired date and time and instructor, and makes the reservation using points.

[0706] Data processing and calculation:

[0707] The reservation information is sent from the terminal to the server, and the server registers the information in the reservation system. The reservation is confirmed in the instructor's schedule management system.

[0708] Output: The user is provided with a reservation confirmation message and information on how to establish a connection at the specified time.

[0709] In this way, users can efficiently learn a language through specific input, processing, and output at each step.

[0710] (Application example 1)

[0711] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0712] Conventional language learning systems have the drawback of providing users with few opportunities to practically learn the language actually used in the local area, making it difficult to acquire the high level of language proficiency required for real-life situations. There is also concern that the learning content may become monotonous, reducing user motivation. Furthermore, there are limited means for learning while receiving real-time feedback, which often results in insufficient learning efficiency.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0714] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the user's evaluation, means for reserving a real-time simulated session with a teacher using the points, and means for the user to learn a foreign language by selecting products and completing purchase and sale procedures in the virtual space. This allows the user to efficiently learn a language in an environment similar to real-life situations.

[0715] A "user" is someone who seeks to learn a language using the virtual reality-based language learning system of the present invention.

[0716] A "means for enabling selection" is a device or software that provides an interface and functionality for allowing a user to select the language they wish to learn from multiple foreign languages.

[0717] "Means for generating virtual reality spaces" refers to technologies and systems that create a virtual reality learning environment based on the language selected by the user.

[0718] "Means for analyzing and evaluating speech" refers to technology for analyzing a user's voice data and evaluating its content, specifically a system that uses voice recognition and natural language processing.

[0719] The "means for awarding points" is a mechanism for awarding reward points to a user based on an evaluation of the user's use of language.

[0720] The "means for reserving a simulated session with a real-time instructor" is a mechanism by which a user can use the points they have earned to make a reservation for a real-time language learning session with a real instructor.

[0721] The "means of selecting products in a virtual space and carrying out the buying and selling procedure" is a simulation function that allows users to select products in a virtual reality space and learn a language through that selection.

[0722] "Means for displaying a virtual character" refers to a technique or device for displaying a character for interaction with a user in a virtual reality space.

[0723] The "means for reading newspaper articles and inputting summaries of their contents" refers to an interface and technology that allows a user to view newspaper articles in a virtual reality space and input a summary of the main points.

[0724] The "means of evaluation" is a mechanism for evaluating a user's language proficiency based on the summary and dialogue content entered by the user.

[0725] The present invention relates to a system that allows users to efficiently learn multiple foreign languages ​​using VR (virtual reality). The overall configuration of the system consists of a server, a terminal (a user's VR device), and a user. Specific embodiments of the system are described below.

[0726] Overall structure

[0727] The system begins when the user accesses the server using a VR device, selects the foreign language they want to learn, and logs in. The server manages and processes all data.

[0728] User Interface

[0729] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0730] Virtual reality space generation

[0731] The device retrieves language data from the server based on the user's selection and generates a virtual reality space containing various scenarios in which the user can perform specific tasks (e.g., product selection and purchase procedures).

[0732] Language Learning Activities

[0733] Users interact with virtual characters in a virtual reality space. During the interaction, the device analyzes the user's speech and sends the data to the server. The server evaluates the speech and awards the user points. The user can use these points to challenge themselves to more difficult activities.

[0734] Product Selection Scenario

[0735] Users learn the language through a scenario in which they select and purchase products in a virtual store. They ask questions about the products in the local language and interact with a virtual character acting as a store clerk. Through this process, users can acquire vocabulary and phrases necessary for shopping and everyday conversation.

[0736] Newspaper article summaries

[0737] Another activity involves users reading local newspaper articles in the virtual reality space and summarizing their content, which is then sent to the server, which then evaluates and awards points.

[0738] Simulated sessions with real-time instructors

[0739] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[0740] Hardware and software used

[0741] Hardware

[0742] VR devices (e.g. Oculus Rift, HTC Vive)

[0743] A server (e.g., a Flask application server)

[0744] software

[0745] Speech recognition software (e.g., the SpeechRecognition Python library)

[0746] Translation API (e.g. Google Translate API)

[0747] Specific examples

[0748] For example, if a user wants to learn Japanese and French at the same time, they would follow the following steps: First, the user logs in to the system and selects Japanese and French as the languages ​​they want to learn. Next, the user starts a shopping scenario in a virtual store and selects products while interacting with the clerk in Japanese. Next, the user switches to a scenario where they are ordering at a French cafe and places their order in French.

[0749] Example prompts for generative AI models

[0750] "Design a scenario where a user can select products in a virtual store and have a conversation with a local sales associate."

[0751] "Create an algorithm that analyzes the user's voice and rates them based on a points system."

[0752] This allows users to efficiently improve their language skills in an environment that is close to the real world, and allows them to naturally acquire language skills in actual situations.

[0753] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0754] Step 1:

[0755] The user puts on the VR device and logs in to the system.

[0756] Input: The user enters login information into the VR device.

[0757] Data processing / calculation: The terminal sends the login information to the server for authentication. The server then verifies the user's authentication information by checking it against a database.

[0758] Output: If authentication is successful, the user's account information and learning history will be loaded.

[0759] Step 2:

[0760] The user selects the foreign language they want to learn.

[0761] Input: The user selects the language they wish to learn through the system's interface.

[0762] Data processing / calculation: The device receives the user's selection and sends the information to the server. The server retrieves the data in the selected language and prepares the virtual reality space.

[0763] Output: The virtual reality space is now set up and ready to be displayed to the user.

[0764] Step 3:

[0765] The device generates a virtual reality space.

[0766] Input: Language data obtained from the server and setting information for the virtual reality space.

[0767] Data processing / calculation: The terminal generates a virtual reality space based on the data received from the server.

[0768] Output: The generated virtual reality space is displayed on the user's VR device.

[0769] Step 4:

[0770] The user selects a product in the virtual reality space and completes the transaction process.

[0771] Input: Product information provided in the virtual reality space and user selection operations.

[0772] Data processing / calculation: When a user selects a product and starts a dialogue, the device collects the user's speech as voice data and sends it to the server, which analyzes the voice data and evaluates its content.

[0773] Output: The evaluation results are given to the user as points. The user receives feedback based on their learning status.

[0774] Step 5:

[0775] A user interacts with a virtual character in a virtual reality space.

[0776] Input: User utterances and virtual character configuration information.

[0777] Data processing / calculation: The device analyzes the user's speech in real time and generates appropriate responses from the virtual character. The server continuously records the dialogue and manages it as learning data.

[0778] Output: The interaction with the virtual character progresses and the user receives continuous feedback.

[0779] Step 6:

[0780] A user reads a newspaper article and enters a summary of it.

[0781] Input: Newspaper article provided in virtual reality space and user summary input.

[0782] Data processing / calculation: When a user inputs a summary, the terminal sends the content to the server, which evaluates the summary and assigns points based on the analysis results.

[0783] Output: Points are awarded and the user confirms the evaluation results.

[0784] Step 7:

[0785] Users use points to book simulated sessions with real-time instructors.

[0786] Input: Points acquired by the user and booking actions.

[0787] Data processing / calculation: The server checks the user's point balance and displays a list of available instructors. When a user reserves a session with an instructor, the server manages the connection time and instructor information and confirms the reservation.

[0788] Output: The reservation is confirmed and the user is notified of the reservation details. When the session time arrives, the device establishes a connection with the designated instructor and begins learning in real time.

[0789] In this way, through specific actions and data flows at each step, a system is realized that allows users to efficiently learn a foreign language in a virtual reality space.

[0790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0791] The present invention combines a VR (virtual reality)-based language learning system that enables users to efficiently learn multiple foreign languages ​​with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[0792] Overall structure

[0793] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[0794] User Interface

[0795] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0796] Virtual reality space generation

[0797] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[0798] Language Learning Activities

[0799] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0800] Emotion Engine Functions

[0801] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. This makes the user's learning experience more personalized and efficient.

[0802] Newspaper article summaries

[0803] Another activity is for users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points. During this process, the emotion engine also analyzes the user's emotions and provides appropriate feedback and adjusts the points.

[0804] Points and real instructors

[0805] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[0806] Specific examples

[0807] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. After completing the conversation, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and spontaneity of their use of both languages. In addition, the emotion engine analyzes the user's emotional state in real time and provides appropriate feedback, further improving learning efficiency.

[0808] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[0809] According to the embodiment of the present invention, users can learn multiple foreign languages ​​in a realistic environment, which allows for more efficient language acquisition than conventional learning methods. Furthermore, by effectively utilizing the emotion engine, users can enjoy an optimal learning experience that is tailored to their emotional state.

[0810] The processing flow will be explained below.

[0811] Step 1:

[0812] The user puts on the VR device, starts the system, and the login screen appears.

[0813] The terminal displays the user's login information in an input form and prompts them to enter it.

[0814] Step 2:

[0815] The user enters their username and password and clicks the login button.

[0816] The terminal receives the input and sends the login information to the server.

[0817] Step 3:

[0818] The server checks the received login information against its database and authenticates it.

[0819] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[0820] Step 4:

[0821] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[0822] Users select 3-5 languages ​​they want to learn and press the next button.

[0823] Step 5:

[0824] The terminal transmits the user's selection to the server.

[0825] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[0826] Step 6:

[0827] After selecting a language, the user will be taken to the avatar customization screen.

[0828] The device will display the avatar settings options.

[0829] Users customize their avatar's appearance and complete the setup.

[0830] The device sends the set avatar to the server.

[0831] Step 7:

[0832] The server stores the received avatar information and generates the user's avatar in the VR space.

[0833] Step 8:

[0834] The user selects the first activity (e.g., shopping) in the VR space and starts.

[0835] The terminal requests scenario data for a shopping activity from the server.

[0836] The server transmits the corresponding scenario data to the terminal.

[0837] Step 9:

[0838] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[0839] The user speaks through a microphone and answers questions posed by the virtual character.

[0840] Step 10:

[0841] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[0842] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[0843] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[0844] Step 11:

[0845] The terminal displays the points and the evaluation results to the user.

[0846] Step 12:

[0847] The user selects the "read newspaper article" activity and presses the activity start button.

[0848] The terminal transmits activity selection information to the server.

[0849] The server selects the appropriate newspaper article and sends it to the terminal.

[0850] Step 13:

[0851] The terminal displays the newspaper article on the screen for the user to view.

[0852] The user reads the article and enters a summary in the input field.

[0853] Step 14:

[0854] After the user has finished entering the summary, he or she presses the send button.

[0855] The terminal transmits the summary data to the server.

[0856] Step 15:

[0857] The server reviews the summaries, rates them for accuracy, and calculates points.

[0858] The evaluation and point calculation results are sent back to the terminal.

[0859] Step 16:

[0860] The terminal displays the evaluation results and points to the user.

[0861] Step 17:

[0862] Users can use the points they earn to book sessions with instructors.

[0863] The terminal displays the reservation request and prompts the user for input.

[0864] Step 18:

[0865] The user enters the date, time and point information and presses the reservation button.

[0866] The terminal sends a reservation request to the server.

[0867] Step 19:

[0868] The server checks the instructor's schedule against the user's request and confirms the reservation.

[0869] The server returns reservation confirmation information to the terminal.

[0870] Step 20:

[0871] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[0872] Users can directly talk with instructors to deepen their learning.

[0873] Step 21:

[0874] While the user is active in the VR space, the device uses an emotion engine to analyze the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine will detect this and send it to the server.

[0875] Step 22:

[0876] The server adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. For example, if the emotion engine determines that the user is tired, it will switch to a lighter task or display an encouraging message.

[0877] Step 23:

[0878] When a user reads a newspaper article and summarizes it, the device uses an emotion engine to analyze the user's emotional state. For example, if the user is feeling stressed, the device may take an approach such as adjusting the evaluation criteria for summarizing.

[0879] Step 24:

[0880] During a simulated session with a real-time instructor, the device uses an emotion engine to monitor the user's emotional state. For example, if the user is nervous, the emotion engine provides this information to the instructor in real time, allowing the instructor to provide appropriate feedback and support.

[0881] This provides a learning experience that takes the user's emotions into consideration, maximizing learning efficiency and effectiveness.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Conventional language learning systems have the problem of being unable to respond to individual learning needs due to insufficient feedback on users' learning progress and emotional state. In particular, they provide uniform learning content without considering users' emotions, making it difficult to achieve efficient learning. Furthermore, they are unable to effectively evaluate users' learning outcomes and reflect the results in the learning process, making it difficult to maintain motivation to learn.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0886] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to study, means for generating a virtual reality space based on the user's selection, means for displaying a virtual character with which the user can interact in the virtual reality space, means for analyzing and evaluating the user's speech, an emotion engine that includes an analysis of the user's emotional state and means for adjusting the difficulty level of the learning content and the feedback content based on the emotional state, means for awarding points based on the evaluation obtained by the user, and means for booking a simulated session with a real-time instructor using the points. This makes it possible to provide a personalized learning experience based on the user's learning progress and emotional state.

[0887] "User" refers to an individual or organization that uses the system to learn a foreign language.

[0888] "Virtual reality space" refers to a digital environment different from the real world that users can enter through VR devices.

[0889] "Virtual character" refers to a digital character that is displayed to interact with a user within a virtual reality space.

[0890] "Utterance" refers to the verbal expressions used by a user when interacting with virtual characters and other users in a virtual reality space.

[0891] "Emotion engine" refers to a software or hardware system that monitors a user's facial expressions and vocal tone to analyze the user's emotional state.

[0892] "Emotional state" refers to the psychological and emotional state, such as stress, fatigue, or satisfaction, that a user experiences while using a system.

[0893] "Evaluation" refers to the analysis and scoring of a user's speech and activity by the server.

[0894] "Points" refer to rewards or scores awarded based on a user's learning outcomes.

[0895] A "simulated session" refers to a virtual online learning session between a real-time instructor and a user.

[0896] The present invention combines a VR-based language learning system that enables users to efficiently learn a foreign language with an emotion engine that recognizes the user's emotions. This system consists of a user, a terminal (the user's VR device), a server, and an emotion engine. Specific embodiments of this system are described below.

[0897] User Login and Language Selection

[0898] The user puts on a VR device (e.g., Oculus Rift or HTC Vive) and connects to the provided login screen. By entering their user ID and password, they are granted access to the system. Then, the user selects the foreign language they wish to learn on the dashboard. The options include multiple foreign languages ​​(e.g., Japanese, French).

[0899] Virtual reality space generation

[0900] The device retrieves language data from the server based on the user's selection. During this process, the device sends a request to the server (e.g., a cloud service), which returns information about the user's selected language from a database (e.g., MySQL). Using the retrieved data, the device generates a virtual reality space using the Unity engine.

[0901] Interaction in virtual reality space

[0902] The user enters the generated virtual reality space and selects various scenarios (e.g., shopping at a store, ordering at a cafe). The user's avatar is displayed and the user interacts with the virtual character. The user's voice is collected by the VR device's microphone and analyzed in real time.

[0903] Speech and emotion data analysis

[0904] The device sends the user's voice data to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converts it into text. The converted text data is sent to a server, which runs a script to evaluate the content. The server then awards points based on the evaluation results and returns the information to the device.

[0905] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. For example, it uses emotion recognition technology (e.g., Affectiva's SDK) to recognize emotions from facial expressions and transmits the emotion data to the server. The server then dynamically adjusts the difficulty of the learning content and the content of the feedback based on this data.

[0906] Summarizing and evaluating newspaper articles

[0907] The user reads a newspaper article displayed in the virtual reality space and summarizes its contents. The created summary is sent to the server via the device. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summary and award points. During this process, the emotion engine monitors the user's emotions and provides appropriate feedback and adjusts the points.

[0908] Points Redemption and Instructor Sessions

[0909] Users can use their earned points to book a simulated session with a real-time instructor. Once the reservation is confirmed, the server connects the instructor and the user via video chat at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[0910] Examples of concrete examples and prompts

[0911] For example, if a user wants to learn Japanese and French in parallel, they can follow this process:

[0912] 1. The user logs into the system, selects Japanese and French, and begins a conversation in Japanese in the virtual town.

[0913] 2. Once the dialogue is complete, place your order in French at a French cafe.

[0914] 3. During this process, the emotion engine detects the user's stress, and the server automatically adjusts the learning content.

[0915] 4. The user reads the newspaper article in the virtual reality space and sends the summary to the server.

[0916] 5. The server evaluates the summary and awards points.

[0917] 6. Finally, users can use their points to book a simulated session with a real-time instructor, during which the emotion engine monitors the user's emotions and provides real-time feedback to the instructor.

[0918] Prompt Sentence Examples

[0919] "I want to learn Japanese and French at the same time in a VR space. Please simulate shopping in a store in Japanese and ordering at a cafe in French. Also, please use an emotion engine to adjust the learning feedback."

[0920] This system allows users to efficiently learn multiple languages ​​in a realistic virtual environment. It also utilizes an emotion engine to provide the optimal learning environment for each user.

[0921] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0922] Step 1:

[0923] The user puts on the VR device and enters their user ID and password on the login screen. Once the input is complete and the authentication server verifies the authentication information and permits login, the user can access the dashboard. This allows the user to proceed to the next step.

[0924] Input: User ID, Password

[0925] Output: User authentication status (login status)

[0926] Step 2:

[0927] Users select the foreign language they want to learn on the dashboard. Once the system receives the user's selection, it sends that information to the server, which retrieves data about the selected language from a database and sends it back to the device, which then loads content related to the language the user wants to learn.

[0928] Input: Selected foreign language

[0929] Output: Data related to the selected foreign language

[0930] Step 3:

[0931] The device receives the language data sent from the server and generates a virtual reality space. At this time, the device uses a game engine such as the Unity engine to create the graphics and scenario for the virtual space. This allows the user to access a virtual reality space that corresponds to the selected language.

[0932] Input: Language data sent from the server

[0933] Output: Generated virtual reality space

[0934] Step 4:

[0935] The user enters the generated virtual reality space and selects a specific scenario (e.g., shopping at a store, ordering at a cafe). Once the selection is complete, a conversation with a virtual character based on the scenario begins. The user's speech is collected by the microphone of the VR device.

[0936] Input: Select a specific scenario

[0937] Output: Start of dialogue with virtual character

[0938] Step 5:

[0939] The device analyzes the user's voice data in real time. The voice data is sent to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converted into text data. The converted text data is sent to a server, which then runs a script to evaluate the content. Points are awarded based on the evaluation results.

[0940] Input: User's voice data

[0941] Output: Evaluation results and points awarded

[0942] Step 6:

[0943] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. The emotion data is sent to a server, which then uses this data to perform calculations to adjust the difficulty of the learning content and the content of the feedback. This provides an optimal learning environment according to the user's emotional state.

[0944] Input: User's facial expressions and voice tone

[0945] Output: Adjustment of learning content and feedback

[0946] Step 7:

[0947] Users read newspaper articles displayed in a virtual reality space and create summaries. The summaries are then sent to a server via their devices. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summaries, and points are awarded based on the evaluation results.

[0948] Input: User-generated abstract

[0949] Output: Evaluation results and points awarded

[0950] Step 8:

[0951] Users can use their earned points to book a simulated session with a real-time instructor. The reservation information is sent to the server, which then connects the instructor and user via a video chat platform at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[0952] Input: Reservation information and points

[0953] Output: Connection with instructor and real-time feedback

[0954] (Application example 2)

[0955] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0956] Conventional language learning systems do not provide sufficient support for users to efficiently learn multiple foreign languages, and it is difficult to provide an optimal learning experience that reflects the user's emotional state in real time while learning.

[0957] Furthermore, the lack of a virtual store customer service simulation as a practical application environment for language learning makes it difficult to implement practical learning that takes into account real-world applications. To solve these problems, a system is needed that can grasp the user's emotional state in real time and adjust the learning content based on that, while providing a virtual experience that is close to real-life situations.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0959] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the evaluation obtained by the user, means for reserving a simulated session with a real-time instructor using the points, means for recognizing the user's emotional state and analyzing the data to optimize the learning experience, and means for the user to simulate customer service in a foreign language in a virtual store. This enables a user to efficiently acquire multiple foreign languages ​​through practical language use scenarios while receiving learning content personalized in real time according to their emotional state.

[0960] "User" refers to an individual who uses the system to learn a foreign language.

[0961] "Multiple foreign languages" refers to multiple different languages ​​that a user can select at will to learn.

[0962] "Virtual reality space" refers to a computer-generated simulated environment that a user experiences through a VR device.

[0963] "Utterance" refers to verbal communication that the user makes while studying.

[0964] "Evaluation" refers to analyzing the content of a user's utterance and converting it into a score based on its accuracy and appropriateness.

[0965] "Points" refer to scores or rewards based on evaluations that users obtain through learning activities.

[0966] "Real-time instructor" refers to a human instructor who provides direct instruction to a user through the system.

[0967] "Simulated session" refers to a simulated learning or interaction session within a virtual environment.

[0968] "Emotional state" refers to the psychological state of the user that is analyzed from facial expressions, tone of voice, and the like.

[0969] "Virtual store" refers to an environment set in virtual reality that resembles a physical store.

[0970] "Customer service simulation" refers to a learning activity in which a user interacts as a store clerk or a customer in a virtual store, simulating a scenario in which the user provides or receives service.

[0971] The present invention is a system that allows users to efficiently learn multiple foreign languages, and provides a more personalized learning experience by combining a VR-based virtual reality space with an emotion engine that analyzes the user's emotional state. A specific embodiment of this system will be described below.

[0972] Overall structure

[0973] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[0974] User Interface

[0975] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[0976] Virtual reality space generation

[0977] The terminal acquires language data from the server based on the user's selection and generates a virtual reality space. Various scenarios are set in this space, and the user can perform specific tasks (e.g., customer service simulations in a virtual store) within the space. Specifically, the system provides scenarios for the user to ask for product explanations or place orders as a customer.

[0978] Language Learning Activities

[0979] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[0980] Emotion Engine Functions

[0981] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. The emotion engine used can be, for example, the Affectiva SDK.

[0982] Points and real instructors

[0983] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[0984] Hardware and software used

[0985] Hardware:

[0986] VR device (e.g., Oculus Rift): Used to display the virtual space and capture user input.

[0987] Emotion engine (e.g., Affectiva SDK): Recognizes emotions by analyzing the user's facial expression data.

[0988] software:

[0989] Language processing engine (e.g., Google Cloud Translation API): Analyzes the user's speech and translates it into the foreign language of their choice.

[0990] Database: Stores and manages users' learning progress and points.

[0991] Specific examples

[0992] For example, in a scenario where a user is a salesperson in a virtual store explaining a product in French, the system would prompt the user with questions such as, "What are the features of this product?" The user would respond to the question in French, and their responses would be analyzed and evaluated. If the emotion engine detects the user becoming confused or stressed, the system would adjust its learning and provide clearer explanations or additional support.

[0993] Prompt Sentence Examples

[0994] "As a customer in a virtual store in a VR space, ask for product details in French."

[0995] "Use an emotion engine to analyze user emotions and optimize the learning experience."

[0996] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0997] Step 1: User Login and Language Selection

[0998] The user puts on the VR device and logs in to the system. The system receives the user's ID and password as input and authenticates them. If authentication is successful, the user selects one or more foreign languages ​​to learn. The selected language data is sent to the server.

[0999] Step 2: Creating a virtual reality space

[1000] The server acquires language data based on the user's selection and sends information to the terminal (VR device) to generate a virtual reality space. The terminal generates the virtual reality space and displays it to the user. At this time, scenarios and tasks within the virtual space (for example, a customer service simulation in a virtual store) are also set.

[1001] Step 3: User begins interacting with virtual character

[1002] The user begins to interact with a virtual character in a virtual reality space. The virtual character presents the user with a predefined prompt, such as "Please tell me about the features of this product." The VR device captures the user's voice and sends it to the server as audio data.

[1003] Step 4: Analyzing and evaluating user voice

[1004] The server analyzes the received voice data using a language processing engine (for example, Google Cloud Translation API). As a result of the analysis, the voice input content is converted into text data and evaluated. This evaluation is based on criteria such as accuracy, appropriateness, and pronunciation. The evaluation results are awarded to the user as points.

[1005] Step 5: Emotion analysis using the emotion engine

[1006] Facial expression data and voice tone data captured during user interaction are analyzed using an emotion engine (e.g., Affectiva SDK). The emotion engine recognizes the user's emotional state and sends the results to the server. The server uses this data to determine the user's level of fatigue or stress.

[1007] Step 6: Adjust your learning

[1008] The server then adjusts the user's learning based on the analyzed emotional data, adjusting the level of feedback and explanation provided to the user. For example, if the user is confused, the server will provide clearer explanations and additional support.

[1009] Step 7: Use points and simulate a real-time instructor session

[1010] Users use points earned in the virtual reality space to reserve a simulated session with a real-time instructor. The server checks point balances and manages reservations. Once a reservation is confirmed, a connection is established between the instructor and the user at the specified time. During the session, the emotion engine monitors the user's state and provides real-time feedback to the instructor.

[1011] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1012] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1013] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1014] [Third embodiment]

[1015] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1016] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1018] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1019] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1020] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1022] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1023] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1025] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1026] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1027] The present invention relates to a virtual reality (VR)-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[1028] Overall structure

[1029] This system consists of a user, a terminal (a user's VR device), and a server. The user accesses the system using the VR device, and the server manages and processes the entire data.

[1030] User Interface

[1031] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1032] Virtual reality space generation

[1033] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[1034] Language Learning Activities

[1035] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1036] Newspaper article summaries

[1037] Another activity is to allow users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points.

[1038] Points and real instructors

[1039] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[1040] Specific examples

[1041] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and interact with a virtual store clerk in Japanese. After completing the interaction, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and fluency of their use of both languages.

[1042] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[1043] According to the embodiments of the present invention, a user can learn multiple foreign languages ​​in a realistic environment, enabling more efficient language acquisition compared to conventional learning methods.

[1044] The processing flow will be explained below.

[1045] Step 1:

[1046] The user puts on the VR device, starts the system, and the login screen appears.

[1047] The terminal displays the user's login information in an input form and prompts them to enter it.

[1048] Step 2:

[1049] The user enters their username and password and clicks the login button.

[1050] The terminal receives the input and sends the login information to the server.

[1051] Step 3:

[1052] The server checks the received login information against its database and authenticates it.

[1053] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[1054] Step 4:

[1055] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[1056] Users select 3-5 languages ​​they want to learn and press the next button.

[1057] Step 5:

[1058] The terminal transmits the user's selection to the server.

[1059] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[1060] Step 6:

[1061] After selecting a language, the user will be taken to the avatar customization screen.

[1062] The device will display the avatar settings options.

[1063] Users customize their avatar's appearance and complete the setup.

[1064] The device sends the set avatar to the server.

[1065] Step 7:

[1066] The server stores the received avatar information and generates the user's avatar in the VR space.

[1067] Step 8:

[1068] The user selects the first activity (e.g., shopping) in the VR space and starts.

[1069] The terminal requests scenario data for a shopping activity from the server.

[1070] The server transmits the corresponding scenario data to the terminal.

[1071] Step 9:

[1072] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[1073] The user speaks through a microphone and answers questions posed by the virtual character.

[1074] Step 10:

[1075] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[1076] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[1077] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[1078] Step 11:

[1079] The terminal displays the points and the evaluation results to the user.

[1080] Step 12:

[1081] The user selects the "read newspaper article" activity and presses the activity start button.

[1082] The terminal transmits activity selection information to the server.

[1083] The server selects the appropriate newspaper article and sends it to the terminal.

[1084] Step 13:

[1085] The terminal displays the newspaper article on the screen for the user to view.

[1086] The user reads the article and enters a summary in the input field.

[1087] Step 14:

[1088] After the user has finished entering the summary, he or she presses the send button.

[1089] The terminal transmits the summary data to the server.

[1090] Step 15:

[1091] The server reviews the summaries, rates them for accuracy, and calculates points.

[1092] The evaluation and point calculation results are sent back to the terminal.

[1093] Step 16:

[1094] The terminal displays the evaluation results and points to the user.

[1095] Step 17:

[1096] Users can use the points they earn to book sessions with instructors.

[1097] The terminal displays the reservation request and prompts the user for input.

[1098] Step 18:

[1099] The user enters the date, time and point information and presses the reservation button.

[1100] The terminal sends a reservation request to the server.

[1101] Step 19:

[1102] The server checks the instructor's schedule against the user's request and confirms the reservation.

[1103] The server returns reservation confirmation information to the terminal.

[1104] Step 20:

[1105] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[1106] Users can directly talk with instructors to deepen their learning.

[1107] Example 1

[1108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1109] Traditional language learning methods often involve the use of textbooks and vocabulary books, making it difficult to efficiently improve actual communication skills. Furthermore, learning multiple foreign languages ​​at once can lead to reduced learning efficiency. Furthermore, to practice actual conversations, students must travel to the location or schedule sessions with specific instructors, which can be time-consuming and costly. There is a need to solve these problems and provide a more efficient and practical language learning environment.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1111] In this invention, the server includes means for allowing a user to select multiple languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's voice, means for assigning a score based on the evaluation obtained by the user, and means for booking a simulated session with a real-time educator using the score, thereby enabling a user to efficiently learn multiple languages ​​through practical language activities in the virtual reality space and to use the acquired score to participate in a session with a real-time educator.

[1112] A "user" is someone who operates the system and learns a language.

[1113] "Language" refers to multiple foreign languages ​​selected by the User for the purpose of learning and using them.

[1114] "Selection mechanisms" refers to interfaces or features that allow users to choose the language they want to learn from multiple options.

[1115] "Virtual reality space" refers to a three-dimensional virtual environment that a user can access through a VR device.

[1116] "Means of generation" refers to the software and hardware functions for constructing and displaying virtual reality spaces.

[1117] "Means for analyzing and evaluating speech" refers to algorithms or systems that convert a user's speech into text using speech recognition technology and evaluate the content of that text.

[1118] "Score" refers to a score given based on a user's learning activities and evaluation results.

[1119] "Means for awarding" refers to the system or algorithm used to assign points to users based on their evaluation results.

[1120] "Real-time educator" refers to a lecturer or trainer with whom a user interacts during learning, either through virtual reality or in person.

[1121] "Simulated Session" refers to a virtual learning session conducted in real time between a user and an educator.

[1122] "Means for Booking" refers to an interface or system that allows a user to book a session with a real-time educator using a score.

[1123] "Virtual character" refers to an animated character that is displayed for a user to interact with within a virtual reality space.

[1124] "News article" refers to information or news articles that a user can view within a virtual reality space.

[1125] "Summary" refers to text or speech that the user has written in a concise summary of the contents of a newspaper article.

[1126] "Means for evaluation" refers to the system or algorithm that analyzes the content of the summary entered by the user and determines its quality and accuracy.

[1127] MODE FOR CARRYING OUT THE INVENTION

[1128] The present invention relates to a VR-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[1129] Hardware and software used

[1130] This system consists of a user, a terminal (the user's VR device), and a server. The specific hardware and software used are as follows:

[1131] Hardware:

[1132] VR device (e.g. Oculus Quest 2)

[1133] software:

[1134] VR applications (e.g. apps developed with Unity)

[1135] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1136] Database system (e.g. MySQL)

[1137] User operations

[1138] The user puts on the VR device, launches a dedicated application, and logs into the system. After logging in, the user selects the language they want to learn and enters the virtual reality space. Based on the selected language, the server sends the corresponding language data to the device.

[1139] Virtual reality space generation

[1140] The device uses the Unity engine to generate a virtual reality space based on the language data received from the server. This space contains virtual characters and specific scenarios (e.g., virtual towns, cafes), and is constructed so that users can carry out activities.

[1141] Language Learning Activities

[1142] Users interact with characters in a virtual reality space. During the interaction, the user's voice is collected by the device's microphone and converted into text in real time using a speech recognition API. The converted text data is sent to a server, which evaluates the content and assigns a score. This score is used to visualize the user's evaluation and progress.

[1143] Newspaper article summaries

[1144] As another learning activity, users can read newspaper articles in a virtual reality space. After reading the article, they summarize the content and send the summary data to a server via their device. The server uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and assign an appropriate score.

[1145] Using points and sessions with real instructors

[1146] The points earned by users can be used to book a simulated session with a real-time educator. Users use their points to book a session with a teacher through the server, and once the reservation is confirmed, a connection is established at the specified time. Users can converse directly with the teacher and receive feedback on their pronunciation and grammar.

[1147] Examples of concrete examples and prompts

[1148] For example, if a user is learning Japanese and French, they log in to the system through a VR device, select Japanese or French, and then enter the virtual reality space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. They then switch to a scenario where they order at a French cafe and place their order in French. Through this series of activities, the user naturally improves their usage frequency and fluency in both languages.

[1149] (Example of a prompt)

[1150] "If a user wants to learn Japanese and French, they log in through a VR device and go through a shopping scenario in a virtual town in Japanese and a cafe ordering scenario in French. They can then use their points to book a session with a real-time instructor to further their learning."

[1151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1152] Step 1:

[1153] Logging in to the system

[1154] Input: The user puts on the VR device and enters their ID and password to access the system.

[1155] Specific behavior:

[1156] The user puts on the VR device, launches the dedicated application, enters their ID and password on the login screen, and presses the "Login" button.

[1157] Data processing and calculation:

[1158] The terminal sends the entered ID and password to the server, which checks the database and, if authentication is successful, sends the user information back to the terminal.

[1159] Output: A successful authentication message that allows the user to access the main menu.

[1160] Step 2:

[1161] Selecting a language to learn

[1162] Input: The user selects the language they want to learn.

[1163] Specific behavior:

[1164] The user selects the language they want to learn (e.g., Japanese, French) from the main menu and performs the selection using the VR controller.

[1165] Data processing and calculation:

[1166] The terminal transmits the selected language data to the server, and the server prepares the corresponding language data and learning content and transmits them to the terminal.

[1167] Output: The learning content is displayed according to the user's language selection.

[1168] Step 3:

[1169] Virtual reality space generation

[1170] Input: Language data and scenario information sent by the server to the device.

[1171] Specific behavior:

[1172] The device uses the Unity engine to generate a virtual reality space based on the received language data and scenario information, placing textures and objects according to the scenario and generating a user avatar.

[1173] Data processing and calculation:

[1174] The device loads and renders 3D models and textures for the virtual space based on language data and scenario information.

[1175] Output: The virtual reality space is displayed on the user's VR device.

[1176] Step 4:

[1177] Conducting language learning activities

[1178] Input: User's voice input.

[1179] Specific behavior:

[1180] The user controls an avatar and interacts with virtual store clerks and characters. During the interaction, the user's voice is collected by the device's microphone.

[1181] Data processing and calculation:

[1182] The device converts the collected voice data into text in real time using a speech recognition API, and sends the converted text data to a server, which evaluates the content and calculates a score.

[1183] Output: The server evaluates the score and gives it back to the user.

[1184] Step 5:

[1185] Newspaper article summary activity

[1186] Input: A newspaper article that the user reads in a virtual reality space.

[1187] Specific behavior:

[1188] The user moves to a designated location in the virtual reality space and reads a newspaper article. After reading the article, the user summarizes the content and inputs it into the terminal.

[1189] Data processing and calculation:

[1190] The device sends the summary data to a server, which uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and calculate a score.

[1191] Output: The server evaluates the summary and gives the score as feedback to the user.

[1192] Step 6:

[1193] Book a session with an educator in real time

[1194] Input: User-specified desired date and time and session reservation information with the educator.

[1195] Specific behavior:

[1196] The user selects the "Book a session with an instructor" option from the main menu, chooses the desired date and time and instructor, and makes the reservation using points.

[1197] Data processing and calculation:

[1198] The reservation information is sent from the terminal to the server, and the server registers the information in the reservation system. The reservation is confirmed in the instructor's schedule management system.

[1199] Output: The user is provided with a reservation confirmation message and information on how to establish a connection at the specified time.

[1200] In this way, users can efficiently learn a language through specific input, processing, and output at each step.

[1201] (Application example 1)

[1202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1203] Conventional language learning systems have the drawback of providing users with few opportunities to practically learn the language actually used in the local area, making it difficult to acquire the high level of language proficiency required for real-life situations. There is also concern that the learning content may become monotonous, reducing user motivation. Furthermore, there are limited means for learning while receiving real-time feedback, which often results in insufficient learning efficiency.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1205] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the user's evaluation, means for reserving a real-time simulated session with a teacher using the points, and means for the user to learn a foreign language by selecting products and completing purchase and sale procedures in the virtual space. This allows the user to efficiently learn a language in an environment similar to real-life situations.

[1206] A "user" is someone who seeks to learn a language using the virtual reality-based language learning system of the present invention.

[1207] A "means for enabling selection" is a device or software that provides an interface and functionality for allowing a user to select the language they wish to learn from multiple foreign languages.

[1208] "Means for generating virtual reality spaces" refers to technologies and systems that create a virtual reality learning environment based on the language selected by the user.

[1209] "Means for analyzing and evaluating speech" refers to technology for analyzing a user's voice data and evaluating its content, specifically a system that uses voice recognition and natural language processing.

[1210] The "means for awarding points" is a mechanism for awarding reward points to a user based on an evaluation of the user's use of language.

[1211] The "means for reserving a simulated session with a real-time instructor" is a mechanism by which a user can use the points they have earned to make a reservation for a real-time language learning session with a real instructor.

[1212] The "means of selecting products in a virtual space and carrying out the buying and selling procedure" is a simulation function that allows users to select products in a virtual reality space and learn a language through that selection.

[1213] "Means for displaying a virtual character" refers to a technique or device for displaying a character for interaction with a user in a virtual reality space.

[1214] The "means for reading newspaper articles and inputting summaries of their contents" refers to an interface and technology that allows a user to view newspaper articles in a virtual reality space and input a summary of the main points.

[1215] The "means of evaluation" is a mechanism for evaluating a user's language proficiency based on the summary and dialogue content entered by the user.

[1216] The present invention relates to a system that allows users to efficiently learn multiple foreign languages ​​using VR (virtual reality). The overall configuration of the system consists of a server, a terminal (a user's VR device), and a user. Specific embodiments of the system are described below.

[1217] Overall structure

[1218] The system begins when the user accesses the server using a VR device, selects the foreign language they want to learn, and logs in. The server manages and processes all data.

[1219] User Interface

[1220] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1221] Virtual reality space generation

[1222] The device retrieves language data from the server based on the user's selection and generates a virtual reality space containing various scenarios in which the user can perform specific tasks (e.g., product selection and purchase procedures).

[1223] Language Learning Activities

[1224] Users interact with virtual characters in a virtual reality space. During the interaction, the device analyzes the user's speech and sends the data to the server. The server evaluates the speech and awards the user points. The user can use these points to challenge themselves to more difficult activities.

[1225] Product Selection Scenario

[1226] Users learn the language through a scenario in which they select and purchase products in a virtual store. They ask questions about the products in the local language and interact with a virtual character acting as a store clerk. Through this process, users can acquire vocabulary and phrases necessary for shopping and everyday conversation.

[1227] Newspaper article summaries

[1228] Another activity involves users reading local newspaper articles in the virtual reality space and summarizing their content, which is then sent to the server, which then evaluates and awards points.

[1229] Simulated sessions with real-time instructors

[1230] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[1231] Hardware and software used

[1232] Hardware

[1233] VR devices (e.g. Oculus Rift, HTC Vive)

[1234] A server (e.g., a Flask application server)

[1235] software

[1236] Speech recognition software (e.g., the SpeechRecognition Python library)

[1237] Translation API (e.g. Google Translate API)

[1238] Specific examples

[1239] For example, if a user wants to learn Japanese and French at the same time, they would follow the following steps: First, the user logs in to the system and selects Japanese and French as the languages ​​they want to learn. Next, the user starts a shopping scenario in a virtual store and selects products while interacting with the clerk in Japanese. Next, the user switches to a scenario where they are ordering at a French cafe and places their order in French.

[1240] Example prompts for generative AI models

[1241] "Design a scenario where a user can select products in a virtual store and have a conversation with a local sales associate."

[1242] "Create an algorithm that analyzes the user's voice and rates them based on a points system."

[1243] This allows users to efficiently improve their language skills in an environment that is close to the real world, and allows them to naturally acquire language skills in actual situations.

[1244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1245] Step 1:

[1246] The user puts on the VR device and logs in to the system.

[1247] Input: The user enters login information into the VR device.

[1248] Data processing / calculation: The terminal sends the login information to the server for authentication. The server then verifies the user's authentication information by checking it against a database.

[1249] Output: If authentication is successful, the user's account information and learning history will be loaded.

[1250] Step 2:

[1251] The user selects the foreign language they want to learn.

[1252] Input: The user selects the language they wish to learn through the system's interface.

[1253] Data processing / calculation: The device receives the user's selection and sends the information to the server. The server retrieves the data in the selected language and prepares the virtual reality space.

[1254] Output: The virtual reality space is now set up and ready to be displayed to the user.

[1255] Step 3:

[1256] The device generates a virtual reality space.

[1257] Input: Language data obtained from the server and setting information for the virtual reality space.

[1258] Data processing / calculation: The terminal generates a virtual reality space based on the data received from the server.

[1259] Output: The generated virtual reality space is displayed on the user's VR device.

[1260] Step 4:

[1261] The user selects a product in the virtual reality space and completes the transaction process.

[1262] Input: Product information provided in the virtual reality space and user selection operations.

[1263] Data processing / calculation: When a user selects a product and starts a dialogue, the device collects the user's speech as voice data and sends it to the server, which analyzes the voice data and evaluates its content.

[1264] Output: The evaluation results are given to the user as points. The user receives feedback based on their learning status.

[1265] Step 5:

[1266] A user interacts with a virtual character in a virtual reality space.

[1267] Input: User utterances and virtual character configuration information.

[1268] Data processing / calculation: The device analyzes the user's speech in real time and generates appropriate responses from the virtual character. The server continuously records the dialogue and manages it as learning data.

[1269] Output: The interaction with the virtual character progresses and the user receives continuous feedback.

[1270] Step 6:

[1271] A user reads a newspaper article and enters a summary of it.

[1272] Input: Newspaper article provided in virtual reality space and user summary input.

[1273] Data processing / calculation: When a user inputs a summary, the terminal sends the content to the server, which evaluates the summary and assigns points based on the analysis results.

[1274] Output: Points are awarded and the user confirms the evaluation results.

[1275] Step 7:

[1276] Users use points to book simulated sessions with real-time instructors.

[1277] Input: Points acquired by the user and booking actions.

[1278] Data processing / calculation: The server checks the user's point balance and displays a list of available instructors. When a user reserves a session with an instructor, the server manages the connection time and instructor information and confirms the reservation.

[1279] Output: The reservation is confirmed and the user is notified of the reservation details. When the session time arrives, the device establishes a connection with the designated instructor and begins learning in real time.

[1280] In this way, through specific actions and data flows at each step, a system is realized that allows users to efficiently learn a foreign language in a virtual reality space.

[1281] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1282] The present invention combines a VR (virtual reality)-based language learning system that enables users to efficiently learn multiple foreign languages ​​with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[1283] Overall structure

[1284] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[1285] User Interface

[1286] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1287] Virtual reality space generation

[1288] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[1289] Language Learning Activities

[1290] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1291] Emotion Engine Functions

[1292] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. This makes the user's learning experience more personalized and efficient.

[1293] Newspaper article summaries

[1294] Another activity is for users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points. During this process, the emotion engine also analyzes the user's emotions and provides appropriate feedback and adjusts the points.

[1295] Points and real instructors

[1296] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[1297] Specific examples

[1298] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. After completing the conversation, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and spontaneity of their use of both languages. In addition, the emotion engine analyzes the user's emotional state in real time and provides appropriate feedback, further improving learning efficiency.

[1299] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[1300] According to the embodiment of the present invention, users can learn multiple foreign languages ​​in a realistic environment, which allows for more efficient language acquisition than conventional learning methods. Furthermore, by effectively utilizing the emotion engine, users can enjoy an optimal learning experience that is tailored to their emotional state.

[1301] The processing flow will be explained below.

[1302] Step 1:

[1303] The user puts on the VR device, starts the system, and the login screen appears.

[1304] The terminal displays the user's login information in an input form and prompts them to enter it.

[1305] Step 2:

[1306] The user enters their username and password and clicks the login button.

[1307] The terminal receives the input and sends the login information to the server.

[1308] Step 3:

[1309] The server checks the received login information against its database and authenticates it.

[1310] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[1311] Step 4:

[1312] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[1313] Users select 3-5 languages ​​they want to learn and press the next button.

[1314] Step 5:

[1315] The terminal transmits the user's selection to the server.

[1316] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[1317] Step 6:

[1318] After selecting a language, the user will be taken to the avatar customization screen.

[1319] The device will display the avatar settings options.

[1320] Users customize their avatar's appearance and complete the setup.

[1321] The device sends the set avatar to the server.

[1322] Step 7:

[1323] The server stores the received avatar information and generates the user's avatar in the VR space.

[1324] Step 8:

[1325] The user selects the first activity (e.g., shopping) in the VR space and starts.

[1326] The terminal requests scenario data for a shopping activity from the server.

[1327] The server transmits the corresponding scenario data to the terminal.

[1328] Step 9:

[1329] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[1330] The user speaks through a microphone and answers questions posed by the virtual character.

[1331] Step 10:

[1332] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[1333] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[1334] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[1335] Step 11:

[1336] The terminal displays the points and the evaluation results to the user.

[1337] Step 12:

[1338] The user selects the "read newspaper article" activity and presses the activity start button.

[1339] The terminal transmits activity selection information to the server.

[1340] The server selects the appropriate newspaper article and sends it to the terminal.

[1341] Step 13:

[1342] The terminal displays the newspaper article on the screen for the user to view.

[1343] The user reads the article and enters a summary in the input field.

[1344] Step 14:

[1345] After the user has finished entering the summary, he or she presses the send button.

[1346] The terminal transmits the summary data to the server.

[1347] Step 15:

[1348] The server reviews the summaries, rates them for accuracy, and calculates points.

[1349] The evaluation and point calculation results are sent back to the terminal.

[1350] Step 16:

[1351] The terminal displays the evaluation results and points to the user.

[1352] Step 17:

[1353] Users can use the points they earn to book sessions with instructors.

[1354] The terminal displays the reservation request and prompts the user for input.

[1355] Step 18:

[1356] The user enters the date, time and point information and presses the reservation button.

[1357] The terminal sends a reservation request to the server.

[1358] Step 19:

[1359] The server checks the instructor's schedule against the user's request and confirms the reservation.

[1360] The server returns reservation confirmation information to the terminal.

[1361] Step 20:

[1362] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[1363] Users can directly talk with instructors to deepen their learning.

[1364] Step 21:

[1365] While the user is active in the VR space, the device uses an emotion engine to analyze the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine will detect this and send it to the server.

[1366] Step 22:

[1367] The server adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. For example, if the emotion engine determines that the user is tired, it will switch to a lighter task or display an encouraging message.

[1368] Step 23:

[1369] When a user reads a newspaper article and summarizes it, the device uses an emotion engine to analyze the user's emotional state. For example, if the user is feeling stressed, the device may take an approach such as adjusting the evaluation criteria for summarizing.

[1370] Step 24:

[1371] During a simulated session with a real-time instructor, the device uses an emotion engine to monitor the user's emotional state. For example, if the user is nervous, the emotion engine provides this information to the instructor in real time, allowing the instructor to provide appropriate feedback and support.

[1372] This provides a learning experience that takes the user's emotions into consideration, maximizing learning efficiency and effectiveness.

[1373] Example 2

[1374] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1375] Conventional language learning systems have the problem of being unable to respond to individual learning needs due to insufficient feedback on users' learning progress and emotional state. In particular, they provide uniform learning content without considering users' emotions, making it difficult to achieve efficient learning. Furthermore, they are unable to effectively evaluate users' learning outcomes and reflect the results in the learning process, making it difficult to maintain motivation to learn.

[1376] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1377] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to study, means for generating a virtual reality space based on the user's selection, means for displaying a virtual character with which the user can interact in the virtual reality space, means for analyzing and evaluating the user's speech, an emotion engine that includes an analysis of the user's emotional state and means for adjusting the difficulty level of the learning content and the feedback content based on the emotional state, means for awarding points based on the evaluation obtained by the user, and means for booking a simulated session with a real-time instructor using the points. This makes it possible to provide a personalized learning experience based on the user's learning progress and emotional state.

[1378] "User" refers to an individual or organization that uses the system to learn a foreign language.

[1379] "Virtual reality space" refers to a digital environment different from the real world that users can enter through VR devices.

[1380] "Virtual character" refers to a digital character that is displayed to interact with a user within a virtual reality space.

[1381] "Utterance" refers to the verbal expressions used by a user when interacting with virtual characters and other users in a virtual reality space.

[1382] "Emotion engine" refers to a software or hardware system that monitors a user's facial expressions and vocal tone to analyze the user's emotional state.

[1383] "Emotional state" refers to the psychological and emotional state, such as stress, fatigue, or satisfaction, that a user experiences while using a system.

[1384] "Evaluation" refers to the analysis and scoring of a user's speech and activity by the server.

[1385] "Points" refer to rewards or scores awarded based on a user's learning outcomes.

[1386] A "simulated session" refers to a virtual online learning session between a real-time instructor and a user.

[1387] The present invention combines a VR-based language learning system that enables users to efficiently learn a foreign language with an emotion engine that recognizes the user's emotions. This system consists of a user, a terminal (the user's VR device), a server, and an emotion engine. Specific embodiments of this system are described below.

[1388] User Login and Language Selection

[1389] The user puts on a VR device (e.g., Oculus Rift or HTC Vive) and connects to the provided login screen. By entering their user ID and password, they are granted access to the system. Then, the user selects the foreign language they wish to learn on the dashboard. The options include multiple foreign languages ​​(e.g., Japanese, French).

[1390] Virtual reality space generation

[1391] The device retrieves language data from the server based on the user's selection. During this process, the device sends a request to the server (e.g., a cloud service), which returns information about the user's selected language from a database (e.g., MySQL). Using the retrieved data, the device generates a virtual reality space using the Unity engine.

[1392] Interaction in virtual reality space

[1393] The user enters the generated virtual reality space and selects various scenarios (e.g., shopping at a store, ordering at a cafe). The user's avatar is displayed and the user interacts with the virtual character. The user's voice is collected by the VR device's microphone and analyzed in real time.

[1394] Speech and emotion data analysis

[1395] The device sends the user's voice data to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converts it into text. The converted text data is sent to a server, which runs a script to evaluate the content. The server then awards points based on the evaluation results and returns the information to the device.

[1396] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. For example, it uses emotion recognition technology (e.g., Affectiva's SDK) to recognize emotions from facial expressions and transmits the emotion data to the server. The server then dynamically adjusts the difficulty of the learning content and the content of the feedback based on this data.

[1397] Summarizing and evaluating newspaper articles

[1398] The user reads a newspaper article displayed in the virtual reality space and summarizes its contents. The created summary is sent to the server via the device. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summary and award points. During this process, the emotion engine monitors the user's emotions and provides appropriate feedback and adjusts the points.

[1399] Points Redemption and Instructor Sessions

[1400] Users can use their earned points to book a simulated session with a real-time instructor. Once the reservation is confirmed, the server connects the instructor and the user via video chat at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[1401] Examples of concrete examples and prompts

[1402] For example, if a user wants to learn Japanese and French in parallel, they can follow this process:

[1403] 1. The user logs into the system, selects Japanese and French, and begins a conversation in Japanese in the virtual town.

[1404] 2. Once the dialogue is complete, place your order in French at a French cafe.

[1405] 3. During this process, the emotion engine detects the user's stress, and the server automatically adjusts the learning content.

[1406] 4. The user reads the newspaper article in the virtual reality space and sends the summary to the server.

[1407] 5. The server evaluates the summary and awards points.

[1408] 6. Finally, users can use their points to book a simulated session with a real-time instructor, during which the emotion engine monitors the user's emotions and provides real-time feedback to the instructor.

[1409] Prompt Sentence Examples

[1410] "I want to learn Japanese and French at the same time in a VR space. Please simulate shopping in a store in Japanese and ordering at a cafe in French. Also, please use an emotion engine to adjust the learning feedback."

[1411] This system allows users to efficiently learn multiple languages ​​in a realistic virtual environment. It also utilizes an emotion engine to provide the optimal learning environment for each user.

[1412] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1413] Step 1:

[1414] The user puts on the VR device and enters their user ID and password on the login screen. Once the input is complete and the authentication server verifies the authentication information and permits login, the user can access the dashboard. This allows the user to proceed to the next step.

[1415] Input: User ID, Password

[1416] Output: User authentication status (login status)

[1417] Step 2:

[1418] Users select the foreign language they want to learn on the dashboard. Once the system receives the user's selection, it sends that information to the server, which retrieves data about the selected language from a database and sends it back to the device, which then loads content related to the language the user wants to learn.

[1419] Input: Selected foreign language

[1420] Output: Data related to the selected foreign language

[1421] Step 3:

[1422] The device receives the language data sent from the server and generates a virtual reality space. At this time, the device uses a game engine such as the Unity engine to create the graphics and scenario for the virtual space. This allows the user to access a virtual reality space that corresponds to the selected language.

[1423] Input: Language data sent from the server

[1424] Output: Generated virtual reality space

[1425] Step 4:

[1426] The user enters the generated virtual reality space and selects a specific scenario (e.g., shopping at a store, ordering at a cafe). Once the selection is complete, a conversation with a virtual character based on the scenario begins. The user's speech is collected by the microphone of the VR device.

[1427] Input: Select a specific scenario

[1428] Output: Start of dialogue with virtual character

[1429] Step 5:

[1430] The device analyzes the user's voice data in real time. The voice data is sent to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converted into text data. The converted text data is sent to a server, which then runs a script to evaluate the content. Points are awarded based on the evaluation results.

[1431] Input: User's voice data

[1432] Output: Evaluation results and points awarded

[1433] Step 6:

[1434] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. The emotion data is sent to a server, which then uses this data to perform calculations to adjust the difficulty of the learning content and the content of the feedback. This provides an optimal learning environment according to the user's emotional state.

[1435] Input: User's facial expressions and voice tone

[1436] Output: Adjustment of learning content and feedback

[1437] Step 7:

[1438] Users read newspaper articles displayed in a virtual reality space and create summaries. The summaries are then sent to a server via their devices. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summaries, and points are awarded based on the evaluation results.

[1439] Input: User-generated abstract

[1440] Output: Evaluation results and points awarded

[1441] Step 8:

[1442] Users can use their earned points to book a simulated session with a real-time instructor. The reservation information is sent to the server, which then connects the instructor and user via a video chat platform at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[1443] Input: Reservation information and points

[1444] Output: Connection with instructor and real-time feedback

[1445] (Application example 2)

[1446] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1447] Conventional language learning systems do not provide sufficient support for users to efficiently learn multiple foreign languages, and it is difficult to provide an optimal learning experience that reflects the user's emotional state in real time while learning.

[1448] Furthermore, the lack of a virtual store customer service simulation as a practical application environment for language learning makes it difficult to implement practical learning that takes into account real-world applications. To solve these problems, a system is needed that can grasp the user's emotional state in real time and adjust the learning content based on that, while providing a virtual experience that is close to real-life situations.

[1449] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1450] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the evaluation obtained by the user, means for reserving a simulated session with a real-time instructor using the points, means for recognizing the user's emotional state and analyzing the data to optimize the learning experience, and means for the user to simulate customer service in a foreign language in a virtual store. This enables a user to efficiently acquire multiple foreign languages ​​through practical language use scenarios while receiving learning content personalized in real time according to their emotional state.

[1451] "User" refers to an individual who uses the system to learn a foreign language.

[1452] "Multiple foreign languages" refers to multiple different languages ​​that a user can select at will to learn.

[1453] "Virtual reality space" refers to a computer-generated simulated environment that a user experiences through a VR device.

[1454] "Utterance" refers to verbal communication that the user makes while studying.

[1455] "Evaluation" refers to analyzing the content of a user's utterance and converting it into a score based on its accuracy and appropriateness.

[1456] "Points" refer to scores or rewards based on evaluations that users obtain through learning activities.

[1457] "Real-time instructor" refers to a human instructor who provides direct instruction to a user through the system.

[1458] "Simulated session" refers to a simulated learning or interaction session within a virtual environment.

[1459] "Emotional state" refers to the psychological state of the user that is analyzed from facial expressions, tone of voice, and the like.

[1460] "Virtual store" refers to an environment set in virtual reality that resembles a physical store.

[1461] "Customer service simulation" refers to a learning activity in which a user interacts as a store clerk or a customer in a virtual store, simulating a scenario in which the user provides or receives service.

[1462] The present invention is a system that allows users to efficiently learn multiple foreign languages, and provides a more personalized learning experience by combining a VR-based virtual reality space with an emotion engine that analyzes the user's emotional state. A specific embodiment of this system will be described below.

[1463] Overall structure

[1464] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[1465] User Interface

[1466] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1467] Virtual reality space generation

[1468] The terminal acquires language data from the server based on the user's selection and generates a virtual reality space. Various scenarios are set in this space, and the user can perform specific tasks (e.g., customer service simulations in a virtual store) within the space. Specifically, the system provides scenarios for the user to ask for product explanations or place orders as a customer.

[1469] Language Learning Activities

[1470] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1471] Emotion Engine Functions

[1472] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. The emotion engine used can be, for example, the Affectiva SDK.

[1473] Points and real instructors

[1474] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[1475] Hardware and software used

[1476] Hardware:

[1477] VR device (e.g., Oculus Rift): Used to display the virtual space and capture user input.

[1478] Emotion engine (e.g., Affectiva SDK): Recognizes emotions by analyzing the user's facial expression data.

[1479] software:

[1480] Language processing engine (e.g., Google Cloud Translation API): Analyzes the user's speech and translates it into the foreign language of their choice.

[1481] Database: Stores and manages users' learning progress and points.

[1482] Specific examples

[1483] For example, in a scenario where a user is a salesperson in a virtual store explaining a product in French, the system would prompt the user with questions such as, "What are the features of this product?" The user would respond to the question in French, and their responses would be analyzed and evaluated. If the emotion engine detects the user becoming confused or stressed, the system would adjust its learning and provide clearer explanations or additional support.

[1484] Prompt Sentence Examples

[1485] "As a customer in a virtual store in a VR space, ask for product details in French."

[1486] "Use an emotion engine to analyze user emotions and optimize the learning experience."

[1487] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1488] Step 1: User Login and Language Selection

[1489] The user puts on the VR device and logs in to the system. The system receives the user's ID and password as input and authenticates them. If authentication is successful, the user selects one or more foreign languages ​​to learn. The selected language data is sent to the server.

[1490] Step 2: Creating a virtual reality space

[1491] The server acquires language data based on the user's selection and sends information to the terminal (VR device) to generate a virtual reality space. The terminal generates the virtual reality space and displays it to the user. At this time, scenarios and tasks within the virtual space (for example, a customer service simulation in a virtual store) are also set.

[1492] Step 3: User begins interacting with virtual character

[1493] The user begins to interact with a virtual character in a virtual reality space. The virtual character presents the user with a predefined prompt, such as "Please tell me about the features of this product." The VR device captures the user's voice and sends it to the server as audio data.

[1494] Step 4: Analyzing and evaluating user voice

[1495] The server analyzes the received voice data using a language processing engine (for example, Google Cloud Translation API). As a result of the analysis, the voice input content is converted into text data and evaluated. This evaluation is based on criteria such as accuracy, appropriateness, and pronunciation. The evaluation results are awarded to the user as points.

[1496] Step 5: Emotion analysis using the emotion engine

[1497] Facial expression data and voice tone data captured during user interaction are analyzed using an emotion engine (e.g., Affectiva SDK). The emotion engine recognizes the user's emotional state and sends the results to the server. The server uses this data to determine the user's level of fatigue or stress.

[1498] Step 6: Adjust your learning

[1499] The server then adjusts the user's learning based on the analyzed emotional data, adjusting the level of feedback and explanation provided to the user. For example, if the user is confused, the server will provide clearer explanations and additional support.

[1500] Step 7: Use points and simulate a real-time instructor session

[1501] Users use points earned in the virtual reality space to reserve a simulated session with a real-time instructor. The server checks point balances and manages reservations. Once a reservation is confirmed, a connection is established between the instructor and the user at the specified time. During the session, the emotion engine monitors the user's state and provides real-time feedback to the instructor.

[1502] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1503] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1504] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1505] [Fourth embodiment]

[1506] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1507] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1508] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1509] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1510] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1511] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1512] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1513] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1514] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1517] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1518] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] The present invention relates to a virtual reality (VR)-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[1520] Overall structure

[1521] This system consists of a user, a terminal (a user's VR device), and a server. The user accesses the system using the VR device, and the server manages and processes the entire data.

[1522] User Interface

[1523] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1524] Virtual reality space generation

[1525] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[1526] Language Learning Activities

[1527] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1528] Newspaper article summaries

[1529] Another activity is to allow users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points.

[1530] Points and real instructors

[1531] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[1532] Specific examples

[1533] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and interact with a virtual store clerk in Japanese. After completing the interaction, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and fluency of their use of both languages.

[1534] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[1535] According to the embodiments of the present invention, a user can learn multiple foreign languages ​​in a realistic environment, enabling more efficient language acquisition compared to conventional learning methods.

[1536] The processing flow will be explained below.

[1537] Step 1:

[1538] The user puts on the VR device, starts the system, and the login screen appears.

[1539] The terminal displays the user's login information in an input form and prompts them to enter it.

[1540] Step 2:

[1541] The user enters their username and password and clicks the login button.

[1542] The terminal receives the input and sends the login information to the server.

[1543] Step 3:

[1544] The server checks the received login information against its database and authenticates it.

[1545] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[1546] Step 4:

[1547] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[1548] Users select 3-5 languages ​​they want to learn and press the next button.

[1549] Step 5:

[1550] The terminal transmits the user's selection to the server.

[1551] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[1552] Step 6:

[1553] After selecting a language, the user will be taken to the avatar customization screen.

[1554] The device will display the avatar settings options.

[1555] Users customize their avatar's appearance and complete the setup.

[1556] The device sends the set avatar to the server.

[1557] Step 7:

[1558] The server stores the received avatar information and generates the user's avatar in the VR space.

[1559] Step 8:

[1560] The user selects the first activity (e.g., shopping) in the VR space and starts.

[1561] The terminal requests scenario data for a shopping activity from the server.

[1562] The server transmits the corresponding scenario data to the terminal.

[1563] Step 9:

[1564] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[1565] The user speaks through a microphone and answers questions posed by the virtual character.

[1566] Step 10:

[1567] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[1568] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[1569] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[1570] Step 11:

[1571] The terminal displays the points and the evaluation results to the user.

[1572] Step 12:

[1573] The user selects the "read newspaper article" activity and presses the activity start button.

[1574] The terminal transmits activity selection information to the server.

[1575] The server selects the appropriate newspaper article and sends it to the terminal.

[1576] Step 13:

[1577] The terminal displays the newspaper article on the screen for the user to view.

[1578] The user reads the article and enters a summary in the input field.

[1579] Step 14:

[1580] After the user has finished entering the summary, he or she presses the send button.

[1581] The terminal transmits the summary data to the server.

[1582] Step 15:

[1583] The server reviews the summaries, rates them for accuracy, and calculates points.

[1584] The evaluation and point calculation results are sent back to the terminal.

[1585] Step 16:

[1586] The terminal displays the evaluation results and points to the user.

[1587] Step 17:

[1588] Users can use the points they earn to book sessions with instructors.

[1589] The terminal displays the reservation request and prompts the user for input.

[1590] Step 18:

[1591] The user enters the date, time and point information and presses the reservation button.

[1592] The terminal sends a reservation request to the server.

[1593] Step 19:

[1594] The server checks the instructor's schedule against the user's request and confirms the reservation.

[1595] The server returns reservation confirmation information to the terminal.

[1596] Step 20:

[1597] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[1598] Users can directly talk with instructors to deepen their learning.

[1599] Example 1

[1600] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1601] Traditional language learning methods often involve the use of textbooks and vocabulary books, making it difficult to efficiently improve actual communication skills. Furthermore, learning multiple foreign languages ​​at once can lead to reduced learning efficiency. Furthermore, to practice actual conversations, students must travel to the location or schedule sessions with specific instructors, which can be time-consuming and costly. There is a need to solve these problems and provide a more efficient and practical language learning environment.

[1602] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1603] In this invention, the server includes means for allowing a user to select multiple languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's voice, means for assigning a score based on the evaluation obtained by the user, and means for booking a simulated session with a real-time educator using the score, thereby enabling a user to efficiently learn multiple languages ​​through practical language activities in the virtual reality space and to use the acquired score to participate in a session with a real-time educator.

[1604] A "user" is someone who operates the system and learns a language.

[1605] "Language" refers to multiple foreign languages ​​selected by the User for the purpose of learning and using them.

[1606] "Selection mechanisms" refers to interfaces or features that allow users to choose the language they want to learn from multiple options.

[1607] "Virtual reality space" refers to a three-dimensional virtual environment that a user can access through a VR device.

[1608] "Means of generation" refers to the software and hardware functions for constructing and displaying virtual reality spaces.

[1609] "Means for analyzing and evaluating speech" refers to algorithms or systems that convert a user's speech into text using speech recognition technology and evaluate the content of that text.

[1610] "Score" refers to a score given based on a user's learning activities and evaluation results.

[1611] "Means for awarding" refers to the system or algorithm used to assign points to users based on their evaluation results.

[1612] "Real-time educator" refers to a lecturer or trainer with whom a user interacts during learning, either through virtual reality or in person.

[1613] "Simulated Session" refers to a virtual learning session conducted in real time between a user and an educator.

[1614] "Means for Booking" refers to an interface or system that allows a user to book a session with a real-time educator using a score.

[1615] "Virtual character" refers to an animated character that is displayed for a user to interact with within a virtual reality space.

[1616] "News article" refers to information or news articles that a user can view within a virtual reality space.

[1617] "Summary" refers to text or speech that the user has written in a concise summary of the contents of a newspaper article.

[1618] "Means for evaluation" refers to the system or algorithm that analyzes the content of the summary entered by the user and determines its quality and accuracy.

[1619] MODE FOR CARRYING OUT THE INVENTION

[1620] The present invention relates to a VR-based language learning system that enables users to learn multiple foreign languages ​​efficiently and in a short period of time. Specific embodiments of the system will be described below.

[1621] Hardware and software used

[1622] This system consists of a user, a terminal (the user's VR device), and a server. The specific hardware and software used are as follows:

[1623] Hardware:

[1624] VR device (e.g. Oculus Quest 2)

[1625] software:

[1626] VR applications (e.g. apps developed with Unity)

[1627] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1628] Database system (e.g. MySQL)

[1629] User operations

[1630] The user puts on the VR device, launches a dedicated application, and logs into the system. After logging in, the user selects the language they want to learn and enters the virtual reality space. Based on the selected language, the server sends the corresponding language data to the device.

[1631] Virtual reality space generation

[1632] The device uses the Unity engine to generate a virtual reality space based on the language data received from the server. This space contains virtual characters and specific scenarios (e.g., virtual towns, cafes), and is constructed so that users can carry out activities.

[1633] Language Learning Activities

[1634] Users interact with characters in a virtual reality space. During the interaction, the user's voice is collected by the device's microphone and converted into text in real time using a speech recognition API. The converted text data is sent to a server, which evaluates the content and assigns a score. This score is used to visualize the user's evaluation and progress.

[1635] Newspaper article summaries

[1636] As another learning activity, users can read newspaper articles in a virtual reality space. After reading the article, they summarize the content and send the summary data to a server via their device. The server uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and assign an appropriate score.

[1637] Using points and sessions with real instructors

[1638] The points earned by users can be used to book a simulated session with a real-time educator. Users use their points to book a session with a teacher through the server, and once the reservation is confirmed, a connection is established at the specified time. Users can converse directly with the teacher and receive feedback on their pronunciation and grammar.

[1639] Examples of concrete examples and prompts

[1640] For example, if a user is learning Japanese and French, they log in to the system through a VR device, select Japanese or French, and then enter the virtual reality space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. They then switch to a scenario where they order at a French cafe and place their order in French. Through this series of activities, the user naturally improves their usage frequency and fluency in both languages.

[1641] (Example of a prompt)

[1642] "If a user wants to learn Japanese and French, they log in through a VR device and go through a shopping scenario in a virtual town in Japanese and a cafe ordering scenario in French. They can then use their points to book a session with a real-time instructor to further their learning."

[1643] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1644] Step 1:

[1645] Logging in to the system

[1646] Input: The user puts on the VR device and enters their ID and password to access the system.

[1647] Specific behavior:

[1648] The user puts on the VR device, launches the dedicated application, enters their ID and password on the login screen, and presses the "Login" button.

[1649] Data processing and calculation:

[1650] The terminal sends the entered ID and password to the server, which checks the database and, if authentication is successful, sends the user information back to the terminal.

[1651] Output: A successful authentication message that allows the user to access the main menu.

[1652] Step 2:

[1653] Selecting a language to learn

[1654] Input: The user selects the language they want to learn.

[1655] Specific behavior:

[1656] The user selects the language they want to learn (e.g., Japanese, French) from the main menu and performs the selection using the VR controller.

[1657] Data processing and calculation:

[1658] The terminal transmits the selected language data to the server, and the server prepares the corresponding language data and learning content and transmits them to the terminal.

[1659] Output: The learning content is displayed according to the user's language selection.

[1660] Step 3:

[1661] Virtual reality space generation

[1662] Input: Language data and scenario information sent by the server to the device.

[1663] Specific behavior:

[1664] The device uses the Unity engine to generate a virtual reality space based on the received language data and scenario information, placing textures and objects according to the scenario and generating a user avatar.

[1665] Data processing and calculation:

[1666] The device loads and renders 3D models and textures for the virtual space based on language data and scenario information.

[1667] Output: The virtual reality space is displayed on the user's VR device.

[1668] Step 4:

[1669] Conducting language learning activities

[1670] Input: User's voice input.

[1671] Specific behavior:

[1672] The user controls an avatar and interacts with virtual store clerks and characters. During the interaction, the user's voice is collected by the device's microphone.

[1673] Data processing and calculation:

[1674] The device converts the collected voice data into text in real time using a speech recognition API, and sends the converted text data to a server, which evaluates the content and calculates a score.

[1675] Output: The server evaluates the score and gives it back to the user.

[1676] Step 5:

[1677] Newspaper article summary activity

[1678] Input: A newspaper article that the user reads in a virtual reality space.

[1679] Specific behavior:

[1680] The user moves to a designated location in the virtual reality space and reads a newspaper article. After reading the article, the user summarizes the content and inputs it into the terminal.

[1681] Data processing and calculation:

[1682] The device sends the summary data to a server, which uses a generative AI model (e.g., GPT-4) to evaluate the quality of the summary and calculate a score.

[1683] Output: The server evaluates the summary and gives the score as feedback to the user.

[1684] Step 6:

[1685] Book a session with an educator in real time

[1686] Input: User-specified desired date and time and session reservation information with the educator.

[1687] Specific behavior:

[1688] The user selects the "Book a session with an instructor" option from the main menu, chooses the desired date and time and instructor, and makes the reservation using points.

[1689] Data processing and calculation:

[1690] The reservation information is sent from the terminal to the server, and the server registers the information in the reservation system. The reservation is confirmed in the instructor's schedule management system.

[1691] Output: The user is provided with a reservation confirmation message and information on how to establish a connection at the specified time.

[1692] In this way, users can efficiently learn a language through specific input, processing, and output at each step.

[1693] (Application example 1)

[1694] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1695] Conventional language learning systems have the drawback of providing users with few opportunities to practically learn the language actually used in the local area, making it difficult to acquire the high level of language proficiency required for real-life situations. There is also concern that the learning content may become monotonous, reducing user motivation. Furthermore, there are limited means for learning while receiving real-time feedback, which often results in insufficient learning efficiency.

[1696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1697] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the user's evaluation, means for reserving a real-time simulated session with a teacher using the points, and means for the user to learn a foreign language by selecting products and completing purchase and sale procedures in the virtual space. This allows the user to efficiently learn a language in an environment similar to real-life situations.

[1698] A "user" is someone who seeks to learn a language using the virtual reality-based language learning system of the present invention.

[1699] A "means for enabling selection" is a device or software that provides an interface and functionality for allowing a user to select the language they wish to learn from multiple foreign languages.

[1700] "Means for generating virtual reality spaces" refers to technologies and systems that create a virtual reality learning environment based on the language selected by the user.

[1701] "Means for analyzing and evaluating speech" refers to technology for analyzing a user's voice data and evaluating its content, specifically a system that uses voice recognition and natural language processing.

[1702] The "means for awarding points" is a mechanism for awarding reward points to a user based on an evaluation of the user's use of language.

[1703] The "means for reserving a simulated session with a real-time instructor" is a mechanism by which a user can use the points they have earned to make a reservation for a real-time language learning session with a real instructor.

[1704] The "means of selecting products in a virtual space and carrying out the buying and selling procedure" is a simulation function that allows users to select products in a virtual reality space and learn a language through that selection.

[1705] "Means for displaying a virtual character" refers to a technique or device for displaying a character for interaction with a user in a virtual reality space.

[1706] The "means for reading newspaper articles and inputting summaries of their contents" refers to an interface and technology that allows a user to view newspaper articles in a virtual reality space and input a summary of the main points.

[1707] The "means of evaluation" is a mechanism for evaluating a user's language proficiency based on the summary and dialogue content entered by the user.

[1708] The present invention relates to a system that allows users to efficiently learn multiple foreign languages ​​using VR (virtual reality). The overall configuration of the system consists of a server, a terminal (a user's VR device), and a user. Specific embodiments of the system are described below.

[1709] Overall structure

[1710] The system begins when the user accesses the server using a VR device, selects the foreign language they want to learn, and logs in. The server manages and processes all data.

[1711] User Interface

[1712] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1713] Virtual reality space generation

[1714] The device retrieves language data from the server based on the user's selection and generates a virtual reality space containing various scenarios in which the user can perform specific tasks (e.g., product selection and purchase procedures).

[1715] Language Learning Activities

[1716] Users interact with virtual characters in a virtual reality space. During the interaction, the device analyzes the user's speech and sends the data to the server. The server evaluates the speech and awards the user points. The user can use these points to challenge themselves to more difficult activities.

[1717] Product Selection Scenario

[1718] Users learn the language through a scenario in which they select and purchase products in a virtual store. They ask questions about the products in the local language and interact with a virtual character acting as a store clerk. Through this process, users can acquire vocabulary and phrases necessary for shopping and everyday conversation.

[1719] Newspaper article summaries

[1720] Another activity involves users reading local newspaper articles in the virtual reality space and summarizing their content, which is then sent to the server, which then evaluates and awards points.

[1721] Simulated sessions with real-time instructors

[1722] The points that users have earned can be used for simulated sessions with instructors in real time. Users can use their points to reserve sessions with instructors through the server. Once the reservation is confirmed, a connection is established according to the session time, and the user can talk directly with the instructor and further their learning.

[1723] Hardware and software used

[1724] Hardware

[1725] VR devices (e.g. Oculus Rift, HTC Vive)

[1726] A server (e.g., a Flask application server)

[1727] software

[1728] Speech recognition software (e.g., the SpeechRecognition Python library)

[1729] Translation API (e.g. Google Translate API)

[1730] Specific examples

[1731] For example, if a user wants to learn Japanese and French at the same time, they would follow the following steps: First, the user logs in to the system and selects Japanese and French as the languages ​​they want to learn. Next, the user starts a shopping scenario in a virtual store and selects products while interacting with the clerk in Japanese. Next, the user switches to a scenario where they are ordering at a French cafe and places their order in French.

[1732] Example prompts for generative AI models

[1733] "Design a scenario where a user can select products in a virtual store and have a conversation with a local sales associate."

[1734] "Create an algorithm that analyzes the user's voice and rates them based on a points system."

[1735] This allows users to efficiently improve their language skills in an environment that is close to the real world, and allows them to naturally acquire language skills in actual situations.

[1736] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1737] Step 1:

[1738] The user puts on the VR device and logs in to the system.

[1739] Input: The user enters login information into the VR device.

[1740] Data processing / calculation: The terminal sends the login information to the server for authentication. The server then verifies the user's authentication information by checking it against a database.

[1741] Output: If authentication is successful, the user's account information and learning history will be loaded.

[1742] Step 2:

[1743] The user selects the foreign language they want to learn.

[1744] Input: The user selects the language they wish to learn through the system's interface.

[1745] Data processing / calculation: The device receives the user's selection and sends the information to the server. The server retrieves the data in the selected language and prepares the virtual reality space.

[1746] Output: The virtual reality space is now set up and ready to be displayed to the user.

[1747] Step 3:

[1748] The device generates a virtual reality space.

[1749] Input: Language data obtained from the server and setting information for the virtual reality space.

[1750] Data processing / calculation: The terminal generates a virtual reality space based on the data received from the server.

[1751] Output: The generated virtual reality space is displayed on the user's VR device.

[1752] Step 4:

[1753] The user selects a product in the virtual reality space and completes the transaction process.

[1754] Input: Product information provided in the virtual reality space and user selection operations.

[1755] Data processing / calculation: When a user selects a product and starts a dialogue, the device collects the user's speech as voice data and sends it to the server, which analyzes the voice data and evaluates its content.

[1756] Output: The evaluation results are given to the user as points. The user receives feedback based on their learning status.

[1757] Step 5:

[1758] A user interacts with a virtual character in a virtual reality space.

[1759] Input: User utterances and virtual character configuration information.

[1760] Data processing / calculation: The device analyzes the user's speech in real time and generates appropriate responses from the virtual character. The server continuously records the dialogue and manages it as learning data.

[1761] Output: The interaction with the virtual character progresses and the user receives continuous feedback.

[1762] Step 6:

[1763] A user reads a newspaper article and enters a summary of it.

[1764] Input: Newspaper article provided in virtual reality space and user summary input.

[1765] Data processing / calculation: When a user inputs a summary, the terminal sends the content to the server, which evaluates the summary and assigns points based on the analysis results.

[1766] Output: Points are awarded and the user confirms the evaluation results.

[1767] Step 7:

[1768] Users use points to book simulated sessions with real-time instructors.

[1769] Input: Points acquired by the user and booking actions.

[1770] Data processing / calculation: The server checks the user's point balance and displays a list of available instructors. When a user reserves a session with an instructor, the server manages the connection time and instructor information and confirms the reservation.

[1771] Output: The reservation is confirmed and the user is notified of the reservation details. When the session time arrives, the device establishes a connection with the designated instructor and begins learning in real time.

[1772] In this way, through specific actions and data flows at each step, a system is realized that allows users to efficiently learn a foreign language in a virtual reality space.

[1773] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1774] The present invention combines a VR (virtual reality)-based language learning system that enables users to efficiently learn multiple foreign languages ​​with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[1775] Overall structure

[1776] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[1777] User Interface

[1778] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1779] Virtual reality space generation

[1780] The device retrieves language data from the server based on the user's selection and generates a virtual reality space, where various scenarios are set and the user can perform specific tasks (e.g., shopping at a store, ordering at a cafe).

[1781] Language Learning Activities

[1782] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1783] Emotion Engine Functions

[1784] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. This makes the user's learning experience more personalized and efficient.

[1785] Newspaper article summaries

[1786] Another activity is for users to read local newspaper articles in the virtual reality space. After reading the article, the user summarizes the content and sends the summary data to the server via their device. The server evaluates the content of the summary and awards points. During this process, the emotion engine also analyzes the user's emotions and provides appropriate feedback and adjusts the points.

[1787] Points and real instructors

[1788] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[1789] Specific examples

[1790] For example, if a user wishes to learn Japanese and French in parallel, they log in to the system, select these languages, and enter the VR space. As their first activity, they select a shopping scenario in a virtual Japanese town and converse with a virtual store clerk in Japanese. After completing the conversation, the user switches to a scenario in which they order at a French cafe and place their order in French. This series of activities naturally improves the frequency and spontaneity of their use of both languages. In addition, the emotion engine analyzes the user's emotional state in real time and provides appropriate feedback, further improving learning efficiency.

[1791] Furthermore, once users have earned a certain number of points, they can book a simulated session with a real-time instructor. During the session, users can receive feedback on pronunciation and grammar through conversations with the instructor, deepening their learning. In this way, this system allows users to efficiently learn multiple languages ​​and improve their practical language skills.

[1792] According to the embodiment of the present invention, users can learn multiple foreign languages ​​in a realistic environment, which allows for more efficient language acquisition than conventional learning methods. Furthermore, by effectively utilizing the emotion engine, users can enjoy an optimal learning experience that is tailored to their emotional state.

[1793] The processing flow will be explained below.

[1794] Step 1:

[1795] The user puts on the VR device, starts the system, and the login screen appears.

[1796] The terminal displays the user's login information in an input form and prompts them to enter it.

[1797] Step 2:

[1798] The user enters their username and password and clicks the login button.

[1799] The terminal receives the input and sends the login information to the server.

[1800] Step 3:

[1801] The server checks the received login information against its database and authenticates it.

[1802] If the authentication is successful, the user's profile information is obtained and returned to the terminal.

[1803] Step 4:

[1804] The terminal displays the authentication result to the user, and if successful, displays a language selection screen.

[1805] Users select 3-5 languages ​​they want to learn and press the next button.

[1806] Step 5:

[1807] The terminal transmits the user's selection to the server.

[1808] The server prepares the data for the selected language (vocabulary, grammar, audio data, etc.) and delivers it to the terminal.

[1809] Step 6:

[1810] After selecting a language, the user will be taken to the avatar customization screen.

[1811] The device will display the avatar settings options.

[1812] Users customize their avatar's appearance and complete the setup.

[1813] The device sends the set avatar to the server.

[1814] Step 7:

[1815] The server stores the received avatar information and generates the user's avatar in the VR space.

[1816] Step 8:

[1817] The user selects the first activity (e.g., shopping) in the VR space and starts.

[1818] The terminal requests scenario data for a shopping activity from the server.

[1819] The server transmits the corresponding scenario data to the terminal.

[1820] Step 9:

[1821] The terminal displays the scenario data, and the user begins to interact with the virtual character.

[1822] The user speaks through a microphone and answers questions posed by the virtual character.

[1823] Step 10:

[1824] The device analyzes the user's voice, converts it into text data, and sends it to the server.

[1825] The server compares the user's response text with pre-set correct answer data and performs an evaluation.

[1826] Based on the evaluation results, points are calculated and the results are sent back to the terminal.

[1827] Step 11:

[1828] The terminal displays the points and the evaluation results to the user.

[1829] Step 12:

[1830] The user selects the "read newspaper article" activity and presses the activity start button.

[1831] The terminal transmits activity selection information to the server.

[1832] The server selects the appropriate newspaper article and sends it to the terminal.

[1833] Step 13:

[1834] The terminal displays the newspaper article on the screen for the user to view.

[1835] The user reads the article and enters a summary in the input field.

[1836] Step 14:

[1837] After the user has finished entering the summary, he or she presses the send button.

[1838] The terminal transmits the summary data to the server.

[1839] Step 15:

[1840] The server reviews the summaries, rates them for accuracy, and calculates points.

[1841] The evaluation and point calculation results are sent back to the terminal.

[1842] Step 16:

[1843] The terminal displays the evaluation results and points to the user.

[1844] Step 17:

[1845] Users can use the points they earn to book sessions with instructors.

[1846] The terminal displays the reservation request and prompts the user for input.

[1847] Step 18:

[1848] The user enters the date, time and point information and presses the reservation button.

[1849] The terminal sends a reservation request to the server.

[1850] Step 19:

[1851] The server checks the instructor's schedule against the user's request and confirms the reservation.

[1852] The server returns reservation confirmation information to the terminal.

[1853] Step 20:

[1854] The server establishes a connection with the instructor at the start of a session and delivers real-time audio and video data to the terminal.

[1855] Users can directly talk with instructors to deepen their learning.

[1856] Step 21:

[1857] While the user is active in the VR space, the device uses an emotion engine to analyze the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine will detect this and send it to the server.

[1858] Step 22:

[1859] The server adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. For example, if the emotion engine determines that the user is tired, it will switch to a lighter task or display an encouraging message.

[1860] Step 23:

[1861] When a user reads a newspaper article and summarizes it, the device uses an emotion engine to analyze the user's emotional state. For example, if the user is feeling stressed, the device may take an approach such as adjusting the evaluation criteria for summarizing.

[1862] Step 24:

[1863] During a simulated session with a real-time instructor, the device uses an emotion engine to monitor the user's emotional state. For example, if the user is nervous, the emotion engine provides this information to the instructor in real time, allowing the instructor to provide appropriate feedback and support.

[1864] This provides a learning experience that takes the user's emotions into consideration, maximizing learning efficiency and effectiveness.

[1865] Example 2

[1866] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1867] Conventional language learning systems have the problem of being unable to respond to individual learning needs due to insufficient feedback on users' learning progress and emotional state. In particular, they provide uniform learning content without considering users' emotions, making it difficult to achieve efficient learning. Furthermore, they are unable to effectively evaluate users' learning outcomes and reflect the results in the learning process, making it difficult to maintain motivation to learn.

[1868] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1869] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to study, means for generating a virtual reality space based on the user's selection, means for displaying a virtual character with which the user can interact in the virtual reality space, means for analyzing and evaluating the user's speech, an emotion engine that includes an analysis of the user's emotional state and means for adjusting the difficulty level of the learning content and the feedback content based on the emotional state, means for awarding points based on the evaluation obtained by the user, and means for booking a simulated session with a real-time instructor using the points. This makes it possible to provide a personalized learning experience based on the user's learning progress and emotional state.

[1870] "User" refers to an individual or organization that uses the system to learn a foreign language.

[1871] "Virtual reality space" refers to a digital environment different from the real world that users can enter through VR devices.

[1872] "Virtual character" refers to a digital character that is displayed to interact with a user within a virtual reality space.

[1873] "Utterance" refers to the verbal expressions used by a user when interacting with virtual characters and other users in a virtual reality space.

[1874] "Emotion engine" refers to a software or hardware system that monitors a user's facial expressions and vocal tone to analyze the user's emotional state.

[1875] "Emotional state" refers to the psychological and emotional state, such as stress, fatigue, or satisfaction, that a user experiences while using a system.

[1876] "Evaluation" refers to the analysis and scoring of a user's speech and activity by the server.

[1877] "Points" refer to rewards or scores awarded based on a user's learning outcomes.

[1878] A "simulated session" refers to a virtual online learning session between a real-time instructor and a user.

[1879] The present invention combines a VR-based language learning system that enables users to efficiently learn a foreign language with an emotion engine that recognizes the user's emotions. This system consists of a user, a terminal (the user's VR device), a server, and an emotion engine. Specific embodiments of this system are described below.

[1880] User Login and Language Selection

[1881] The user puts on a VR device (e.g., Oculus Rift or HTC Vive) and connects to the provided login screen. By entering their user ID and password, they are granted access to the system. Then, the user selects the foreign language they wish to learn on the dashboard. The options include multiple foreign languages ​​(e.g., Japanese, French).

[1882] Virtual reality space generation

[1883] The device retrieves language data from the server based on the user's selection. During this process, the device sends a request to the server (e.g., a cloud service), which returns information about the user's selected language from a database (e.g., MySQL). Using the retrieved data, the device generates a virtual reality space using the Unity engine.

[1884] Interaction in virtual reality space

[1885] The user enters the generated virtual reality space and selects various scenarios (e.g., shopping at a store, ordering at a cafe). The user's avatar is displayed and the user interacts with the virtual character. The user's voice is collected by the VR device's microphone and analyzed in real time.

[1886] Speech and emotion data analysis

[1887] The device sends the user's voice data to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converts it into text. The converted text data is sent to a server, which runs a script to evaluate the content. The server then awards points based on the evaluation results and returns the information to the device.

[1888] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. For example, it uses emotion recognition technology (e.g., Affectiva's SDK) to recognize emotions from facial expressions and transmits the emotion data to the server. The server then dynamically adjusts the difficulty of the learning content and the content of the feedback based on this data.

[1889] Summarizing and evaluating newspaper articles

[1890] The user reads a newspaper article displayed in the virtual reality space and summarizes its contents. The created summary is sent to the server via the device. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summary and award points. During this process, the emotion engine monitors the user's emotions and provides appropriate feedback and adjusts the points.

[1891] Points Redemption and Instructor Sessions

[1892] Users can use their earned points to book a simulated session with a real-time instructor. Once the reservation is confirmed, the server connects the instructor and the user via video chat at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[1893] Examples of concrete examples and prompts

[1894] For example, if a user wants to learn Japanese and French in parallel, they can follow this process:

[1895] 1. The user logs into the system, selects Japanese and French, and begins a conversation in Japanese in the virtual town.

[1896] 2. Once the dialogue is complete, place your order in French at a French cafe.

[1897] 3. During this process, the emotion engine detects the user's stress, and the server automatically adjusts the learning content.

[1898] 4. The user reads the newspaper article in the virtual reality space and sends the summary to the server.

[1899] 5. The server evaluates the summary and awards points.

[1900] 6. Finally, users can use their points to book a simulated session with a real-time instructor, during which the emotion engine monitors the user's emotions and provides real-time feedback to the instructor.

[1901] Prompt Sentence Examples

[1902] "I want to learn Japanese and French at the same time in a VR space. Please simulate shopping in a store in Japanese and ordering at a cafe in French. Also, please use an emotion engine to adjust the learning feedback."

[1903] This system allows users to efficiently learn multiple languages ​​in a realistic virtual environment. It also utilizes an emotion engine to provide the optimal learning environment for each user.

[1904] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1905] Step 1:

[1906] The user puts on the VR device and enters their user ID and password on the login screen. Once the input is complete and the authentication server verifies the authentication information and permits login, the user can access the dashboard. This allows the user to proceed to the next step.

[1907] Input: User ID, Password

[1908] Output: User authentication status (login status)

[1909] Step 2:

[1910] Users select the foreign language they want to learn on the dashboard. Once the system receives the user's selection, it sends that information to the server, which retrieves data about the selected language from a database and sends it back to the device, which then loads content related to the language the user wants to learn.

[1911] Input: Selected foreign language

[1912] Output: Data related to the selected foreign language

[1913] Step 3:

[1914] The device receives the language data sent from the server and generates a virtual reality space. At this time, the device uses a game engine such as the Unity engine to create the graphics and scenario for the virtual space. This allows the user to access a virtual reality space that corresponds to the selected language.

[1915] Input: Language data sent from the server

[1916] Output: Generated virtual reality space

[1917] Step 4:

[1918] The user enters the generated virtual reality space and selects a specific scenario (e.g., shopping at a store, ordering at a cafe). Once the selection is complete, a conversation with a virtual character based on the scenario begins. The user's speech is collected by the microphone of the VR device.

[1919] Input: Select a specific scenario

[1920] Output: Start of dialogue with virtual character

[1921] Step 5:

[1922] The device analyzes the user's voice data in real time. The voice data is sent to a speech recognition service (e.g., Google Cloud Speech-to-Text) and converted into text data. The converted text data is sent to a server, which then runs a script to evaluate the content. Points are awarded based on the evaluation results.

[1923] Input: User's voice data

[1924] Output: Evaluation results and points awarded

[1925] Step 6:

[1926] The emotion engine collects and analyzes the user's facial expressions and vocal tone via a camera and microphone. The emotion data is sent to a server, which then uses this data to perform calculations to adjust the difficulty of the learning content and the content of the feedback. This provides an optimal learning environment according to the user's emotional state.

[1927] Input: User's facial expressions and voice tone

[1928] Output: Adjustment of learning content and feedback

[1929] Step 7:

[1930] Users read newspaper articles displayed in a virtual reality space and create summaries. The summaries are then sent to a server via their devices. The server uses a natural language processing engine (e.g., a generative AI model) to evaluate the summaries, and points are awarded based on the evaluation results.

[1931] Input: User-generated abstract

[1932] Output: Evaluation results and points awarded

[1933] Step 8:

[1934] Users can use their earned points to book a simulated session with a real-time instructor. The reservation information is sent to the server, which then connects the instructor and user via a video chat platform at the start of the session. During this time, the emotion engine monitors the user's facial expressions and voice and provides real-time feedback to the instructor.

[1935] Input: Reservation information and points

[1936] Output: Connection with instructor and real-time feedback

[1937] (Application example 2)

[1938] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1939] Conventional language learning systems do not provide sufficient support for users to efficiently learn multiple foreign languages, and it is difficult to provide an optimal learning experience that reflects the user's emotional state in real time while learning.

[1940] Furthermore, the lack of a virtual store customer service simulation as a practical application environment for language learning makes it difficult to implement practical learning that takes into account real-world applications. To solve these problems, a system is needed that can grasp the user's emotional state in real time and adjust the learning content based on that, while providing a virtual experience that is close to real-life situations.

[1941] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1942] In this invention, the server includes means for allowing a user to select multiple foreign languages ​​to learn, means for generating a virtual reality space based on the user's selection, means for analyzing and evaluating the user's speech, means for awarding points based on the evaluation obtained by the user, means for reserving a simulated session with a real-time instructor using the points, means for recognizing the user's emotional state and analyzing the data to optimize the learning experience, and means for the user to simulate customer service in a foreign language in a virtual store. This enables a user to efficiently acquire multiple foreign languages ​​through practical language use scenarios while receiving learning content personalized in real time according to their emotional state.

[1943] "User" refers to an individual who uses the system to learn a foreign language.

[1944] "Multiple foreign languages" refers to multiple different languages ​​that a user can select at will to learn.

[1945] "Virtual reality space" refers to a computer-generated simulated environment that a user experiences through a VR device.

[1946] "Utterance" refers to verbal communication that the user makes while studying.

[1947] "Evaluation" refers to analyzing the content of a user's utterance and converting it into a score based on its accuracy and appropriateness.

[1948] "Points" refer to scores or rewards based on evaluations that users obtain through learning activities.

[1949] "Real-time instructor" refers to a human instructor who provides direct instruction to a user through the system.

[1950] "Simulated session" refers to a simulated learning or interaction session within a virtual environment.

[1951] "Emotional state" refers to the psychological state of the user that is analyzed from facial expressions, tone of voice, and the like.

[1952] "Virtual store" refers to an environment set in virtual reality that resembles a physical store.

[1953] "Customer service simulation" refers to a learning activity in which a user interacts as a store clerk or a customer in a virtual store, simulating a scenario in which the user provides or receives service.

[1954] The present invention is a system that allows users to efficiently learn multiple foreign languages, and provides a more personalized learning experience by combining a VR-based virtual reality space with an emotion engine that analyzes the user's emotional state. A specific embodiment of this system will be described below.

[1955] Overall structure

[1956] This system consists of a user, a terminal (user's VR device), a server, and an emotion engine. The user accesses the system using a VR device, and the server manages and processes the entire data. The emotion engine is responsible for recognizing the user's emotions and sending that data to the server.

[1957] User Interface

[1958] Users put on a VR device and log in to the system. After logging in, they can select the language they want to learn from multiple foreign languages. Once the selection is complete, they are transported to a virtual reality space. The user's avatar is displayed in this space, and they can participate in various activities through their avatar.

[1959] Virtual reality space generation

[1960] The terminal acquires language data from the server based on the user's selection and generates a virtual reality space. Various scenarios are set in this space, and the user can perform specific tasks (e.g., customer service simulations in a virtual store) within the space. Specifically, the system provides scenarios for the user to ask for product explanations or place orders as a customer.

[1961] Language Learning Activities

[1962] Users interact with virtual characters in a virtual reality space. During the interaction, the user's voice is analyzed by the device. The analyzed data is sent to a server, which evaluates it and awards points. Users can earn these points and challenge themselves to more difficult activities.

[1963] Emotion Engine Functions

[1964] While the user is active in the VR space, the emotion engine analyzes the user's facial expressions and vocal tone to recognize their emotional state. For example, if the user is feeling tired or stressed, the emotion engine detects this and sends it to the server. The server then adjusts the difficulty of the learning content and the content of the feedback based on the emotional data. The emotion engine used can be, for example, the Affectiva SDK.

[1965] Points and real instructors

[1966] The points earned by users can be used for simulated sessions with instructors in real time. Users use their points to reserve a session with an instructor through the server. Once the reservation is confirmed, a connection is established at the scheduled session time, allowing the user to converse directly with the instructor and further their learning. In this case, the emotion engine also monitors the user's emotional state and provides real-time feedback to the instructor.

[1967] Hardware and software used

[1968] Hardware:

[1969] VR device (e.g., Oculus Rift): Used to display the virtual space and capture user input.

[1970] Emotion engine (e.g., Affectiva SDK): Recognizes emotions by analyzing the user's facial expression data.

[1971] software:

[1972] Language processing engine (e.g., Google Cloud Translation API): Analyzes the user's speech and translates it into the foreign language of their choice.

[1973] Database: Stores and manages users' learning progress and points.

[1974] Specific examples

[1975] For example, in a scenario where a user is a salesperson in a virtual store explaining a product in French, the system would prompt the user with questions such as, "What are the features of this product?" The user would respond to the question in French, and their responses would be analyzed and evaluated. If the emotion engine detects the user becoming confused or stressed, the system would adjust its learning and provide clearer explanations or additional support.

[1976] Prompt Sentence Examples

[1977] "As a customer in a virtual store in a VR space, ask for product details in French."

[1978] "Use an emotion engine to analyze user emotions and optimize the learning experience."

[1979] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1980] Step 1: User Login and Language Selection

[1981] The user puts on the VR device and logs in to the system. The system receives the user's ID and password as input and authenticates them. If authentication is successful, the user selects one or more foreign languages ​​to learn. The selected language data is sent to the server.

[1982] Step 2: Creating a virtual reality space

[1983] The server acquires language data based on the user's selection and sends information to the terminal (VR device) to generate a virtual reality space. The terminal generates the virtual reality space and displays it to the user. At this time, scenarios and tasks within the virtual space (for example, a customer service simulation in a virtual store) are also set.

[1984] Step 3: User begins interacting with virtual character

[1985] The user begins to interact with a virtual character in a virtual reality space. The virtual character presents the user with a predefined prompt, such as "Please tell me about the features of this product." The VR device captures the user's voice and sends it to the server as audio data.

[1986] Step 4: Analyzing and evaluating user voice

[1987] The server analyzes the received voice data using a language processing engine (for example, Google Cloud Translation API). As a result of the analysis, the voice input content is converted into text data and evaluated. This evaluation is based on criteria such as accuracy, appropriateness, and pronunciation. The evaluation results are awarded to the user as points.

[1988] Step 5: Emotion analysis using the emotion engine

[1989] Facial expression data and voice tone data captured during user interaction are analyzed using an emotion engine (e.g., Affectiva SDK). The emotion engine recognizes the user's emotional state and sends the results to the server. The server uses this data to determine the user's level of fatigue or stress.

[1990] Step 6: Adjust your learning

[1991] The server then adjusts the user's learning based on the analyzed emotional data, adjusting the level of feedback and explanation provided to the user. For example, if the user is confused, the server will provide clearer explanations and additional support.

[1992] Step 7: Use points and simulate a real-time instructor session

[1993] Users use points earned in the virtual reality space to reserve a simulated session with a real-time instructor. The server checks point balances and manages reservations. Once a reservation is confirmed, a connection is established between the instructor and the user at the specified time. During the session, the emotion engine monitors the user's state and provides real-time feedback to the instructor.

[1994] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1995] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1996] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1997] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1998] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1999] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2000] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2001] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2002] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2003] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2004] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2005] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2006] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes...

Claims

1. means for allowing a user to select a plurality of foreign languages ​​to learn; means for generating a virtual reality space based on a user's selection; means for analyzing and evaluating user utterances; A means for awarding points based on the evaluations obtained by the user; a means for using said points to reserve a simulated session with a real-time instructor; A system including:

2. The system of claim 1 further comprising means for displaying a virtual character with which a user interacts within the virtual reality space.

3. 2. The system according to claim 1, further comprising means for allowing a user to read a newspaper article in said virtual reality space and input a summary of the article's contents, and means for evaluating said input summary.

4. 10. The system of claim 1, further comprising means for establishing a connection with an instructor to conduct a simulated session with the real-time instructor.

5. 2. The system according to claim 1, further comprising means for maintaining a learning database of said plurality of foreign languages ​​and for recording a learning history.

Images