System

Abstract

To provide a comprehensive learning system for providing a user with customization of a learning curriculum, implementation of a virtual project, and real-time feedback from login.SOLUTION: A means for inputting login information, a means for allowing a server to authenticate the login information, acquire profile data and transmit it to a terminal, a means for displaying profile information and selecting a field or a topic to be learned, and a means for allowing the server to generate a learning curriculum based on the selection information and transmit it to the terminal; A system comprising: means for displaying; means for a user selecting a virtual project and transmitting to a server; means for the server setting up a virtual environment and loading the selected project; means for transmitting environment data to a terminal; means for connecting the user to the virtual environment; means for the server monitoring the user's progress and providing real-time feedback using generative AI; and means for displaying the feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern engineering education, providing custom learning tailored to the needs of individual learners and realistic project experiences is a difficult challenge. Traditional learning systems and educational platforms often have standardized curricula, making it difficult to provide instruction and real-time feedback tailored to individual learners. Furthermore, project experiences to cultivate practical skills are limited due to real-world constraints and lack of resources. Given these circumstances, there is a need to develop systems that provide more personalized, practical learning experiences. [Means for solving the problem]

[0005] The present invention provides a comprehensive learning system that allows users to log in, customize their learning curriculum, complete virtual projects, and receive real-time feedback. The system of the present invention includes the following means.

[0006] 1. A means for the user to enter login information and a means for the device to send the entered login information to the server.

[0007] 2. A means for the server to authenticate the login information, obtain the user's profile data, and send the profile data to the device.

[0008] 3. A means for the user to select the field or topic they wish to study, and a means for the terminal to send the selection information to the server, which then generates a learning curriculum based on the selection information.

[0009] 4. A means for the server to send the generated learning curriculum to the terminal and for the terminal to display the curriculum.

[0010] 5. A means for the user to select a virtual project, the terminal to send the project selection information to the server, and the server to configure the virtual environment and load the project.

[0011] 6. A means by which the server sends virtual environment configuration information to the terminal, and the terminal connects the user to the virtual environment.

[0012] 7. A means for the server to monitor the user's progress and provide real-time feedback using generative AI, and a means for the device to display the feedback.

[0013] This allows users to receive a learning experience customized to their individual needs, and allows them to improve their practical skills through hands-on project experience in a virtual reality environment powered by the Metaverse.

[0014] "Login information" means the identification information used by a user to access a system, and typically consists of a username and password.

[0015] "Terminal" means the hardware device used by a User to access the System, including PCs, tablets, VR headsets, etc.

[0016] A "server" is a computer system that performs the main processing of a system and stores and manages data.

[0017] "Profile data" refers to data such as a user's personal information and learning history, and serves as the basis for individual customization in the system.

[0018] A "learning curriculum" is a plan that defines the content that users should learn and the structure of the course, and serves as a guideline for providing custom learning.

[0019] "Virtual projects" refer to simulations and practical assignments provided on the system that users experience in a virtual environment.

[0020] A "virtual environment" is a virtual reality space recreated using metaverse technology, providing a project environment that users can actually experience.

[0021] "Progress information" is data that indicates the user's current progress in a virtual project or learning curriculum.

[0022] "Generative AI" is an artificial intelligence technology that generates customized feedback and advice based on a user's data and progress.

[0023] "Real-time feedback" refers to advice and guidance provided instantly based on a user's actions and progress. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] This invention is a new learning system that combines generative AI and the Metaverse, aiming to provide users with customized learning experiences and practical project experiences.

[0046] System configuration and program processing

[0047] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[0048] 1. User login and profile settings

[0049] User: Uses a terminal to access the system login screen and enters a username and password.

[0050] On the device: The login information entered by the user is encrypted and sent to the server.

[0051] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[0052] Server: Encrypts the profile data and sends it back to the device.

[0053] Terminal: Decodes the received profile data and displays it on the screen.

[0054] Examples:

[0055] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[0056] 2. Customize your learning curriculum

[0057] Users: Choose the subject or topic they want to learn about.

[0058] Terminal: Encrypts the selected information and sends it to the server.

[0059] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[0060] Server: Encrypts the generated learning curriculum and sends it to the device.

[0061] Terminal: Decodes the received curriculum and displays it to the user.

[0062] Examples:

[0063] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0064] 3. Setting up and participating in virtual projects

[0065] Users: Select a virtual project within the curriculum.

[0066] On your device: Encrypt the project selection information and send it to the server.

[0067] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[0068] Server: Encrypts the virtual environment configuration information and sends it to the device.

[0069] Terminal: Connects the user to the virtual environment based on the received configuration information.

[0070] Examples:

[0071] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[0072] 4. Personalized instruction and real-time feedback

[0073] Users: Work on projects in a virtual environment.

[0074] Device: Sends completed tasks and progress information to the server in real time.

[0075] Server: Analyzes progress information and uses generative AI to generate feedback appropriate for the user.

[0076] Server: Encrypts the generated feedback and sends it to the device.

[0077] Terminal: Display feedback on the user's screen.

[0078] Examples:

[0079] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0080] As described above, this system leverages generative AI and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to grow.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The user uses a terminal to access the system's login screen and enters their username and password.

[0084] Step 2:

[0085] The terminal encrypts the entered login information and sends it to the server.

[0086] Step 3:

[0087] The server compares the received login information with the user data in the database and performs authentication.

[0088] Step 4:

[0089] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[0090] Step 5:

[0091] The device decodes the received profile data and displays it on the screen.

[0092] Step 6:

[0093] Users select the field or topic they want to learn on their device.

[0094] Step 7:

[0095] The terminal encrypts the selected information and sends it to the server.

[0096] Step 8:

[0097] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[0098] Step 9:

[0099] The learning curriculum generated by the server is encrypted and sent to the terminal.

[0100] Step 10:

[0101] The terminal decodes the received curriculum and displays it to the user.

[0102] Step 11:

[0103] A user selects a virtual project within a curriculum.

[0104] Step 12:

[0105] The terminal encrypts the project selection information and transmits it to the server.

[0106] Step 13:

[0107] The server sets up the metaverse virtual environment and loads the selected project contents.

[0108] Step 14:

[0109] The server encrypts the virtual environment configuration information and sends it to the terminal.

[0110] Step 15:

[0111] The terminal connects the user to the virtual environment based on the configuration information received.

[0112] Step 16:

[0113] Users work on projects in a virtual environment.

[0114] Step 17:

[0115] The device sends progress information to the server in real time.

[0116] Step 18:

[0117] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[0118] Step 19:

[0119] The server generates feedback that is encrypted and sent to the terminal.

[0120] Step 20:

[0121] The terminal displays the received feedback to the user and provides specific advice.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] Conventional learning systems often offer only a standard curriculum, making it difficult to customize to meet individual user needs. They also struggle to provide real-time feedback and practical project experiences. Providing a learning environment utilizing a virtual reality environment, in particular, poses numerous technical challenges and is difficult to implement.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes means for authenticating login information and obtaining user profile data, means for generating a learning curriculum using a generative AI model based on the selection information, means for configuring a metaverse environment and loading the selected projects, and means for monitoring completed tasks and progress information in real time and providing real-time feedback using the generative AI model, thereby providing a personalized learning experience and hands-on project experience with real-time feedback.

[0127] "User" refers to the person who operates the system and experiences the learning curriculum and virtual projects.

[0128] "Device" refers to a computer or mobile device on which a user performs operations such as entering login information, selecting a course of study, or participating in a virtual project.

[0129] "Server" refers to a central processing unit that authenticates user login information, obtains profile data, generates learning curricula using generative AI models, sets up the metaverse environment, and generates real-time feedback.

[0130] "Generative AI model" refers to an artificial intelligence algorithm used to generate personalized curriculum based on user selections and provide real-time feedback.

[0131] "Login Information" refers to authentication information, such as a username and password, entered by a User to access a System.

[0132] "Profile Data" refers to data that includes personal information such as a user's name, learning history, and settings.

[0133] A "learning curriculum" refers to a series of learning content and assignments generated based on the field or topic a user wants to learn.

[0134] A "virtual project" refers to a specific challenge or task that a user participates in and completes within a virtual reality environment.

[0135] A "metaverse environment" refers to a virtual work or learning environment built using virtual reality technology.

[0136] "Real-time feedback" refers to instantaneous evaluation and advice provided to users based on their progress.

[0137] This invention is a new learning system that combines generative AI models and the Metaverse. The purpose of this system is to provide users with customized learning experiences and practical project experiences. Below, we will explain how to implement this system in detail.

[0138] This system consists of three main components: a server, a terminal, and a user.

[0139] Hardware and software used

[0140] Server: Serves as a central processing unit, authenticating users, managing profile data, generating learning curricula using generative AI models (e.g., GPT-4 (registered trademark)), setting up the metaverse environment, and generating real-time feedback. Implemented in a server system that also includes databases and AI algorithms.

[0141] Terminal: A device operated by a user, such as a computer, smartphone, or tablet. The software on the terminal provides the user interface, performs encryption / decryption processing, communicates with the server, and displays information to the user.

[0142] User: An individual who uses the learning system. The user operates the device and progresses with the learning.

[0143] Program processing

[0144] User login and profile settings

[0145] A user uses a terminal to access the system's login screen and enters their username and password. The terminal encrypts this login information (e.g., AES encryption) and sends it to the server. The server decrypts the received login information and authenticates it by comparing it with information in a database. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it to the terminal. The terminal decrypts the data and displays it to the user.

[0146] Customized learning curriculum

[0147] The user selects the field or topic they wish to study. The device encrypts the selection information (e.g., RSA encryption) and sends it to the server. The server analyzes the received selection information using a generative AI model (e.g., GPT-4) and generates a learning curriculum optimal for the user. The generated curriculum is encrypted and sent to the device. The device decrypts the curriculum and displays it to the user.

[0148] Examples:

[0149] The user selects that they want to learn machine learning and data science. The device RSA-encrypts this information and sends it to the server. The server uses GPT-4 to generate a customized curriculum and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0150] Setting up and participating in virtual projects

[0151] The user selects a virtual project within the curriculum. The device encrypts the selection information and sends it to the server. The server analyzes the received information and sets up the virtual project within the metaverse environment. The set virtual environment information is then encrypted and sent to the device. The device decrypts it and connects the user to the virtual environment.

[0152] Examples:

[0153] The user selects a virtual data science project. The device encrypts this selection information and sends it to the server. The server configures the metaverse environment and loads the project content. The server encrypts the environment configuration information and sends it to the device, which then connects the user to the virtual environment.

[0154] Personalized instruction and real-time feedback

[0155] Users work on projects in a virtual environment. The device sends completed tasks and progress information to the server in real time. The server analyzes the progress information and generates appropriate feedback for the user using a generative AI model (e.g., Chat GPT). The generated feedback is encrypted and sent to the device, which then displays the feedback on the user's screen.

[0156] Examples:

[0157] The user performs data analysis in a virtual environment. The device sends progress information to the server in real time. The server uses this information to generate feedback using a generative AI model and sends it to the device. The device then displays the feedback on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0158] Prompt Sentence Examples

[0159] "Users want to learn machine learning from the basics to advanced applications. Please generate a customized learning curriculum."

[0160] As described above, the system leverages generative AI models and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to help them grow.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] A user accesses the login screen of the system using a terminal and enters a user name and password, which become the input information.

[0164] The terminal encrypts the entered login information using the AES encryption algorithm and sends it to the server, where it outputs the encrypted login information.

[0165] Step 2:

[0166] The server decrypts the received encrypted login information using the AES decryption algorithm.

[0167] The server compares the decrypted login information with the user data in the database and performs authentication. The user data from the database is used as input information. If authentication is successful, the user's profile data is retrieved.

[0168] The profile data is encrypted using the AES encryption algorithm and sent to the terminal, and this encrypted profile data is the output.

[0169] Step 3:

[0170] The terminal decrypts the received encrypted profile data using the AES decryption algorithm.

[0171] The decrypted profile data is displayed on the user's screen, which is the output.

[0172] Step 4:

[0173] The user selects the field or topic they want to learn about, and this selection information becomes the input information.

[0174] The terminal encrypts the user's selection information using the RSA encryption algorithm and sends it to the server, where the encrypted selection information is output.

[0175] Step 5:

[0176] The server decrypts the received encrypted selection information using the RSA decryption algorithm.

[0177] The server generates a custom learning curriculum using a generative AI model (e.g., GPT-4) based on the decoded selection information. This selection information serves as input information.

[0178] The generated learning curriculum is encrypted using the AES encryption algorithm and sent to the terminal. This encrypted learning curriculum is the output.

[0179] Step 6:

[0180] The terminal decrypts the received encrypted learning curriculum using the AES decryption algorithm.

[0181] The decoded learning curriculum is displayed on the user's screen, which is the output.

[0182] Step 7:

[0183] The user selects a virtual project within the curriculum, and this selection information becomes the input information.

[0184] The device hashes the project selection information using the SHA-256 hashing algorithm and sends it to the server, where the hashed selection information is the output.

[0185] Step 8:

[0186] The server parses the received hashed project selection information.

[0187] The server configures the metaverse environment and loads the selected project contents. This configuration information becomes the input information.

[0188] The configured virtual environment information is hashed using SHA-256 and sent to the terminal. This hashed configuration information is output.

[0189] Step 9:

[0190] The terminal connects the user to the metaverse virtual project based on the received hashed virtual environment information.

[0191] The output is that the user is connected to the virtual environment.

[0192] Step 10:

[0193] Users work on projects within a virtual environment, and their actions within this environment provide input information.

[0194] The terminal transmits completed tasks and progress information to the server in real time using the WebSocket protocol, and this progress information is the output.

[0195] Step 11:

[0196] The server analyzes the received progress information and generates optimal feedback for the user using a generative AI model (e.g., Chat GPT). The progress information is used as input.

[0197] The generated feedback is encrypted via WebSocket and sent to the terminal. This encrypted feedback becomes the output.

[0198] Step 12:

[0199] The terminal decrypts the received encrypted feedback.

[0200] The decoded feedback is displayed on the user's screen, which is the output.

[0201] In this way, by performing specific operations and processing and calculating data at each step, it is possible to provide the user with a customized learning experience.

[0202] (Application example 1)

[0203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0204] Conventional learning systems have struggled to provide customized learning tailored to individual users' needs or practical project experiences in virtual environments. They also struggled to provide optimal feedback on users' learning progress in real time. Especially in distance learning and online education, there is a lack of individualized instruction, which can lead to insufficient learning outcomes.

[0205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0206] In this invention, the server includes a means for generating customized advice based on the user's learning progress and displaying it to the user on the terminal, a means for providing real-time feedback using generative AI, and a means for providing a project experience in a virtual reality environment using the metaverse, allowing users to enjoy customized learning and practical project experience tailored to their individual needs and receive optimal feedback in real time.

[0207] "User" refers to someone who uses the system to study or work on a project.

[0208] "Terminal" refers to a device operated by a user, used to input login information and selection information, display profile data and learning curriculum, connect to a virtual environment, and so on.

[0209] "Server" refers to a device that receives and processes information sent from a terminal and has functions such as user authentication, acquisition of profile data, generation of learning curriculum, setting up a virtual environment, and generation of real-time feedback.

[0210] "Login Information" refers to the authentication information required for a user to access a system, and typically consists of a username and password.

[0211] "Profile data" refers to data that includes information such as the user's personal information, learning history, and skill level, and is acquired by the server and returned to the terminal when authentication is successful.

[0212] "Learning curriculum" refers to the learning content or study plan for a specific field or topic that is generated by the server based on user selection information.

[0213] A "virtual project" refers to a specific assignment or task selected by a user and executed in a virtual environment within the metaverse.

[0214] The "metaverse" is a virtual environment constructed using virtual reality technology, and refers to a virtual space in which users can interact with each other.

[0215] "Generative AI" refers to artificial intelligence technology that generates appropriate advice and feedback based on a user's learning progress and project content.

[0216] "Feedback" refers to the evaluation and advice provided in real time by generative AI for tasks or projects that users are currently working on.

[0217] "Customized advice" refers to personalized advice that suggests specific content and methods based on the user's learning progress and skill level.

[0218] Three main components are required to implement this invention: a server, a terminal, and a user. The system provides users with a customized learning experience and project experience in a virtual environment.

[0219] First, a user uses a terminal to access the system's login screen. Here, they enter their username and password. The terminal encrypts the entered login information and sends it to the server. The server compares the received login information with the user data in its database and performs authentication. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it back to the terminal. The terminal decrypts the profile data and displays it to the user.

[0220] Next, the learning curriculum is customized. The user selects the fields and topics they want to study. The device encrypts these selections and sends them to the server. The server uses a generative AI model to generate a custom learning curriculum based on the user's selections. The generated curriculum is then encrypted again and sent to the device. The device decrypts the curriculum data and displays it to the user.

[0221] The user then selects a virtual project within the curriculum. The device encrypts the project selection information and sends it to the server. The server then configures the metaverse virtual environment and loads the selected project content. The virtual environment configuration information is then encrypted and sent to the device. The device then connects the user to the virtual environment based on the configuration information.

[0222] As a user works on a project in a virtual environment, the device sends completed tasks and progress information to the server in real time. The server analyzes this information and uses a generative AI model to generate appropriate feedback for the user. The generated feedback is encrypted and sent to the device, which then decrypts it and displays it to the user. This feedback can include specific advice, such as "Try a different regression algorithm for this step."

[0223] The hardware used includes smartphones and head-mounted displays (HMDs), and the software includes Unity (for building the virtual environment), AWS (for servers and databases), and OpenAI's GPT-4, a generative AI.

[0224] Below are some examples of prompt sentences.

[0225] "User wants feedback on data analysis. Please provide specific advice."

[0226] The system allows users to receive customized learning tailored to their individual needs and improve their skills through hands-on project experience in a virtual environment, with real-time feedback providing a significant learning experience.

[0227] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0228] Step 1:

[0229] The user accesses the login screen of the learning system using a smartphone or head-mounted display (HMD). The user enters their username and password. This is used as input data to proceed to the next step.

[0230] Step 2:

[0231] The terminal encrypts the login information entered by the user. This encrypted data is sent to the server. The input is the login information, and the output is the encrypted login data.

[0232] Step 3:

[0233] The server decrypts the encrypted login information it receives and authenticates it by comparing it with the user data in the database. If authentication is successful, it obtains the user's profile data. The input is the encrypted login data, and the output is the user's profile data.

[0234] Step 4:

[0235] The server re-encrypts the acquired profile data and sends it to the terminal. The input is the profile data, and the output is the encrypted profile data.

[0236] Step 5:

[0237] The device decrypts the received profile data and displays it to the user, allowing the user to check their own profile information. The input is encrypted profile data, and the output is decrypted profile data.

[0238] Step 6:

[0239] The user selects the field or topic they want to study, and with this as input, they move on to the next step.

[0240] Step 7:

[0241] The terminal encrypts information about the learning field or topic selected by the user and sends it to the server. The input is the selected information, and the output is the encrypted selected information.

[0242] Step 8:

[0243] The server decodes the received selection information and generates a custom learning curriculum using a generative AI model. The input is the selection information and the output is the custom learning curriculum.

[0244] Step 9:

[0245] The server encrypts the generated curriculum and sends it to the terminal. The input is the custom curriculum, and the output is the encrypted curriculum.

[0246] Step 10:

[0247] The terminal decrypts the received curriculum and displays it to the user. The user can then view the customized curriculum. The input is the encrypted curriculum, and the output is the decrypted curriculum.

[0248] Step 11:

[0249] The user selects a virtual project within the learning curriculum, which is used as input data to proceed to the next step.

[0250] Step 12:

[0251] The terminal encrypts the project selection information and sends it to the server. The input is the project selection information, and the output is the encrypted project selection information.

[0252] Step 13:

[0253] The server decodes the received project selection information, configures the metaverse virtual environment, and loads the selected project content. The input is the project selection information, and the output is the virtual environment configuration information.

[0254] Step 14:

[0255] The server encrypts the virtual environment setting information and sends it to the terminal. The input is the virtual environment setting information, and the output is the encrypted virtual environment setting information.

[0256] Step 15:

[0257] The terminal decrypts the received virtual environment setting information and connects the user to the virtual environment. The input is the encrypted virtual environment setting information, and the output is the connection to the virtual environment.

[0258] Step 16:

[0259] A user works on a project in a virtual environment. The device sends completed tasks and progress information to the server in real time. The input is tasks and progress information, and the output is encrypted task and progress information.

[0260] Step 17:

[0261] The server interprets the received progress information and uses a generative AI model to generate feedback appropriate for the user. The input is progress information, and the output is customized feedback.

[0262] Step 18:

[0263] The server generates and encrypts the feedback and sends it to the device. The input is the customized feedback, and the output is the encrypted feedback.

[0264] Step 19:

[0265] The device decrypts the received feedback and displays it to the user, allowing them to receive specific advice based on their progress. The input is encrypted feedback, and the output is decrypted feedback.

[0266] Through this process, users can receive personalized learning experiences and optimal feedback in real time. The entire system is powered by a combination of hardware such as smartphones and head-mounted displays, and software such as Unity, AWS, and generative AI models.

[0267] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0268] This invention is a new learning system that combines generative AI, the Metaverse, and an emotion engine that recognizes users' emotions. This system provides users with customized learning experiences and practical projects, and provides real-time feedback and curriculum adjustments based on the user's emotions.

[0269] System configuration and program processing

[0270] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[0271] 1. User login and profile settings

[0272] User: Uses a terminal to access the system login screen and enters a username and password.

[0273] On the device: The login information entered by the user is encrypted and sent to the server.

[0274] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[0275] Server: Encrypts the profile data and sends it back to the device.

[0276] Terminal: Decodes the received profile data and displays it on the screen.

[0277] Examples:

[0278] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[0279] 2. Customize your learning curriculum

[0280] Users: Select the subject or topic they want to learn on their device.

[0281] Terminal: Encrypts the selected information and sends it to the server.

[0282] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[0283] Server: Encrypts the generated learning curriculum and sends it to the device.

[0284] Terminal: Decodes the received curriculum and displays it to the user.

[0285] Examples:

[0286] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0287] 3. Emotion recognition and curriculum adjustment using an emotion engine

[0288] Users: Experience changes in emotions (e.g., joy, confusion, anger, etc.) as they progress through the learning process.

[0289] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[0290] Terminal: The acquired data is sent to the emotion engine and the user's emotions are analyzed.

[0291] Server: Based on the analyzed emotional data, the learning curriculum and project content are adjusted as needed.

[0292] Examples:

[0293] If a user shows a confused expression during the project, the device captures the user's facial expression through the camera, and the emotion engine analyzes it to detect "confusion." The server uses this information to adjust the curriculum, for example, by providing additional explanations or hints.

[0294] 4. Setting up and participating in virtual projects

[0295] Users: Select a virtual project within the curriculum.

[0296] On your device: Encrypt the project selection information and send it to the server.

[0297] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[0298] Server: Encrypts the virtual environment configuration information and sends it to the device.

[0299] Terminal: Connects the user to the virtual environment based on the received configuration information.

[0300] Examples:

[0301] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[0302] 5. Personalized instruction and real-time feedback

[0303] Users: Work on projects in a virtual environment.

[0304] Device: Sends progress information to the server in real time.

[0305] Server: Analyzes progress information and uses generative AI to generate appropriate feedback for the user.

[0306] Server: Encrypts the generated feedback and sends it to the device.

[0307] Terminal: Display feedback on the user's screen.

[0308] Examples:

[0309] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0310] As described above, this system utilizes generative AI, metaverse technology, and an emotion engine to provide users with customized learning experiences and hands-on project experiences. Users can hone their skills in a realistic virtual environment and grow while receiving appropriate feedback in real time. Furthermore, the emotion engine enables instruction and curriculum to be adjusted based on the user's emotions, creating a more personalized learning experience.

[0311] The processing flow will be explained below.

[0312] Step 1:

[0313] The user uses a terminal to access the system's login screen and enters their username and password.

[0314] Step 2:

[0315] The terminal encrypts the entered login information and sends it to the server.

[0316] Step 3:

[0317] The server compares the received login information with the user data in the database and performs authentication.

[0318] Step 4:

[0319] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[0320] Step 5:

[0321] The device decodes the received profile data and displays it on the screen.

[0322] Step 6:

[0323] Users select the field or topic they want to learn through the device interface.

[0324] Step 7:

[0325] The terminal encrypts the selection information and transmits it to the server.

[0326] Step 8:

[0327] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[0328] Step 9:

[0329] The learning curriculum generated by the server is encrypted and sent to the terminal.

[0330] Step 10:

[0331] The terminal decodes the received curriculum and displays it to the user.

[0332] Step 11:

[0333] As the user progresses through the curriculum, the device uses a camera and microphone to capture facial expressions and voice in real time.

[0334] Step 12:

[0335] The device sends the acquired emotional data (facial expressions, voice, etc.) to the emotion engine, which analyzes the user's emotions.

[0336] Step 13:

[0337] The server adjusts the learning curriculum and project content as needed based on the analyzed emotional data.

[0338] Step 14:

[0339] A user selects a virtual project within a curriculum.

[0340] Step 15:

[0341] The terminal encrypts the project selection information and transmits it to the server.

[0342] Step 16:

[0343] The server sets up the metaverse virtual environment and loads the selected project contents.

[0344] Step 17:

[0345] The server encrypts the virtual environment configuration information and sends it to the terminal.

[0346] Step 18:

[0347] The terminal connects the user to the virtual environment based on the configuration information received.

[0348] Step 19:

[0349] Users work on projects in a virtual environment.

[0350] Step 20:

[0351] The device sends progress information to the server in real time.

[0352] Step 21:

[0353] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[0354] Step 22:

[0355] The server generates feedback that is encrypted and sent to the terminal.

[0356] Step 23:

[0357] The terminal displays the received feedback to the user and provides specific advice.

[0358] Example 2

[0359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Conventional learning systems have difficulty dynamically adjusting learning curriculum and feedback based on the user's individual emotions and progress, resulting in problems such as reduced learning efficiency and motivation. Furthermore, project experiences using virtual reality are limited, and a more realistic learning environment is needed.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0362] In this invention, the server includes a means for analyzing a user's emotional data and adjusting the curriculum, a means for providing customized real-time feedback using a generative AI model, and a means for providing a project experience using virtual reality technology, thereby personalizing the user's learning experience and enabling appropriate feedback and curriculum adjustment according to emotions and progress.

[0363] "User" refers to an individual who uses this system to learn or experience a project.

[0364] "Terminal" refers to an information processing device used by a user, such as a computer, tablet, or smartphone.

[0365] "Server" refers to the central computer device that performs the main processing and data management of the System.

[0366] "Login Information" means authentication information, such as a username and password, required for a User to access a System.

[0367] "Profile data" refers to data about a user, such as personal information, history, and learning progress.

[0368] "Learning curriculum" refers to the learning plans and materials created based on the field or topic selected by the user.

[0369] An "emotion analysis system" refers to a system that analyzes acquired facial expressions and voice data of a user and recognizes their emotions.

[0370] "Virtual environment" refers to a simulated environment constructed using virtual reality technology.

[0371] A "generative AI model" refers to an artificial intelligence model that generates learning curriculum and feedback based on user input data and situations.

[0372] "Real-time feedback" refers to advice and information provided instantly to users based on their progress and actions.

[0373] The system of the present invention is composed of three main components: a user, a terminal, and a server. The roles and operations of each component are described below.

[0374] System Overview

[0375] This system comprehensively handles everything from login authentication to learning curriculum generation, real-time feedback, virtual project setup and participation, and even curriculum adjustment based on sentiment analysis. By communicating with each other, the server, terminals, and users provide the optimal learning experience for users.

[0376] Hardware and software used

[0377] Device: Computer, tablet, smartphone, etc.

[0378] Server: A high-performance computing device that works in conjunction with a database.

[0379] Generative AI models: Artificial intelligence models such as GPT-3® and ChatGPT®.

[0380] Sentiment analysis system: Emotion detection API (e.g., Microsoft® Azure® Emotion API).

[0381] Virtual reality technology: Virtual environment engines that support the Metaverse (e.g., Unity, Unreal Engine).

[0382] Specific examples of program processing

[0383] 1. User login and profile settings

[0384] Examples:

[0385] User: Enter "user123" and "password" on the login screen.

[0386] Terminal: Encrypt "user123" and "password" with AES256 and send to the server.

[0387] Server: Search for "user123" in the database and match the password. If successful, retrieve the user's profile (e.g., name, email address).

[0388] Server: The acquired profile data is encrypted using AES256 and sent to the device.

[0389] Terminal: Decrypts the encrypted data and displays "Hello, user123" on the user's screen.

[0390] Example prompt sentence:

[0391] "Please enter your username and password to log in."

[0392] 2. Customize your learning curriculum

[0393] Examples:

[0394] User: Choose Machine Learning and Data Science.

[0395] Terminal: Encrypts the selected information and sends it to the server.

[0396] Server: Generates a curriculum for "Machine Learning and Data Science" using GPT-3.

[0397] Server: Encrypts the curriculum and sends it to the device.

[0398] Terminal: Decodes the curriculum and displays it to the user (e.g., "Chapter 1: Data Preprocessing, Chapter 2: Basic Algorithms").

[0399] Example prompt sentence:

[0400] "Choose the field you want to study. Example: Machine Learning, Data Science."

[0401] 3. Emotion recognition and curriculum adjustment using an emotion engine

[0402] Examples:

[0403] User: Shows confused expression while studying.

[0404] Device: Capture confused expressions through camera and microphone.

[0405] Terminal: Sends data to an emotion analysis system to recognize "confusion."

[0406] Server: Uses generative AI models to generate additional hints and explanations in response to puzzles and adjust the learning curriculum.

[0407] Example prompt sentence:

[0408] "Confused expression detected, show hint."

[0409] 4. Setting up and participating in virtual projects

[0410] Examples:

[0411] User: Choose a data science project.

[0412] Terminal: This selection information is encrypted and sent to the server.

[0413] Server: Sets up the metaverse virtual environment and loads project content (e.g., analyzing data, building models).

[0414] Server: Encrypts the configuration information and sends it to the device.

[0415] Terminal: Connects the user to the virtual environment.

[0416] Example prompt sentence:

[0417] "Choose a hypothetical project. Example: Data Analysis Project."

[0418] 5. Personalized instruction and real-time feedback

[0419] Examples:

[0420] User: Work on data analysis within the virtual environment.

[0421] Terminal: Sends work progress information (e.g., data preprocessing completed) to the server in real time.

[0422] Server: Parses progress information and generates feedback using ChatGPT (e.g., "Try a specific algorithm at this step").

[0423] Server: Encrypts the feedback information and sends it to the device.

[0424] Terminal: Display feedback information on the user's screen.

[0425] Example prompt sentence:

[0426] "We are sending you project status updates, please wait."

[0427] Through these steps, the system provides users with a customized, efficient, and meaningful learning experience. By combining generative AI models, sentiment analysis systems, and virtual reality technology, it is possible to monitor users' progress in real time and provide optimal feedback.

[0428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0429] The flow of this system's program processing

[0430] Step 1: Enter user login information

[0431] 1. Input:

[0432] User: Enter your username and password on the login screen.

[0433] 2. Processing:

[0434] Terminal: The entered username and password are encrypted using the AES256 algorithm.

[0435] 3. Output:

[0436] Encrypted login information is generated.

[0437] Step 2: Submit and authenticate login information

[0438] 1. Input:

[0439] Encrypted login information.

[0440] 2. Processing:

[0441] On your device: Sends encrypted login information to the server.

[0442] Server: receives the encrypted login information and authenticates it against the user data in the database (using the SHA-256 algorithm).

[0443] 3. Output:

[0444] The authentication result (success or failure).

[0445] On success, the user's profile data.

[0446] Step 3: Submit and view your profile data

[0447] 1. Input:

[0448] Authentication result and profile data (if successful).

[0449] 2. Processing:

[0450] Server: If authentication is successful, the profile data is encrypted with AES256 and sent to the device.

[0451] Terminal: Decrypts the received profile data and displays it on the user's screen.

[0452] 3. Output:

[0453] The user's profile will be displayed on the screen.

[0454] Step 4: Choose a subject or topic you want to study

[0455] 1. Input:

[0456] Users: Choose the field or topic they want to learn about (e.g., machine learning, data science).

[0457] 2. Processing:

[0458] Terminal: Encrypts the selected information and sends it to the server.

[0459] 3. Output:

[0460] Encrypted selection information.

[0461] Step 5: Generate and submit a custom learning curriculum

[0462] 1. Input:

[0463] Encrypted selection information.

[0464] 2. Processing:

[0465] Server: Decodes the selection information and generates a custom learning curriculum using a generative AI model (e.g., GPT-3).

[0466] Server: Encrypts the generated curriculum with AES256 and sends it to the device.

[0467] 3. Output:

[0468] Encrypted custom learning curriculum.

[0469] Step 6: Decipher and display the curriculum

[0470] 1. Input:

[0471] Encrypted custom learning curriculum.

[0472] 2. Processing:

[0473] Terminal: Decodes the received curriculum and displays it to the user.

[0474] 3. Output:

[0475] A customized learning curriculum is displayed to the user.

[0476] Step 7: Acquire and analyze emotion data

[0477] 1. Input:

[0478] User: Facial expressions and voice changes while learning.

[0479] 2. Processing:

[0480] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[0481] Terminal: The acquired data is sent to the emotion analysis system for analysis.

[0482] 3. Output:

[0483] Sentiment analysis results (e.g., happy, confused, angry).

[0484] Step 8: Adjust the curriculum

[0485] 1. Input:

[0486] Emotion analysis results.

[0487] 2. Processing:

[0488] Server: Based on the analyzed emotional data, the learning curriculum is adjusted as needed.

[0489] 3. Output:

[0490] Coordinated learning curriculum.

[0491] Step 9: Select and configure your virtual project

[0492] 1. Input:

[0493] User: Select a virtual project.

[0494] 2. Processing:

[0495] On your device: Encrypt the project selection information and send it to the server.

[0496] Server: Sets up the metaverse virtual environment based on the selected project and loads the project.

[0497] Server: Encrypts the configuration information and sends it to the device.

[0498] 3. Output:

[0499] Virtual environment configuration data.

[0500] Step 10: Connect to your virtual environment

[0501] 1. Input:

[0502] Virtual environment configuration data.

[0503] 2. Processing:

[0504] Terminal: Connects the user to the virtual environment based on the received configuration data.

[0505] 3. Output:

[0506] The user is connected to the virtual environment.

[0507] Step 11: Submit project work progress and generate feedback

[0508] 1. Input:

[0509] User: Working on a project in a virtual environment.

[0510] 2. Processing:

[0511] Terminal: Sends work progress information to the server in real time.

[0512] Server: Analyzes progress information and generates optimal feedback using generative AI models.

[0513] Server: Encrypts the feedback and sends it to the device.

[0514] 3. Output:

[0515] Real-time feedback.

[0516] Step 12: Viewing feedback

[0517] 1. Input:

[0518] Encrypted real-time feedback.

[0519] 2. Processing:

[0520] Terminal: Decodes the feedback and displays it to the user.

[0521] 3. Output:

[0522] Appropriate feedback is provided to the user.

[0523] The above are the specific processing steps of this system's program, and we have explained in detail how the data is input at each step, the data processing and calculations performed, and how the output is obtained. By combining these steps, it is possible to provide the user with an optimal learning experience.

[0524] (Application example 2)

[0525] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0526] Conventional learning systems have difficulty adjusting in real time to the user's learning progress and emotions, and tend to provide a one-way curriculum. As a result, they are unable to flexibly respond to the user's level of understanding and motivation. Furthermore, even in methods that provide practical learning experiences using virtual environments, the lack of feedback based on the user's emotions and progress can reduce learning effectiveness. Furthermore, there is room for improvement in the operability and accuracy of real-time feedback when using devices such as smartphones and head-mounted displays.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0528] In this invention, the server includes a means for analyzing the user's facial expressions and voice using an emotion engine and adjusting the curriculum based on real-time feedback, a means for the smartphone or head-mounted display to acquire the user's emotion data and send it to the server, and a means for the server to monitor the user's progress based on the emotion data and provide real-time feedback using generative AI, thereby making it possible to provide a customized learning experience according to the user's individual progress and emotions.

[0529] A "terminal" is a device such as a computer, smartphone, tablet, or head-mounted display that a user uses as an interface.

[0530] The "server" is a central processing unit that manages user login information, learning data, and emotional data, and generates and transmits profile data, learning curriculum, feedback, etc.

[0531] An "emotion engine" is software or a system that analyzes a user's voice and facial expression data to identify their emotional state.

[0532] "Generative AI" refers to artificial intelligence that uses machine learning and natural language processing to generate and analyze data.

[0533] "Login information" refers to identification information, such as a username and password, required for a user to access a system.

[0534] "Profile data" refers to data including the user's personal information, learning history, setting information, and the like.

[0535] A "learning curriculum" is a customized learning content and schedule based on a user's chosen field or topic.

[0536] A "Virtual Project" is a virtual learning or work assignment or project that a user can experience within the Metaverse.

[0537] A "virtual environment" is a three-dimensional virtual space constructed using metaverse technology in which users can have virtual experiences.

[0538] "Real-time feedback" is a function that responds immediately to the user's learning progress and emotional state, providing appropriate advice and curriculum adjustments.

[0539] A "smartphone" is a type of mobile device, an advanced portable terminal that can be used to install and use communication functions and a variety of applications.

[0540] A "head-mounted display" is a display device worn on a user's head and is used to provide visual information.

[0541] The system of the present invention combines generative AI, an emotion engine, and metaverse technology to provide users with a customized learning experience. This system is composed of three main components: a server, a terminal, and a user, and is realized through the interaction of these components. A specific embodiment of this system is described below.

[0542] First, the user accesses the login screen using a device (smartphone, tablet, head-mounted display, etc.) and enters their username and password. The device encrypts this login information and sends it to the server. The server compares the received login information with user data in a database and performs authentication. If authentication is successful, the server obtains the profile data, encrypts it, and sends it back to the device. The device decrypts the received profile data and displays it on the user's screen.

[0543] Next, the user selects the field or topic they want to study on their device. The device encrypts the selection and sends it to the server. The server uses generative AI to generate a custom learning curriculum based on the user's selection. The generated learning curriculum is sent from the server to the device, where it is decrypted and displayed to the user.

[0544] As the user progresses with their learning, the device uses cameras and sensors to capture changes in the user's emotions (happiness, confusion, anger, etc.) in real time. The captured data is sent to an emotion engine, which analyzes the user's emotional state. Based on the analyzed data, the server adjusts the necessary parts of the learning curriculum and provides feedback to the user.

[0545] When a user selects a virtual project, the information is sent from the device to the server, which then uses metaverse technology to configure the virtual environment and load the selected project content. The server then sends the environment configuration information to the device, which then connects the user to the virtual environment.

[0546] To achieve these functions, the following hardware and software are used:

[0547] Hardware: smartphones, tablets, head-mounted displays, cameras, sensors

[0548] Software: Python, OpenCV (image processing library), transformers (natural language processing library), Requests (library for API requests)

[0549] For example, if a user selects that they want to learn data science, the device sends the selection information to the server. The server uses generative AI to generate a custom curriculum, which the device then displays to the user. Furthermore, if the user shows a confused expression during the project, the device captures facial expression data through the camera, and the emotion engine detects this as "confusion." Based on this information, the server can provide additional explanations and hints.

[0550] Example prompts to input to a generative AI model:

[0551] A user is confused about: [machine learning algorithm parameter tuning]

[0552] Example of feedback generated based on this prompt:

[0553] "In this step, it's important to understand the role of each parameter. Parameter A adjusts the scale of the data, and parameter B controls the learning rate. For example..."

[0554] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0555] Step 1:

[0556] The user accesses the login screen using a terminal and enters their username and password.

[0557] Input: Username, Password

[0558] Output: Encrypted login information

[0559] Specific behavior:

[0560] The user enters their username and password into the login screen of their device, which then encrypts the information using the AES encryption algorithm and sends it to the server.

[0561] Step 2:

[0562] The server receives the encrypted login information, decrypts it, and authenticates it by checking it against user data in a database.

[0563] Input: Encrypted login information

[0564] Output: Authentication results, profile data

[0565] Specific behavior:

[0566] The server decrypts the received login information using the AES decryption algorithm and compares it with the user data in its database. If authentication is successful, it retrieves the profile data, encrypts it, and sends it back to the device.

[0567] Step 3:

[0568] The terminal decrypts the profile data received from the server and displays it on the user screen.

[0569] Input: Encrypted profile data

[0570] Output: Display profile information

[0571] Specific behavior:

[0572] The device will then decrypt the received encrypted profile data using the AES decryption algorithm and display it on the user's screen, allowing the user to view their profile information.

[0573] Step 4:

[0574] The user selects the field or topic they wish to study on their device and sends the selected information to the server.

[0575] Input: Select the field or topic you want to study

[0576] Output: Encrypted selection information

[0577] Specific behavior:

[0578] Users select the fields and topics they want to learn on their device, which then encrypts this selection using the AES encryption algorithm and sends it to the server.

[0579] Step 5:

[0580] The server receives the encrypted selection information, decrypts it, and uses generative AI to generate a custom learning curriculum.

[0581] Input: Encrypted selection information

[0582] Output: Custom learning curriculum

[0583] Specific behavior:

[0584] The server decrypts the received selection information using the AES decryption algorithm and inputs it into the generative AI model, which then generates a custom learning curriculum based on the user's selection, encrypts it, and sends it to the device.

[0585] Step 6:

[0586] The terminal receives the encrypted learning curriculum, decrypts it, and displays it to the user.

[0587] Input: Encrypted learning curriculum

[0588] Output: Display of learning curriculum

[0589] Specific behavior:

[0590] The terminal decrypts the received encrypted curriculum using the AES decryption algorithm and displays it on the user's screen, allowing the user to view the customized curriculum.

[0591] Step 7:

[0592] As the user progresses with their learning, the device uses cameras and sensors to capture facial and voice data in real time and sends it to the emotion engine.

[0593] Input: User's facial expression data, voice data

[0594] Output: Emotion analysis results

[0595] Specific behavior:

[0596] The device uses cameras and sensors to capture the user's facial expressions and voice in real time and transmits the data to the emotion engine, which analyzes this data to detect the user's emotional state.

[0597] Step 8:

[0598] The server adjusts the necessary parts of the learning curriculum based on the emotional data received from the emotion engine, generates real-time feedback, and sends it to the terminal.

[0599] Input: Sentiment analysis results, learning progress data

[0600] Output: Curriculum adjustments, real-time feedback

[0601] Specific behavior:

[0602] The server generates customized feedback based on the emotion analysis results and learning progress data. The feedback content is created using a generative AI model, encrypted, and sent to the device.

[0603] Step 9:

[0604] The terminal receives the encrypted feedback, decrypts it and displays it on the user's screen.

[0605] Input: Encrypted feedback

[0606] Output: Display of feedback content

[0607] Specific behavior:

[0608] The device uses the AES decryption algorithm to decrypt the received encrypted feedback and displays it on the user's screen, where the user can see specific advice and tips from the generated AI model.

[0609] Step 10:

[0610] The user selects a virtual project and the terminal transmits the information to the server.

[0611] Input: Virtual project selection information

[0612] Output: Encrypted selection information

[0613] Specific behavior:

[0614] The user selects a virtual project on the terminal, which encrypts the selection information using the AES encryption algorithm and sends it to the server.

[0615] Step 11:

[0616] The server receives the encrypted selection information and uses metaverse technology to set up a virtual environment and load the project content.

[0617] Input: Encrypted selection information

[0618] Output: Virtual environment configuration information

[0619] Specific behavior:

[0620] The server decrypts the received selection information using the AES decryption algorithm, sets up a virtual environment using metaverse technology, loads the selected project content, and sends the configuration information to the terminal.

[0621] Step 12:

[0622] The terminal receives the setting information of the virtual environment and connects the user to the virtual environment.

[0623] Input: Virtual environment configuration information

[0624] Output: Virtual environment connected

[0625] Specific behavior:

[0626] The device decodes the received virtual environment configuration information and connects the user to the virtual environment, allowing the user to participate in virtual projects and progress in their learning.

[0627] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0628] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0629] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0630] [Second embodiment]

[0631] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0632] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0633] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0634] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0635] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0636] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0637] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0638] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0639] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0640] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0641] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0642] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0643] This invention is a new learning system that combines generative AI and the Metaverse, aiming to provide users with customized learning experiences and practical project experiences.

[0644] System configuration and program processing

[0645] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[0646] 1. User login and profile settings

[0647] User: Uses a terminal to access the system login screen and enters a username and password.

[0648] On the device: The login information entered by the user is encrypted and sent to the server.

[0649] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[0650] Server: Encrypts the profile data and sends it back to the device.

[0651] Terminal: Decodes the received profile data and displays it on the screen.

[0652] Examples:

[0653] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[0654] 2. Customize your learning curriculum

[0655] Users: Choose the subject or topic they want to learn about.

[0656] Terminal: Encrypts the selected information and sends it to the server.

[0657] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[0658] Server: Encrypts the generated learning curriculum and sends it to the device.

[0659] Terminal: Decodes the received curriculum and displays it to the user.

[0660] Examples:

[0661] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0662] 3. Setting up and participating in virtual projects

[0663] Users: Select a virtual project within the curriculum.

[0664] On your device: Encrypt the project selection information and send it to the server.

[0665] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[0666] Server: Encrypts the virtual environment configuration information and sends it to the device.

[0667] Terminal: Connects the user to the virtual environment based on the received configuration information.

[0668] Examples:

[0669] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[0670] 4. Personalized instruction and real-time feedback

[0671] Users: Work on projects in a virtual environment.

[0672] Device: Sends completed tasks and progress information to the server in real time.

[0673] Server: Analyzes progress information and uses generative AI to generate feedback appropriate for the user.

[0674] Server: Encrypts the generated feedback and sends it to the device.

[0675] Terminal: Display feedback on the user's screen.

[0676] Examples:

[0677] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0678] As described above, this system leverages generative AI and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to grow.

[0679] The processing flow will be explained below.

[0680] Step 1:

[0681] The user uses a terminal to access the system's login screen and enters their username and password.

[0682] Step 2:

[0683] The terminal encrypts the entered login information and sends it to the server.

[0684] Step 3:

[0685] The server compares the received login information with the user data in the database and performs authentication.

[0686] Step 4:

[0687] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[0688] Step 5:

[0689] The device decodes the received profile data and displays it on the screen.

[0690] Step 6:

[0691] Users select the field or topic they want to learn on their device.

[0692] Step 7:

[0693] The terminal encrypts the selected information and sends it to the server.

[0694] Step 8:

[0695] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[0696] Step 9:

[0697] The learning curriculum generated by the server is encrypted and sent to the terminal.

[0698] Step 10:

[0699] The terminal decodes the received curriculum and displays it to the user.

[0700] Step 11:

[0701] A user selects a virtual project within a curriculum.

[0702] Step 12:

[0703] The terminal encrypts the project selection information and transmits it to the server.

[0704] Step 13:

[0705] The server sets up the metaverse virtual environment and loads the selected project contents.

[0706] Step 14:

[0707] The server encrypts the virtual environment configuration information and sends it to the terminal.

[0708] Step 15:

[0709] The terminal connects the user to the virtual environment based on the configuration information received.

[0710] Step 16:

[0711] Users work on projects in a virtual environment.

[0712] Step 17:

[0713] The device sends progress information to the server in real time.

[0714] Step 18:

[0715] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[0716] Step 19:

[0717] The server generates feedback that is encrypted and sent to the terminal.

[0718] Step 20:

[0719] The terminal displays the received feedback to the user and provides specific advice.

[0720] Example 1

[0721] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] Conventional learning systems often offer only a standard curriculum, making it difficult to customize to meet individual user needs. They also struggle to provide real-time feedback and practical project experiences. Providing a learning environment utilizing a virtual reality environment, in particular, poses numerous technical challenges and is difficult to implement.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0724] In this invention, the server includes means for authenticating login information and obtaining user profile data, means for generating a learning curriculum using a generative AI model based on the selection information, means for configuring a metaverse environment and loading the selected projects, and means for monitoring completed tasks and progress information in real time and providing real-time feedback using the generative AI model, thereby providing a personalized learning experience and hands-on project experience with real-time feedback.

[0725] "User" refers to the person who operates the system and experiences the learning curriculum and virtual projects.

[0726] "Device" refers to a computer or mobile device on which a user performs operations such as entering login information, selecting a course of study, or participating in a virtual project.

[0727] "Server" refers to a central processing unit that authenticates user login information, obtains profile data, generates learning curricula using generative AI models, sets up the metaverse environment, and generates real-time feedback.

[0728] "Generative AI model" refers to an artificial intelligence algorithm used to generate personalized curriculum based on user selections and provide real-time feedback.

[0729] "Login Information" refers to authentication information, such as a username and password, entered by a User to access a System.

[0730] "Profile Data" refers to data that includes personal information such as a user's name, learning history, and settings.

[0731] A "learning curriculum" refers to a series of learning content and assignments generated based on the field or topic a user wants to learn.

[0732] A "virtual project" refers to a specific challenge or task that a user participates in and completes within a virtual reality environment.

[0733] A "metaverse environment" refers to a virtual work or learning environment built using virtual reality technology.

[0734] "Real-time feedback" refers to instantaneous evaluation and advice provided to users based on their progress.

[0735] This invention is a new learning system that combines generative AI models and the Metaverse. The purpose of this system is to provide users with customized learning experiences and practical project experiences. Below, we will explain how to implement this system in detail.

[0736] This system consists of three main components: a server, a terminal, and a user.

[0737] Hardware and software used

[0738] Server: Serves as a central processing unit, authenticating users, managing profile data, generating learning curricula using generative AI models (e.g., GPT-4), configuring the metaverse environment, and generating real-time feedback. Implemented in a server system that also includes databases and AI algorithms.

[0739] Terminal: A device operated by a user, such as a computer, smartphone, or tablet. The software on the terminal provides the user interface, performs encryption / decryption processing, communicates with the server, and displays information to the user.

[0740] User: An individual who uses the learning system. The user operates the device and progresses with the learning.

[0741] Program processing

[0742] User login and profile settings

[0743] A user uses a terminal to access the system's login screen and enters their username and password. The terminal encrypts this login information (e.g., AES encryption) and sends it to the server. The server decrypts the received login information and authenticates it by comparing it with information in a database. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it to the terminal. The terminal decrypts the data and displays it to the user.

[0744] Customized learning curriculum

[0745] The user selects the field or topic they wish to study. The device encrypts the selection information (e.g., RSA encryption) and sends it to the server. The server analyzes the received selection information using a generative AI model (e.g., GPT-4) and generates a learning curriculum optimal for the user. The generated curriculum is encrypted and sent to the device. The device decrypts the curriculum and displays it to the user.

[0746] Examples:

[0747] The user selects that they want to learn machine learning and data science. The device RSA-encrypts this information and sends it to the server. The server uses GPT-4 to generate a customized curriculum and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0748] Setting up and participating in virtual projects

[0749] The user selects a virtual project within the curriculum. The device encrypts the selection information and sends it to the server. The server analyzes the received information and sets up the virtual project within the metaverse environment. The set virtual environment information is then encrypted and sent to the device. The device decrypts it and connects the user to the virtual environment.

[0750] Examples:

[0751] The user selects a virtual data science project. The device encrypts this selection information and sends it to the server. The server configures the metaverse environment and loads the project content. The server encrypts the environment configuration information and sends it to the device, which then connects the user to the virtual environment.

[0752] Personalized instruction and real-time feedback

[0753] Users work on projects in a virtual environment. The device sends completed tasks and progress information to the server in real time. The server analyzes the progress information and generates appropriate feedback for the user using a generative AI model (e.g., Chat GPT). The generated feedback is encrypted and sent to the device, which then displays the feedback on the user's screen.

[0754] Examples:

[0755] The user performs data analysis in a virtual environment. The device sends progress information to the server in real time. The server uses this information to generate feedback using a generative AI model and sends it to the device. The device then displays the feedback on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0756] Prompt Sentence Examples

[0757] "Users want to learn machine learning from the basics to advanced applications. Please generate a customized learning curriculum."

[0758] As described above, the system leverages generative AI models and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to help them grow.

[0759] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0760] Step 1:

[0761] A user accesses the login screen of the system using a terminal and enters a user name and password, which become the input information.

[0762] The terminal encrypts the entered login information using the AES encryption algorithm and sends it to the server, where it outputs the encrypted login information.

[0763] Step 2:

[0764] The server decrypts the received encrypted login information using the AES decryption algorithm.

[0765] The server compares the decrypted login information with the user data in the database and performs authentication. The user data from the database is used as input information. If authentication is successful, the user's profile data is retrieved.

[0766] The profile data is encrypted using the AES encryption algorithm and sent to the terminal, and this encrypted profile data is the output.

[0767] Step 3:

[0768] The terminal decrypts the received encrypted profile data using the AES decryption algorithm.

[0769] The decrypted profile data is displayed on the user's screen, which is the output.

[0770] Step 4:

[0771] The user selects the field or topic they want to learn about, and this selection information becomes the input information.

[0772] The terminal encrypts the user's selection information using the RSA encryption algorithm and sends it to the server, where the encrypted selection information is output.

[0773] Step 5:

[0774] The server decrypts the received encrypted selection information using the RSA decryption algorithm.

[0775] The server generates a custom learning curriculum using a generative AI model (e.g., GPT-4) based on the decoded selection information. This selection information serves as input information.

[0776] The generated learning curriculum is encrypted using the AES encryption algorithm and sent to the terminal. This encrypted learning curriculum is the output.

[0777] Step 6:

[0778] The terminal decrypts the received encrypted learning curriculum using the AES decryption algorithm.

[0779] The decoded learning curriculum is displayed on the user's screen, which is the output.

[0780] Step 7:

[0781] The user selects a virtual project within the curriculum, and this selection information becomes the input information.

[0782] The device hashes the project selection information using the SHA-256 hashing algorithm and sends it to the server, where the hashed selection information is the output.

[0783] Step 8:

[0784] The server parses the received hashed project selection information.

[0785] The server configures the metaverse environment and loads the selected project contents. This configuration information becomes the input information.

[0786] The configured virtual environment information is hashed using SHA-256 and sent to the terminal. This hashed configuration information is output.

[0787] Step 9:

[0788] The terminal connects the user to the metaverse virtual project based on the received hashed virtual environment information.

[0789] The output is that the user is connected to the virtual environment.

[0790] Step 10:

[0791] Users work on projects within a virtual environment, and their actions within this environment provide input information.

[0792] The terminal transmits completed tasks and progress information to the server in real time using the WebSocket protocol, and this progress information is the output.

[0793] Step 11:

[0794] The server analyzes the received progress information and generates optimal feedback for the user using a generative AI model (e.g., Chat GPT). The progress information is used as input.

[0795] The generated feedback is encrypted via WebSocket and sent to the terminal. This encrypted feedback becomes the output.

[0796] Step 12:

[0797] The terminal decrypts the received encrypted feedback.

[0798] The decoded feedback is displayed on the user's screen, which is the output.

[0799] In this way, by performing specific operations and processing and calculating data at each step, it is possible to provide the user with a customized learning experience.

[0800] (Application example 1)

[0801] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0802] Conventional learning systems have struggled to provide customized learning tailored to individual users' needs or practical project experiences in virtual environments. They also struggled to provide optimal feedback on users' learning progress in real time. Especially in distance learning and online education, there is a lack of individualized instruction, which can lead to insufficient learning outcomes.

[0803] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0804] In this invention, the server includes a means for generating customized advice based on the user's learning progress and displaying it to the user on the terminal, a means for providing real-time feedback using generative AI, and a means for providing a project experience in a virtual reality environment using the metaverse, allowing users to enjoy customized learning and practical project experience tailored to their individual needs and receive optimal feedback in real time.

[0805] "User" refers to someone who uses the system to study or work on a project.

[0806] "Terminal" refers to a device operated by a user, used to input login information and selection information, display profile data and learning curriculum, connect to a virtual environment, and so on.

[0807] "Server" refers to a device that receives and processes information sent from a terminal and has functions such as user authentication, acquisition of profile data, generation of learning curriculum, setting up a virtual environment, and generation of real-time feedback.

[0808] "Login Information" refers to the authentication information required for a user to access a system, and typically consists of a username and password.

[0809] "Profile data" refers to data that includes information such as the user's personal information, learning history, and skill level, and is acquired by the server and returned to the terminal when authentication is successful.

[0810] "Learning curriculum" refers to the learning content or study plan for a specific field or topic that is generated by the server based on user selection information.

[0811] A "virtual project" refers to a specific assignment or task selected by a user and executed in a virtual environment within the metaverse.

[0812] The "metaverse" is a virtual environment constructed using virtual reality technology, and refers to a virtual space in which users can interact with each other.

[0813] "Generative AI" refers to artificial intelligence technology that generates appropriate advice and feedback based on a user's learning progress and project content.

[0814] "Feedback" refers to the evaluation and advice provided in real time by generative AI for tasks or projects that users are currently working on.

[0815] "Customized advice" refers to personalized advice that suggests specific content and methods based on the user's learning progress and skill level.

[0816] Three main components are required to implement this invention: a server, a terminal, and a user. The system provides users with a customized learning experience and project experience in a virtual environment.

[0817] First, a user uses a terminal to access the system's login screen. Here, they enter their username and password. The terminal encrypts the entered login information and sends it to the server. The server compares the received login information with the user data in its database and performs authentication. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it back to the terminal. The terminal decrypts the profile data and displays it to the user.

[0818] Next, the learning curriculum is customized. The user selects the fields and topics they want to study. The device encrypts these selections and sends them to the server. The server uses a generative AI model to generate a custom learning curriculum based on the user's selections. The generated curriculum is then encrypted again and sent to the device. The device decrypts the curriculum data and displays it to the user.

[0819] The user then selects a virtual project within the curriculum. The device encrypts the project selection information and sends it to the server. The server then configures the metaverse virtual environment and loads the selected project content. The virtual environment configuration information is then encrypted and sent to the device. The device then connects the user to the virtual environment based on the configuration information.

[0820] As a user works on a project in a virtual environment, the device sends completed tasks and progress information to the server in real time. The server analyzes this information and uses a generative AI model to generate appropriate feedback for the user. The generated feedback is encrypted and sent to the device, which then decrypts it and displays it to the user. This feedback can include specific advice, such as "Try a different regression algorithm for this step."

[0821] The hardware used includes smartphones and head-mounted displays (HMDs), and the software includes Unity (for building the virtual environment), AWS (for servers and databases), and OpenAI's GPT-4 generative AI.

[0822] Below are some examples of prompt sentences.

[0823] "User wants feedback on data analysis. Please provide specific advice."

[0824] The system allows users to receive customized learning tailored to their individual needs and improve their skills through hands-on project experience in a virtual environment, with real-time feedback providing a significant learning experience.

[0825] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0826] Step 1:

[0827] The user accesses the login screen of the learning system using a smartphone or head-mounted display (HMD). The user enters their username and password. This is used as input data to proceed to the next step.

[0828] Step 2:

[0829] The terminal encrypts the login information entered by the user. This encrypted data is sent to the server. The input is the login information, and the output is the encrypted login data.

[0830] Step 3:

[0831] The server decrypts the encrypted login information it receives and authenticates it by comparing it with the user data in the database. If authentication is successful, it obtains the user's profile data. The input is the encrypted login data, and the output is the user's profile data.

[0832] Step 4:

[0833] The server re-encrypts the acquired profile data and sends it to the terminal. The input is the profile data, and the output is the encrypted profile data.

[0834] Step 5:

[0835] The device decrypts the received profile data and displays it to the user, allowing the user to check their own profile information. The input is encrypted profile data, and the output is decrypted profile data.

[0836] Step 6:

[0837] The user selects the field or topic they want to study, and with this as input, they move on to the next step.

[0838] Step 7:

[0839] The terminal encrypts information about the learning field or topic selected by the user and sends it to the server. The input is the selected information, and the output is the encrypted selected information.

[0840] Step 8:

[0841] The server decodes the received selection information and generates a custom learning curriculum using a generative AI model. The input is the selection information and the output is the custom learning curriculum.

[0842] Step 9:

[0843] The server encrypts the generated curriculum and sends it to the terminal. The input is the custom curriculum, and the output is the encrypted curriculum.

[0844] Step 10:

[0845] The terminal decrypts the received curriculum and displays it to the user. The user can then view the customized curriculum. The input is the encrypted curriculum, and the output is the decrypted curriculum.

[0846] Step 11:

[0847] The user selects a virtual project within the learning curriculum, which is used as input data to proceed to the next step.

[0848] Step 12:

[0849] The terminal encrypts the project selection information and sends it to the server. The input is the project selection information, and the output is the encrypted project selection information.

[0850] Step 13:

[0851] The server decodes the received project selection information, configures the metaverse virtual environment, and loads the selected project content. The input is the project selection information, and the output is the virtual environment configuration information.

[0852] Step 14:

[0853] The server encrypts the virtual environment setting information and sends it to the terminal. The input is the virtual environment setting information, and the output is the encrypted virtual environment setting information.

[0854] Step 15:

[0855] The terminal decrypts the received virtual environment setting information and connects the user to the virtual environment. The input is the encrypted virtual environment setting information, and the output is the connection to the virtual environment.

[0856] Step 16:

[0857] A user works on a project in a virtual environment. The device sends completed tasks and progress information to the server in real time. The input is tasks and progress information, and the output is encrypted task and progress information.

[0858] Step 17:

[0859] The server interprets the received progress information and uses a generative AI model to generate feedback appropriate for the user. The input is progress information, and the output is customized feedback.

[0860] Step 18:

[0861] The server generates and encrypts the feedback and sends it to the device. The input is the customized feedback, and the output is the encrypted feedback.

[0862] Step 19:

[0863] The device decrypts the received feedback and displays it to the user, allowing them to receive specific advice based on their progress. The input is encrypted feedback, and the output is decrypted feedback.

[0864] Through this process, users can receive personalized learning experiences and optimal feedback in real time. The entire system is powered by a combination of hardware such as smartphones and head-mounted displays, and software such as Unity, AWS, and generative AI models.

[0865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0866] This invention is a new learning system that combines generative AI, the Metaverse, and an emotion engine that recognizes users' emotions. This system provides users with customized learning experiences and practical projects, and provides real-time feedback and curriculum adjustments based on the user's emotions.

[0867] System configuration and program processing

[0868] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[0869] 1. User login and profile settings

[0870] User: Uses a terminal to access the system login screen and enters a username and password.

[0871] On the device: The login information entered by the user is encrypted and sent to the server.

[0872] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[0873] Server: Encrypts the profile data and sends it back to the device.

[0874] Terminal: Decodes the received profile data and displays it on the screen.

[0875] Examples:

[0876] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[0877] 2. Customize your learning curriculum

[0878] Users: Select the subject or topic they want to learn on their device.

[0879] Terminal: Encrypts the selected information and sends it to the server.

[0880] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[0881] Server: Encrypts the generated learning curriculum and sends it to the device.

[0882] Terminal: Decodes the received curriculum and displays it to the user.

[0883] Examples:

[0884] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[0885] 3. Emotion recognition and curriculum adjustment using an emotion engine

[0886] Users: Experience changes in emotions (e.g., joy, confusion, anger, etc.) as they progress through the learning process.

[0887] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[0888] Terminal: The acquired data is sent to the emotion engine and the user's emotions are analyzed.

[0889] Server: Based on the analyzed emotional data, the learning curriculum and project content are adjusted as needed.

[0890] Examples:

[0891] If a user shows a confused expression during the project, the device captures the user's facial expression through the camera, and the emotion engine analyzes it to detect "confusion." The server uses this information to adjust the curriculum, for example, by providing additional explanations or hints.

[0892] 4. Setting up and participating in virtual projects

[0893] Users: Select a virtual project within the curriculum.

[0894] On your device: Encrypt the project selection information and send it to the server.

[0895] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[0896] Server: Encrypts the virtual environment configuration information and sends it to the device.

[0897] Terminal: Connects the user to the virtual environment based on the received configuration information.

[0898] Examples:

[0899] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[0900] 5. Personalized instruction and real-time feedback

[0901] Users: Work on projects in a virtual environment.

[0902] Device: Sends progress information to the server in real time.

[0903] Server: Analyzes progress information and uses generative AI to generate appropriate feedback for the user.

[0904] Server: Encrypts the generated feedback and sends it to the device.

[0905] Terminal: Display feedback on the user's screen.

[0906] Examples:

[0907] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[0908] As described above, this system utilizes generative AI, metaverse technology, and an emotion engine to provide users with customized learning experiences and hands-on project experiences. Users can hone their skills in a realistic virtual environment and grow while receiving appropriate feedback in real time. Furthermore, the emotion engine enables instruction and curriculum to be adjusted based on the user's emotions, creating a more personalized learning experience.

[0909] The processing flow will be explained below.

[0910] Step 1:

[0911] The user uses a terminal to access the system's login screen and enters their username and password.

[0912] Step 2:

[0913] The terminal encrypts the entered login information and sends it to the server.

[0914] Step 3:

[0915] The server compares the received login information with the user data in the database and performs authentication.

[0916] Step 4:

[0917] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[0918] Step 5:

[0919] The device decodes the received profile data and displays it on the screen.

[0920] Step 6:

[0921] Users select the field or topic they want to learn through the device interface.

[0922] Step 7:

[0923] The terminal encrypts the selection information and transmits it to the server.

[0924] Step 8:

[0925] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[0926] Step 9:

[0927] The learning curriculum generated by the server is encrypted and sent to the terminal.

[0928] Step 10:

[0929] The terminal decodes the received curriculum and displays it to the user.

[0930] Step 11:

[0931] As the user progresses through the curriculum, the device uses a camera and microphone to capture facial expressions and voice in real time.

[0932] Step 12:

[0933] The device sends the acquired emotional data (facial expressions, voice, etc.) to the emotion engine, which analyzes the user's emotions.

[0934] Step 13:

[0935] The server adjusts the learning curriculum and project content as needed based on the analyzed emotional data.

[0936] Step 14:

[0937] A user selects a virtual project within a curriculum.

[0938] Step 15:

[0939] The terminal encrypts the project selection information and transmits it to the server.

[0940] Step 16:

[0941] The server sets up the metaverse virtual environment and loads the selected project contents.

[0942] Step 17:

[0943] The server encrypts the virtual environment configuration information and sends it to the terminal.

[0944] Step 18:

[0945] The terminal connects the user to the virtual environment based on the configuration information received.

[0946] Step 19:

[0947] Users work on projects in a virtual environment.

[0948] Step 20:

[0949] The device sends progress information to the server in real time.

[0950] Step 21:

[0951] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[0952] Step 22:

[0953] The server generates feedback that is encrypted and sent to the terminal.

[0954] Step 23:

[0955] The terminal displays the received feedback to the user and provides specific advice.

[0956] Example 2

[0957] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0958] Conventional learning systems have difficulty dynamically adjusting learning curriculum and feedback based on the user's individual emotions and progress, resulting in problems such as reduced learning efficiency and motivation. Furthermore, project experiences using virtual reality are limited, and a more realistic learning environment is needed.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0960] In this invention, the server includes a means for analyzing a user's emotional data and adjusting the curriculum, a means for providing customized real-time feedback using a generative AI model, and a means for providing a project experience using virtual reality technology, thereby personalizing the user's learning experience and enabling appropriate feedback and curriculum adjustment according to emotions and progress.

[0961] "User" refers to an individual who uses this system to learn or experience a project.

[0962] "Terminal" refers to an information processing device used by a user, such as a computer, tablet, or smartphone.

[0963] "Server" refers to the central computer device that performs the main processing and data management of the System.

[0964] "Login Information" means authentication information, such as a username and password, required for a User to access a System.

[0965] "Profile data" refers to data about a user, such as personal information, history, and learning progress.

[0966] "Learning curriculum" refers to the learning plans and materials created based on the field or topic selected by the user.

[0967] An "emotion analysis system" refers to a system that analyzes acquired facial expressions and voice data of a user and recognizes their emotions.

[0968] "Virtual environment" refers to a simulated environment constructed using virtual reality technology.

[0969] A "generative AI model" refers to an artificial intelligence model that generates learning curriculum and feedback based on user input data and situations.

[0970] "Real-time feedback" refers to advice and information provided instantly to users based on their progress and actions.

[0971] The system of the present invention is composed of three main components: a user, a terminal, and a server. The roles and operations of each component are described below.

[0972] System Overview

[0973] This system comprehensively handles everything from login authentication to learning curriculum generation, real-time feedback, virtual project setup and participation, and even curriculum adjustment based on sentiment analysis. By communicating with each other, the server, terminals, and users provide the optimal learning experience for users.

[0974] Hardware and software used

[0975] Device: Computer, tablet, smartphone, etc.

[0976] Server: A high-performance computing device that works in conjunction with a database.

[0977] Generative AI models: Artificial intelligence models such as GPT-3 and ChatGPT.

[0978] Sentiment analysis system: Emotion detection API (e.g. Microsoft Azure Emotion API).

[0979] Virtual reality technology: Virtual environment engines that support the Metaverse (e.g., Unity, Unreal Engine).

[0980] Specific examples of program processing

[0981] 1. User login and profile settings

[0982] Examples:

[0983] User: Enter "user123" and "password" on the login screen.

[0984] Terminal: Encrypt "user123" and "password" with AES256 and send to the server.

[0985] Server: Search for "user123" in the database and match the password. If successful, retrieve the user's profile (e.g., name, email address).

[0986] Server: The acquired profile data is encrypted using AES256 and sent to the device.

[0987] Terminal: Decrypts the encrypted data and displays "Hello, user123" on the user's screen.

[0988] Example prompt sentence:

[0989] "Please enter your username and password to log in."

[0990] 2. Customize your learning curriculum

[0991] Examples:

[0992] User: Choose Machine Learning and Data Science.

[0993] Terminal: Encrypts the selected information and sends it to the server.

[0994] Server: Generates a curriculum for "Machine Learning and Data Science" using GPT-3.

[0995] Server: Encrypts the curriculum and sends it to the device.

[0996] Terminal: Decodes the curriculum and displays it to the user (e.g., "Chapter 1: Data Preprocessing, Chapter 2: Basic Algorithms").

[0997] Example prompt sentence:

[0998] "Choose the field you want to study. Example: Machine Learning, Data Science."

[0999] 3. Emotion recognition and curriculum adjustment using an emotion engine

[1000] Examples:

[1001] User: Shows confused expression while studying.

[1002] Device: Capture confused expressions through camera and microphone.

[1003] Terminal: Sends data to an emotion analysis system to recognize "confusion."

[1004] Server: Uses generative AI models to generate additional hints and explanations in response to puzzles and adjust the learning curriculum.

[1005] Example prompt sentence:

[1006] "Confused expression detected, show hint."

[1007] 4. Setting up and participating in virtual projects

[1008] Examples:

[1009] User: Choose a data science project.

[1010] Terminal: This selection information is encrypted and sent to the server.

[1011] Server: Sets up the metaverse virtual environment and loads project content (e.g., analyzing data, building models).

[1012] Server: Encrypts the configuration information and sends it to the device.

[1013] Terminal: Connects the user to the virtual environment.

[1014] Example prompt sentence:

[1015] "Choose a hypothetical project. Example: Data Analysis Project."

[1016] 5. Personalized instruction and real-time feedback

[1017] Examples:

[1018] User: Work on data analysis within the virtual environment.

[1019] Terminal: Sends work progress information (e.g., data preprocessing completed) to the server in real time.

[1020] Server: Parses progress information and generates feedback using ChatGPT (e.g., "Try a specific algorithm at this step").

[1021] Server: Encrypts the feedback information and sends it to the device.

[1022] Terminal: Display feedback information on the user's screen.

[1023] Example prompt sentence:

[1024] "We are sending you project status updates, please wait."

[1025] Through these steps, the system provides users with a customized, efficient, and meaningful learning experience. By combining generative AI models, sentiment analysis systems, and virtual reality technology, it is possible to monitor users' progress in real time and provide optimal feedback.

[1026] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1027] The flow of this system's program processing

[1028] Step 1: Enter user login information

[1029] 1. Input:

[1030] User: Enter your username and password on the login screen.

[1031] 2. Processing:

[1032] Terminal: The entered username and password are encrypted using the AES256 algorithm.

[1033] 3. Output:

[1034] Encrypted login information is generated.

[1035] Step 2: Submit and authenticate login information

[1036] 1. Input:

[1037] Encrypted login information.

[1038] 2. Processing:

[1039] On your device: Sends encrypted login information to the server.

[1040] Server: receives the encrypted login information and authenticates it against the user data in the database (using the SHA-256 algorithm).

[1041] 3. Output:

[1042] The authentication result (success or failure).

[1043] On success, the user's profile data.

[1044] Step 3: Submit and view your profile data

[1045] 1. Input:

[1046] Authentication result and profile data (if successful).

[1047] 2. Processing:

[1048] Server: If authentication is successful, the profile data is encrypted with AES256 and sent to the device.

[1049] Terminal: Decrypts the received profile data and displays it on the user's screen.

[1050] 3. Output:

[1051] The user's profile will be displayed on the screen.

[1052] Step 4: Choose a subject or topic you want to study

[1053] 1. Input:

[1054] Users: Choose the field or topic they want to learn about (e.g., machine learning, data science).

[1055] 2. Processing:

[1056] Terminal: Encrypts the selected information and sends it to the server.

[1057] 3. Output:

[1058] Encrypted selection information.

[1059] Step 5: Generate and submit a custom learning curriculum

[1060] 1. Input:

[1061] Encrypted selection information.

[1062] 2. Processing:

[1063] Server: Decodes the selection information and generates a custom learning curriculum using a generative AI model (e.g., GPT-3).

[1064] Server: Encrypts the generated curriculum with AES256 and sends it to the device.

[1065] 3. Output:

[1066] Encrypted custom learning curriculum.

[1067] Step 6: Decipher and display the curriculum

[1068] 1. Input:

[1069] Encrypted custom learning curriculum.

[1070] 2. Processing:

[1071] Terminal: Decodes the received curriculum and displays it to the user.

[1072] 3. Output:

[1073] A customized learning curriculum is displayed to the user.

[1074] Step 7: Acquire and analyze emotion data

[1075] 1. Input:

[1076] User: Facial expressions and voice changes while learning.

[1077] 2. Processing:

[1078] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[1079] Terminal: The acquired data is sent to the emotion analysis system for analysis.

[1080] 3. Output:

[1081] Sentiment analysis results (e.g., happy, confused, angry).

[1082] Step 8: Adjust the curriculum

[1083] 1. Input:

[1084] Emotion analysis results.

[1085] 2. Processing:

[1086] Server: Based on the analyzed emotional data, the learning curriculum is adjusted as needed.

[1087] 3. Output:

[1088] Coordinated learning curriculum.

[1089] Step 9: Select and configure your virtual project

[1090] 1. Input:

[1091] User: Select a virtual project.

[1092] 2. Processing:

[1093] On your device: Encrypt the project selection information and send it to the server.

[1094] Server: Sets up the metaverse virtual environment based on the selected project and loads the project.

[1095] Server: Encrypts the configuration information and sends it to the device.

[1096] 3. Output:

[1097] Virtual environment configuration data.

[1098] Step 10: Connect to your virtual environment

[1099] 1. Input:

[1100] Virtual environment configuration data.

[1101] 2. Processing:

[1102] Terminal: Connects the user to the virtual environment based on the received configuration data.

[1103] 3. Output:

[1104] The user is connected to the virtual environment.

[1105] Step 11: Submit project work progress and generate feedback

[1106] 1. Input:

[1107] User: Working on a project in a virtual environment.

[1108] 2. Processing:

[1109] Terminal: Sends work progress information to the server in real time.

[1110] Server: Analyzes progress information and generates optimal feedback using generative AI models.

[1111] Server: Encrypts the feedback and sends it to the device.

[1112] 3. Output:

[1113] Real-time feedback.

[1114] Step 12: Viewing feedback

[1115] 1. Input:

[1116] Encrypted real-time feedback.

[1117] 2. Processing:

[1118] Terminal: Decodes the feedback and displays it to the user.

[1119] 3. Output:

[1120] Appropriate feedback is provided to the user.

[1121] The above are the specific processing steps of this system's program, and we have explained in detail how the data is input at each step, the data processing and calculations performed, and how the output is obtained. By combining these steps, it is possible to provide the user with an optimal learning experience.

[1122] (Application example 2)

[1123] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1124] Conventional learning systems have difficulty adjusting in real time to the user's learning progress and emotions, and tend to provide a one-way curriculum. As a result, they are unable to flexibly respond to the user's level of understanding and motivation. Furthermore, even in methods that provide practical learning experiences using virtual environments, the lack of feedback based on the user's emotions and progress can reduce learning effectiveness. Furthermore, there is room for improvement in the operability and accuracy of real-time feedback when using devices such as smartphones and head-mounted displays.

[1125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1126] In this invention, the server includes a means for analyzing the user's facial expressions and voice using an emotion engine and adjusting the curriculum based on real-time feedback, a means for the smartphone or head-mounted display to acquire the user's emotion data and send it to the server, and a means for the server to monitor the user's progress based on the emotion data and provide real-time feedback using generative AI, thereby making it possible to provide a customized learning experience according to the user's individual progress and emotions.

[1127] A "terminal" is a device such as a computer, smartphone, tablet, or head-mounted display that a user uses as an interface.

[1128] The "server" is a central processing unit that manages user login information, learning data, and emotional data, and generates and transmits profile data, learning curriculum, feedback, etc.

[1129] An "emotion engine" is software or a system that analyzes a user's voice and facial expression data to identify their emotional state.

[1130] "Generative AI" refers to artificial intelligence that uses machine learning and natural language processing to generate and analyze data.

[1131] "Login information" refers to identification information, such as a username and password, required for a user to access a system.

[1132] "Profile data" refers to data including the user's personal information, learning history, setting information, and the like.

[1133] A "learning curriculum" is a customized learning content and schedule based on a user's chosen field or topic.

[1134] A "Virtual Project" is a virtual learning or work assignment or project that a user can experience within the Metaverse.

[1135] A "virtual environment" is a three-dimensional virtual space constructed using metaverse technology in which users can have virtual experiences.

[1136] "Real-time feedback" is a function that responds immediately to the user's learning progress and emotional state, providing appropriate advice and curriculum adjustments.

[1137] A "smartphone" is a type of mobile device, an advanced portable terminal that can be used to install and use communication functions and a variety of applications.

[1138] A "head-mounted display" is a display device worn on a user's head and is used to provide visual information.

[1139] The system of the present invention combines generative AI, an emotion engine, and metaverse technology to provide users with a customized learning experience. This system is composed of three main components: a server, a terminal, and a user, and is realized through the interaction of these components. A specific embodiment of this system is described below.

[1140] First, the user accesses the login screen using a device (smartphone, tablet, head-mounted display, etc.) and enters their username and password. The device encrypts this login information and sends it to the server. The server compares the received login information with user data in a database and performs authentication. If authentication is successful, the server obtains the profile data, encrypts it, and sends it back to the device. The device decrypts the received profile data and displays it on the user's screen.

[1141] Next, the user selects the field or topic they want to study on their device. The device encrypts the selection and sends it to the server. The server uses generative AI to generate a custom learning curriculum based on the user's selection. The generated learning curriculum is sent from the server to the device, where it is decrypted and displayed to the user.

[1142] As the user progresses with their learning, the device uses cameras and sensors to capture changes in the user's emotions (happiness, confusion, anger, etc.) in real time. The captured data is sent to an emotion engine, which analyzes the user's emotional state. Based on the analyzed data, the server adjusts the necessary parts of the learning curriculum and provides feedback to the user.

[1143] When a user selects a virtual project, the information is sent from the device to the server, which then uses metaverse technology to configure the virtual environment and load the selected project content. The server then sends the environment configuration information to the device, which then connects the user to the virtual environment.

[1144] To achieve these functions, the following hardware and software are used:

[1145] Hardware: smartphones, tablets, head-mounted displays, cameras, sensors

[1146] Software: Python, OpenCV (image processing library), transformers (natural language processing library), Requests (library for API requests)

[1147] For example, if a user selects that they want to learn data science, the device sends the selection information to the server. The server uses generative AI to generate a custom curriculum, which the device then displays to the user. Furthermore, if the user shows a confused expression during the project, the device captures facial expression data through the camera, and the emotion engine detects this as "confusion." Based on this information, the server can provide additional explanations and hints.

[1148] Example prompts to input to a generative AI model:

[1149] A user is confused about: [machine learning algorithm parameter tuning]

[1150] Example of feedback generated based on this prompt:

[1151] "In this step, it's important to understand the role of each parameter. Parameter A adjusts the scale of the data, and parameter B controls the learning rate. For example..."

[1152] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1153] Step 1:

[1154] The user accesses the login screen using a terminal and enters their username and password.

[1155] Input: Username, Password

[1156] Output: Encrypted login information

[1157] Specific behavior:

[1158] The user enters their username and password into the login screen of their device, which then encrypts the information using the AES encryption algorithm and sends it to the server.

[1159] Step 2:

[1160] The server receives the encrypted login information, decrypts it, and authenticates it by checking it against user data in a database.

[1161] Input: Encrypted login information

[1162] Output: Authentication results, profile data

[1163] Specific behavior:

[1164] The server decrypts the received login information using the AES decryption algorithm and compares it with the user data in its database. If authentication is successful, it retrieves the profile data, encrypts it, and sends it back to the device.

[1165] Step 3:

[1166] The terminal decrypts the profile data received from the server and displays it on the user screen.

[1167] Input: Encrypted profile data

[1168] Output: Display profile information

[1169] Specific behavior:

[1170] The device will then decrypt the received encrypted profile data using the AES decryption algorithm and display it on the user's screen, allowing the user to view their profile information.

[1171] Step 4:

[1172] The user selects the field or topic they wish to study on their device and sends the selected information to the server.

[1173] Input: Select the field or topic you want to study

[1174] Output: Encrypted selection information

[1175] Specific behavior:

[1176] Users select the fields and topics they want to learn on their device, which then encrypts this selection using the AES encryption algorithm and sends it to the server.

[1177] Step 5:

[1178] The server receives the encrypted selection information, decrypts it, and uses generative AI to generate a custom learning curriculum.

[1179] Input: Encrypted selection information

[1180] Output: Custom learning curriculum

[1181] Specific behavior:

[1182] The server decrypts the received selection information using the AES decryption algorithm and inputs it into the generative AI model, which then generates a custom learning curriculum based on the user's selection, encrypts it, and sends it to the device.

[1183] Step 6:

[1184] The terminal receives the encrypted learning curriculum, decrypts it, and displays it to the user.

[1185] Input: Encrypted learning curriculum

[1186] Output: Display of learning curriculum

[1187] Specific behavior:

[1188] The terminal decrypts the received encrypted curriculum using the AES decryption algorithm and displays it on the user's screen, allowing the user to view the customized curriculum.

[1189] Step 7:

[1190] As the user progresses with their learning, the device uses cameras and sensors to capture facial and voice data in real time and sends it to the emotion engine.

[1191] Input: User's facial expression data, voice data

[1192] Output: Emotion analysis results

[1193] Specific behavior:

[1194] The device uses cameras and sensors to capture the user's facial expressions and voice in real time and transmits the data to the emotion engine, which analyzes this data to detect the user's emotional state.

[1195] Step 8:

[1196] The server adjusts the necessary parts of the learning curriculum based on the emotional data received from the emotion engine, generates real-time feedback, and sends it to the terminal.

[1197] Input: Sentiment analysis results, learning progress data

[1198] Output: Curriculum adjustments, real-time feedback

[1199] Specific behavior:

[1200] The server generates customized feedback based on the emotion analysis results and learning progress data. The feedback content is created using a generative AI model, encrypted, and sent to the device.

[1201] Step 9:

[1202] The terminal receives the encrypted feedback, decrypts it and displays it on the user's screen.

[1203] Input: Encrypted feedback

[1204] Output: Display of feedback content

[1205] Specific behavior:

[1206] The device uses the AES decryption algorithm to decrypt the received encrypted feedback and displays it on the user's screen, where the user can see specific advice and tips from the generated AI model.

[1207] Step 10:

[1208] The user selects a virtual project and the terminal transmits the information to the server.

[1209] Input: Virtual project selection information

[1210] Output: Encrypted selection information

[1211] Specific behavior:

[1212] The user selects a virtual project on the terminal, which encrypts the selection information using the AES encryption algorithm and sends it to the server.

[1213] Step 11:

[1214] The server receives the encrypted selection information and uses metaverse technology to set up a virtual environment and load the project content.

[1215] Input: Encrypted selection information

[1216] Output: Virtual environment configuration information

[1217] Specific behavior:

[1218] The server decrypts the received selection information using the AES decryption algorithm, sets up a virtual environment using metaverse technology, loads the selected project content, and sends the configuration information to the terminal.

[1219] Step 12:

[1220] The terminal receives the setting information of the virtual environment and connects the user to the virtual environment.

[1221] Input: Virtual environment configuration information

[1222] Output: Virtual environment connected

[1223] Specific behavior:

[1224] The device decodes the received virtual environment configuration information and connects the user to the virtual environment, allowing the user to participate in virtual projects and progress in their learning.

[1225] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1226] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1227] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1228] [Third embodiment]

[1229] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1230] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1231] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1232] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1233] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1234] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1235] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1236] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1237] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1238] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1239] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1240] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1241] This invention is a new learning system that combines generative AI and the Metaverse, aiming to provide users with customized learning experiences and practical project experiences.

[1242] System configuration and program processing

[1243] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[1244] 1. User login and profile settings

[1245] User: Uses a terminal to access the system login screen and enters a username and password.

[1246] On the device: The login information entered by the user is encrypted and sent to the server.

[1247] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[1248] Server: Encrypts the profile data and sends it back to the device.

[1249] Terminal: Decodes the received profile data and displays it on the screen.

[1250] Examples:

[1251] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[1252] 2. Customize your learning curriculum

[1253] Users: Choose the subject or topic they want to learn about.

[1254] Terminal: Encrypts the selected information and sends it to the server.

[1255] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[1256] Server: Encrypts the generated learning curriculum and sends it to the device.

[1257] Terminal: Decodes the received curriculum and displays it to the user.

[1258] Examples:

[1259] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[1260] 3. Setting up and participating in virtual projects

[1261] Users: Select a virtual project within the curriculum.

[1262] On your device: Encrypt the project selection information and send it to the server.

[1263] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[1264] Server: Encrypts the virtual environment configuration information and sends it to the device.

[1265] Terminal: Connects the user to the virtual environment based on the received configuration information.

[1266] Examples:

[1267] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[1268] 4. Personalized instruction and real-time feedback

[1269] Users: Work on projects in a virtual environment.

[1270] Device: Sends completed tasks and progress information to the server in real time.

[1271] Server: Analyzes progress information and uses generative AI to generate feedback appropriate for the user.

[1272] Server: Encrypts the generated feedback and sends it to the device.

[1273] Terminal: Display feedback on the user's screen.

[1274] Examples:

[1275] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[1276] As described above, this system leverages generative AI and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to grow.

[1277] The processing flow will be explained below.

[1278] Step 1:

[1279] The user uses a terminal to access the system's login screen and enters their username and password.

[1280] Step 2:

[1281] The terminal encrypts the entered login information and sends it to the server.

[1282] Step 3:

[1283] The server compares the received login information with the user data in the database and performs authentication.

[1284] Step 4:

[1285] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[1286] Step 5:

[1287] The device decodes the received profile data and displays it on the screen.

[1288] Step 6:

[1289] Users select the field or topic they want to learn on their device.

[1290] Step 7:

[1291] The terminal encrypts the selected information and sends it to the server.

[1292] Step 8:

[1293] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[1294] Step 9:

[1295] The learning curriculum generated by the server is encrypted and sent to the terminal.

[1296] Step 10:

[1297] The terminal decodes the received curriculum and displays it to the user.

[1298] Step 11:

[1299] A user selects a virtual project within a curriculum.

[1300] Step 12:

[1301] The terminal encrypts the project selection information and transmits it to the server.

[1302] Step 13:

[1303] The server sets up the metaverse virtual environment and loads the selected project contents.

[1304] Step 14:

[1305] The server encrypts the virtual environment configuration information and sends it to the terminal.

[1306] Step 15:

[1307] The terminal connects the user to the virtual environment based on the configuration information received.

[1308] Step 16:

[1309] Users work on projects in a virtual environment.

[1310] Step 17:

[1311] The device sends progress information to the server in real time.

[1312] Step 18:

[1313] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[1314] Step 19:

[1315] The server generates feedback that is encrypted and sent to the terminal.

[1316] Step 20:

[1317] The terminal displays the received feedback to the user and provides specific advice.

[1318] Example 1

[1319] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1320] Conventional learning systems often offer only a standard curriculum, making it difficult to customize to meet individual user needs. They also struggle to provide real-time feedback and practical project experiences. Providing a learning environment utilizing a virtual reality environment, in particular, poses numerous technical challenges and is difficult to implement.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1322] In this invention, the server includes means for authenticating login information and obtaining user profile data, means for generating a learning curriculum using a generative AI model based on the selection information, means for configuring a metaverse environment and loading the selected projects, and means for monitoring completed tasks and progress information in real time and providing real-time feedback using the generative AI model, thereby providing a personalized learning experience and hands-on project experience with real-time feedback.

[1323] "User" refers to the person who operates the system and experiences the learning curriculum and virtual projects.

[1324] "Device" refers to a computer or mobile device on which a user performs operations such as entering login information, selecting a course of study, or participating in a virtual project.

[1325] "Server" refers to a central processing unit that authenticates user login information, obtains profile data, generates learning curricula using generative AI models, sets up the metaverse environment, and generates real-time feedback.

[1326] "Generative AI model" refers to an artificial intelligence algorithm used to generate personalized curriculum based on user selections and provide real-time feedback.

[1327] "Login Information" refers to authentication information, such as a username and password, entered by a User to access a System.

[1328] "Profile Data" refers to data that includes personal information such as a user's name, learning history, and settings.

[1329] A "learning curriculum" refers to a series of learning content and assignments generated based on the field or topic a user wants to learn.

[1330] A "virtual project" refers to a specific challenge or task that a user participates in and completes within a virtual reality environment.

[1331] A "metaverse environment" refers to a virtual work or learning environment built using virtual reality technology.

[1332] "Real-time feedback" refers to instantaneous evaluation and advice provided to users based on their progress.

[1333] This invention is a new learning system that combines generative AI models and the Metaverse. The purpose of this system is to provide users with customized learning experiences and practical project experiences. Below, we will explain how to implement this system in detail.

[1334] This system consists of three main components: a server, a terminal, and a user.

[1335] Hardware and software used

[1336] Server: Serves as a central processing unit, authenticating users, managing profile data, generating learning curricula using generative AI models (e.g., GPT-4), configuring the metaverse environment, and generating real-time feedback. Implemented in a server system that also includes databases and AI algorithms.

[1337] Terminal: A device operated by a user, such as a computer, smartphone, or tablet. The software on the terminal provides the user interface, performs encryption / decryption processing, communicates with the server, and displays information to the user.

[1338] User: An individual who uses the learning system. The user operates the device and progresses with the learning.

[1339] Program processing

[1340] User login and profile settings

[1341] A user uses a terminal to access the system's login screen and enters their username and password. The terminal encrypts this login information (e.g., AES encryption) and sends it to the server. The server decrypts the received login information and authenticates it by comparing it with information in a database. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it to the terminal. The terminal decrypts the data and displays it to the user.

[1342] Customized learning curriculum

[1343] The user selects the field or topic they wish to study. The device encrypts the selection information (e.g., RSA encryption) and sends it to the server. The server analyzes the received selection information using a generative AI model (e.g., GPT-4) and generates a learning curriculum optimal for the user. The generated curriculum is encrypted and sent to the device. The device decrypts the curriculum and displays it to the user.

[1344] Examples:

[1345] The user selects that they want to learn machine learning and data science. The device RSA-encrypts this information and sends it to the server. The server uses GPT-4 to generate a customized curriculum and sends it to the device. The device then displays the customized curriculum on the user's screen.

[1346] Setting up and participating in virtual projects

[1347] The user selects a virtual project within the curriculum. The device encrypts the selection information and sends it to the server. The server analyzes the received information and sets up the virtual project within the metaverse environment. The set virtual environment information is then encrypted and sent to the device. The device decrypts it and connects the user to the virtual environment.

[1348] Examples:

[1349] The user selects a virtual data science project. The device encrypts this selection information and sends it to the server. The server configures the metaverse environment and loads the project content. The server encrypts the environment configuration information and sends it to the device, which then connects the user to the virtual environment.

[1350] Personalized instruction and real-time feedback

[1351] Users work on projects in a virtual environment. The device sends completed tasks and progress information to the server in real time. The server analyzes the progress information and generates appropriate feedback for the user using a generative AI model (e.g., Chat GPT). The generated feedback is encrypted and sent to the device, which then displays the feedback on the user's screen.

[1352] Examples:

[1353] The user performs data analysis in a virtual environment. The device sends progress information to the server in real time. The server uses this information to generate feedback using a generative AI model and sends it to the device. The device then displays the feedback on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[1354] Prompt Sentence Examples

[1355] "Users want to learn machine learning from the basics to advanced applications. Please generate a customized learning curriculum."

[1356] As described above, the system leverages generative AI models and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to help them grow.

[1357] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1358] Step 1:

[1359] A user accesses the login screen of the system using a terminal and enters a user name and password, which become the input information.

[1360] The terminal encrypts the entered login information using the AES encryption algorithm and sends it to the server, where it outputs the encrypted login information.

[1361] Step 2:

[1362] The server decrypts the received encrypted login information using the AES decryption algorithm.

[1363] The server compares the decrypted login information with the user data in the database and performs authentication. The user data from the database is used as input information. If authentication is successful, the user's profile data is retrieved.

[1364] The profile data is encrypted using the AES encryption algorithm and sent to the terminal, and this encrypted profile data is the output.

[1365] Step 3:

[1366] The terminal decrypts the received encrypted profile data using the AES decryption algorithm.

[1367] The decrypted profile data is displayed on the user's screen, which is the output.

[1368] Step 4:

[1369] The user selects the field or topic they want to learn about, and this selection information becomes the input information.

[1370] The terminal encrypts the user's selection information using the RSA encryption algorithm and sends it to the server, where the encrypted selection information is output.

[1371] Step 5:

[1372] The server decrypts the received encrypted selection information using the RSA decryption algorithm.

[1373] The server generates a custom learning curriculum using a generative AI model (e.g., GPT-4) based on the decoded selection information. This selection information serves as input information.

[1374] The generated learning curriculum is encrypted using the AES encryption algorithm and sent to the terminal. This encrypted learning curriculum is the output.

[1375] Step 6:

[1376] The terminal decrypts the received encrypted learning curriculum using the AES decryption algorithm.

[1377] The decoded learning curriculum is displayed on the user's screen, which is the output.

[1378] Step 7:

[1379] The user selects a virtual project within the curriculum, and this selection information becomes the input information.

[1380] The device hashes the project selection information using the SHA-256 hashing algorithm and sends it to the server, where the hashed selection information is the output.

[1381] Step 8:

[1382] The server parses the received hashed project selection information.

[1383] The server configures the metaverse environment and loads the selected project contents. This configuration information becomes the input information.

[1384] The configured virtual environment information is hashed using SHA-256 and sent to the terminal. This hashed configuration information is output.

[1385] Step 9:

[1386] The terminal connects the user to the metaverse virtual project based on the received hashed virtual environment information.

[1387] The output is that the user is connected to the virtual environment.

[1388] Step 10:

[1389] Users work on projects within a virtual environment, and their actions within this environment provide input information.

[1390] The terminal transmits completed tasks and progress information to the server in real time using the WebSocket protocol, and this progress information is the output.

[1391] Step 11:

[1392] The server analyzes the received progress information and generates optimal feedback for the user using a generative AI model (e.g., Chat GPT). The progress information is used as input.

[1393] The generated feedback is encrypted via WebSocket and sent to the terminal. This encrypted feedback becomes the output.

[1394] Step 12:

[1395] The terminal decrypts the received encrypted feedback.

[1396] The decoded feedback is displayed on the user's screen, which is the output.

[1397] In this way, by performing specific operations and processing and calculating data at each step, it is possible to provide the user with a customized learning experience.

[1398] (Application example 1)

[1399] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1400] Conventional learning systems have struggled to provide customized learning tailored to individual users' needs or practical project experiences in virtual environments. They also struggled to provide optimal feedback on users' learning progress in real time. Especially in distance learning and online education, there is a lack of individualized instruction, which can lead to insufficient learning outcomes.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1402] In this invention, the server includes a means for generating customized advice based on the user's learning progress and displaying it to the user on the terminal, a means for providing real-time feedback using generative AI, and a means for providing a project experience in a virtual reality environment using the metaverse, allowing users to enjoy customized learning and practical project experience tailored to their individual needs and receive optimal feedback in real time.

[1403] "User" refers to someone who uses the system to study or work on a project.

[1404] "Terminal" refers to a device operated by a user, used to input login information and selection information, display profile data and learning curriculum, connect to a virtual environment, and so on.

[1405] "Server" refers to a device that receives and processes information sent from a terminal and has functions such as user authentication, acquisition of profile data, generation of learning curriculum, setting up a virtual environment, and generation of real-time feedback.

[1406] "Login Information" refers to the authentication information required for a user to access a system, and typically consists of a username and password.

[1407] "Profile data" refers to data that includes information such as the user's personal information, learning history, and skill level, and is acquired by the server and returned to the terminal when authentication is successful.

[1408] "Learning curriculum" refers to the learning content or study plan for a specific field or topic that is generated by the server based on user selection information.

[1409] A "virtual project" refers to a specific assignment or task selected by a user and executed in a virtual environment within the metaverse.

[1410] The "metaverse" is a virtual environment constructed using virtual reality technology, and refers to a virtual space in which users can interact with each other.

[1411] "Generative AI" refers to artificial intelligence technology that generates appropriate advice and feedback based on a user's learning progress and project content.

[1412] "Feedback" refers to the evaluation and advice provided in real time by generative AI for tasks or projects that users are currently working on.

[1413] "Customized advice" refers to personalized advice that suggests specific content and methods based on the user's learning progress and skill level.

[1414] Three main components are required to implement this invention: a server, a terminal, and a user. The system provides users with a customized learning experience and project experience in a virtual environment.

[1415] First, a user uses a terminal to access the system's login screen. Here, they enter their username and password. The terminal encrypts the entered login information and sends it to the server. The server compares the received login information with the user data in its database and performs authentication. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it back to the terminal. The terminal decrypts the profile data and displays it to the user.

[1416] Next, the learning curriculum is customized. The user selects the fields and topics they want to study. The device encrypts these selections and sends them to the server. The server uses a generative AI model to generate a custom learning curriculum based on the user's selections. The generated curriculum is then encrypted again and sent to the device. The device decrypts the curriculum data and displays it to the user.

[1417] The user then selects a virtual project within the curriculum. The device encrypts the project selection information and sends it to the server. The server then configures the metaverse virtual environment and loads the selected project content. The virtual environment configuration information is then encrypted and sent to the device. The device then connects the user to the virtual environment based on the configuration information.

[1418] As a user works on a project in a virtual environment, the device sends completed tasks and progress information to the server in real time. The server analyzes this information and uses a generative AI model to generate appropriate feedback for the user. The generated feedback is encrypted and sent to the device, which then decrypts it and displays it to the user. This feedback can include specific advice, such as "Try a different regression algorithm for this step."

[1419] The hardware used includes smartphones and head-mounted displays (HMDs), and the software includes Unity (for building the virtual environment), AWS (for servers and databases), and OpenAI's GPT-4 generative AI.

[1420] Below are some examples of prompt sentences.

[1421] "User wants feedback on data analysis. Please provide specific advice."

[1422] The system allows users to receive customized learning tailored to their individual needs and improve their skills through hands-on project experience in a virtual environment, with real-time feedback providing a significant learning experience.

[1423] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1424] Step 1:

[1425] The user accesses the login screen of the learning system using a smartphone or head-mounted display (HMD). The user enters their username and password. This is used as input data to proceed to the next step.

[1426] Step 2:

[1427] The terminal encrypts the login information entered by the user. This encrypted data is sent to the server. The input is the login information, and the output is the encrypted login data.

[1428] Step 3:

[1429] The server decrypts the encrypted login information it receives and authenticates it by comparing it with the user data in the database. If authentication is successful, it obtains the user's profile data. The input is the encrypted login data, and the output is the user's profile data.

[1430] Step 4:

[1431] The server re-encrypts the acquired profile data and sends it to the terminal. The input is the profile data, and the output is the encrypted profile data.

[1432] Step 5:

[1433] The device decrypts the received profile data and displays it to the user, allowing the user to check their own profile information. The input is encrypted profile data, and the output is decrypted profile data.

[1434] Step 6:

[1435] The user selects the field or topic they want to study, and with this as input, they move on to the next step.

[1436] Step 7:

[1437] The terminal encrypts information about the learning field or topic selected by the user and sends it to the server. The input is the selected information, and the output is the encrypted selected information.

[1438] Step 8:

[1439] The server decodes the received selection information and generates a custom learning curriculum using a generative AI model. The input is the selection information and the output is the custom learning curriculum.

[1440] Step 9:

[1441] The server encrypts the generated curriculum and sends it to the terminal. The input is the custom curriculum, and the output is the encrypted curriculum.

[1442] Step 10:

[1443] The terminal decrypts the received curriculum and displays it to the user. The user can then view the customized curriculum. The input is the encrypted curriculum, and the output is the decrypted curriculum.

[1444] Step 11:

[1445] The user selects a virtual project within the learning curriculum, which is used as input data to proceed to the next step.

[1446] Step 12:

[1447] The terminal encrypts the project selection information and sends it to the server. The input is the project selection information, and the output is the encrypted project selection information.

[1448] Step 13:

[1449] The server decodes the received project selection information, configures the metaverse virtual environment, and loads the selected project content. The input is the project selection information, and the output is the virtual environment configuration information.

[1450] Step 14:

[1451] The server encrypts the virtual environment setting information and sends it to the terminal. The input is the virtual environment setting information, and the output is the encrypted virtual environment setting information.

[1452] Step 15:

[1453] The terminal decrypts the received virtual environment setting information and connects the user to the virtual environment. The input is the encrypted virtual environment setting information, and the output is the connection to the virtual environment.

[1454] Step 16:

[1455] A user works on a project in a virtual environment. The device sends completed tasks and progress information to the server in real time. The input is tasks and progress information, and the output is encrypted task and progress information.

[1456] Step 17:

[1457] The server interprets the received progress information and uses a generative AI model to generate feedback appropriate for the user. The input is progress information, and the output is customized feedback.

[1458] Step 18:

[1459] The server generates and encrypts the feedback and sends it to the device. The input is the customized feedback, and the output is the encrypted feedback.

[1460] Step 19:

[1461] The device decrypts the received feedback and displays it to the user, allowing them to receive specific advice based on their progress. The input is encrypted feedback, and the output is decrypted feedback.

[1462] Through this process, users can receive personalized learning experiences and optimal feedback in real time. The entire system is powered by a combination of hardware such as smartphones and head-mounted displays, and software such as Unity, AWS, and generative AI models.

[1463] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1464] This invention is a new learning system that combines generative AI, the Metaverse, and an emotion engine that recognizes users' emotions. This system provides users with customized learning experiences and practical projects, and provides real-time feedback and curriculum adjustments based on the user's emotions.

[1465] System configuration and program processing

[1466] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[1467] 1. User login and profile settings

[1468] User: Uses a terminal to access the system login screen and enters a username and password.

[1469] On the device: The login information entered by the user is encrypted and sent to the server.

[1470] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[1471] Server: Encrypts the profile data and sends it back to the device.

[1472] Terminal: Decodes the received profile data and displays it on the screen.

[1473] Examples:

[1474] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[1475] 2. Customize your learning curriculum

[1476] Users: Select the subject or topic they want to learn on their device.

[1477] Terminal: Encrypts the selected information and sends it to the server.

[1478] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[1479] Server: Encrypts the generated learning curriculum and sends it to the device.

[1480] Terminal: Decodes the received curriculum and displays it to the user.

[1481] Examples:

[1482] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[1483] 3. Emotion recognition and curriculum adjustment using an emotion engine

[1484] Users: Experience changes in emotions (e.g., joy, confusion, anger, etc.) as they progress through the learning process.

[1485] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[1486] Terminal: The acquired data is sent to the emotion engine and the user's emotions are analyzed.

[1487] Server: Based on the analyzed emotional data, the learning curriculum and project content are adjusted as needed.

[1488] Examples:

[1489] If a user shows a confused expression during the project, the device captures the user's facial expression through the camera, and the emotion engine analyzes it to detect "confusion." The server uses this information to adjust the curriculum, for example, by providing additional explanations or hints.

[1490] 4. Setting up and participating in virtual projects

[1491] Users: Select a virtual project within the curriculum.

[1492] On your device: Encrypt the project selection information and send it to the server.

[1493] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[1494] Server: Encrypts the virtual environment configuration information and sends it to the device.

[1495] Terminal: Connects the user to the virtual environment based on the received configuration information.

[1496] Examples:

[1497] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[1498] 5. Personalized instruction and real-time feedback

[1499] Users: Work on projects in a virtual environment.

[1500] Device: Sends progress information to the server in real time.

[1501] Server: Analyzes progress information and uses generative AI to generate appropriate feedback for the user.

[1502] Server: Encrypts the generated feedback and sends it to the device.

[1503] Terminal: Display feedback on the user's screen.

[1504] Examples:

[1505] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[1506] As described above, this system utilizes generative AI, metaverse technology, and an emotion engine to provide users with customized learning experiences and hands-on project experiences. Users can hone their skills in a realistic virtual environment and grow while receiving appropriate feedback in real time. Furthermore, the emotion engine enables instruction and curriculum to be adjusted based on the user's emotions, creating a more personalized learning experience.

[1507] The processing flow will be explained below.

[1508] Step 1:

[1509] The user uses a terminal to access the system's login screen and enters their username and password.

[1510] Step 2:

[1511] The terminal encrypts the entered login information and sends it to the server.

[1512] Step 3:

[1513] The server compares the received login information with the user data in the database and performs authentication.

[1514] Step 4:

[1515] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[1516] Step 5:

[1517] The device decodes the received profile data and displays it on the screen.

[1518] Step 6:

[1519] Users select the field or topic they want to learn through the device interface.

[1520] Step 7:

[1521] The terminal encrypts the selection information and transmits it to the server.

[1522] Step 8:

[1523] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[1524] Step 9:

[1525] The learning curriculum generated by the server is encrypted and sent to the terminal.

[1526] Step 10:

[1527] The terminal decodes the received curriculum and displays it to the user.

[1528] Step 11:

[1529] As the user progresses through the curriculum, the device uses a camera and microphone to capture facial expressions and voice in real time.

[1530] Step 12:

[1531] The device sends the acquired emotional data (facial expressions, voice, etc.) to the emotion engine, which analyzes the user's emotions.

[1532] Step 13:

[1533] The server adjusts the learning curriculum and project content as needed based on the analyzed emotional data.

[1534] Step 14:

[1535] A user selects a virtual project within a curriculum.

[1536] Step 15:

[1537] The terminal encrypts the project selection information and transmits it to the server.

[1538] Step 16:

[1539] The server sets up the metaverse virtual environment and loads the selected project contents.

[1540] Step 17:

[1541] The server encrypts the virtual environment configuration information and sends it to the terminal.

[1542] Step 18:

[1543] The terminal connects the user to the virtual environment based on the configuration information received.

[1544] Step 19:

[1545] Users work on projects in a virtual environment.

[1546] Step 20:

[1547] The device sends progress information to the server in real time.

[1548] Step 21:

[1549] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[1550] Step 22:

[1551] The server generates feedback that is encrypted and sent to the terminal.

[1552] Step 23:

[1553] The terminal displays the received feedback to the user and provides specific advice.

[1554] Example 2

[1555] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1556] Conventional learning systems have difficulty dynamically adjusting learning curriculum and feedback based on the user's individual emotions and progress, resulting in problems such as reduced learning efficiency and motivation. Furthermore, project experiences using virtual reality are limited, and a more realistic learning environment is needed.

[1557] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1558] In this invention, the server includes a means for analyzing a user's emotional data and adjusting the curriculum, a means for providing customized real-time feedback using a generative AI model, and a means for providing a project experience using virtual reality technology, thereby personalizing the user's learning experience and enabling appropriate feedback and curriculum adjustment according to emotions and progress.

[1559] "User" refers to an individual who uses this system to learn or experience a project.

[1560] "Terminal" refers to an information processing device used by a user, such as a computer, tablet, or smartphone.

[1561] "Server" refers to the central computer device that performs the main processing and data management of the System.

[1562] "Login Information" means authentication information, such as a username and password, required for a User to access a System.

[1563] "Profile data" refers to data about a user, such as personal information, history, and learning progress.

[1564] "Learning curriculum" refers to the learning plans and materials created based on the field or topic selected by the user.

[1565] An "emotion analysis system" refers to a system that analyzes acquired facial expressions and voice data of a user and recognizes their emotions.

[1566] "Virtual environment" refers to a simulated environment constructed using virtual reality technology.

[1567] A "generative AI model" refers to an artificial intelligence model that generates learning curriculum and feedback based on user input data and situations.

[1568] "Real-time feedback" refers to advice and information provided instantly to users based on their progress and actions.

[1569] The system of the present invention is composed of three main components: a user, a terminal, and a server. The roles and operations of each component are described below.

[1570] System Overview

[1571] This system comprehensively handles everything from login authentication to learning curriculum generation, real-time feedback, virtual project setup and participation, and even curriculum adjustment based on sentiment analysis. By communicating with each other, the server, terminals, and users provide the optimal learning experience for users.

[1572] Hardware and software used

[1573] Device: Computer, tablet, smartphone, etc.

[1574] Server: A high-performance computing device that works in conjunction with a database.

[1575] Generative AI models: Artificial intelligence models such as GPT-3 and ChatGPT.

[1576] Sentiment analysis system: Emotion detection API (e.g. Microsoft Azure Emotion API).

[1577] Virtual reality technology: Virtual environment engines that support the Metaverse (e.g., Unity, Unreal Engine).

[1578] Specific examples of program processing

[1579] 1. User login and profile settings

[1580] Examples:

[1581] User: Enter "user123" and "password" on the login screen.

[1582] Terminal: Encrypt "user123" and "password" with AES256 and send to the server.

[1583] Server: Search for "user123" in the database and match the password. If successful, retrieve the user's profile (e.g., name, email address).

[1584] Server: The acquired profile data is encrypted using AES256 and sent to the device.

[1585] Terminal: Decrypts the encrypted data and displays "Hello, user123" on the user's screen.

[1586] Example prompt sentence:

[1587] "Please enter your username and password to log in."

[1588] 2. Customize your learning curriculum

[1589] Examples:

[1590] User: Choose Machine Learning and Data Science.

[1591] Terminal: Encrypts the selected information and sends it to the server.

[1592] Server: Generates a curriculum for "Machine Learning and Data Science" using GPT-3.

[1593] Server: Encrypts the curriculum and sends it to the device.

[1594] Terminal: Decodes the curriculum and displays it to the user (e.g., "Chapter 1: Data Preprocessing, Chapter 2: Basic Algorithms").

[1595] Example prompt sentence:

[1596] "Choose the field you want to study. Example: Machine Learning, Data Science."

[1597] 3. Emotion recognition and curriculum adjustment using an emotion engine

[1598] Examples:

[1599] User: Shows confused expression while studying.

[1600] Device: Capture confused expressions through camera and microphone.

[1601] Terminal: Sends data to an emotion analysis system to recognize "confusion."

[1602] Server: Uses generative AI models to generate additional hints and explanations in response to puzzles and adjust the learning curriculum.

[1603] Example prompt sentence:

[1604] "Confused expression detected, show hint."

[1605] 4. Setting up and participating in virtual projects

[1606] Examples:

[1607] User: Choose a data science project.

[1608] Terminal: This selection information is encrypted and sent to the server.

[1609] Server: Sets up the metaverse virtual environment and loads project content (e.g., analyzing data, building models).

[1610] Server: Encrypts the configuration information and sends it to the device.

[1611] Terminal: Connects the user to the virtual environment.

[1612] Example prompt sentence:

[1613] "Choose a hypothetical project. Example: Data Analysis Project."

[1614] 5. Personalized instruction and real-time feedback

[1615] Examples:

[1616] User: Work on data analysis within the virtual environment.

[1617] Terminal: Sends work progress information (e.g., data preprocessing completed) to the server in real time.

[1618] Server: Parses progress information and generates feedback using ChatGPT (e.g., "Try a specific algorithm at this step").

[1619] Server: Encrypts the feedback information and sends it to the device.

[1620] Terminal: Display feedback information on the user's screen.

[1621] Example prompt sentence:

[1622] "We are sending you project status updates, please wait."

[1623] Through these steps, the system provides users with a customized, efficient, and meaningful learning experience. By combining generative AI models, sentiment analysis systems, and virtual reality technology, it is possible to monitor users' progress in real time and provide optimal feedback.

[1624] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1625] The flow of this system's program processing

[1626] Step 1: Enter user login information

[1627] 1. Input:

[1628] User: Enter your username and password on the login screen.

[1629] 2. Processing:

[1630] Terminal: The entered username and password are encrypted using the AES256 algorithm.

[1631] 3. Output:

[1632] Encrypted login information is generated.

[1633] Step 2: Submit and authenticate login information

[1634] 1. Input:

[1635] Encrypted login information.

[1636] 2. Processing:

[1637] On your device: Sends encrypted login information to the server.

[1638] Server: receives the encrypted login information and authenticates it against the user data in the database (using the SHA-256 algorithm).

[1639] 3. Output:

[1640] The authentication result (success or failure).

[1641] On success, the user's profile data.

[1642] Step 3: Submit and view your profile data

[1643] 1. Input:

[1644] Authentication result and profile data (if successful).

[1645] 2. Processing:

[1646] Server: If authentication is successful, the profile data is encrypted with AES256 and sent to the device.

[1647] Terminal: Decrypts the received profile data and displays it on the user's screen.

[1648] 3. Output:

[1649] The user's profile will be displayed on the screen.

[1650] Step 4: Choose a subject or topic you want to study

[1651] 1. Input:

[1652] Users: Choose the field or topic they want to learn about (e.g., machine learning, data science).

[1653] 2. Processing:

[1654] Terminal: Encrypts the selected information and sends it to the server.

[1655] 3. Output:

[1656] Encrypted selection information.

[1657] Step 5: Generate and submit a custom learning curriculum

[1658] 1. Input:

[1659] Encrypted selection information.

[1660] 2. Processing:

[1661] Server: Decodes the selection information and generates a custom learning curriculum using a generative AI model (e.g., GPT-3).

[1662] Server: Encrypts the generated curriculum with AES256 and sends it to the device.

[1663] 3. Output:

[1664] Encrypted custom learning curriculum.

[1665] Step 6: Decipher and display the curriculum

[1666] 1. Input:

[1667] Encrypted custom learning curriculum.

[1668] 2. Processing:

[1669] Terminal: Decodes the received curriculum and displays it to the user.

[1670] 3. Output:

[1671] A customized learning curriculum is displayed to the user.

[1672] Step 7: Acquire and analyze emotion data

[1673] 1. Input:

[1674] User: Facial expressions and voice changes while learning.

[1675] 2. Processing:

[1676] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[1677] Terminal: The acquired data is sent to the emotion analysis system for analysis.

[1678] 3. Output:

[1679] Sentiment analysis results (e.g., happy, confused, angry).

[1680] Step 8: Adjust the curriculum

[1681] 1. Input:

[1682] Emotion analysis results.

[1683] 2. Processing:

[1684] Server: Based on the analyzed emotional data, the learning curriculum is adjusted as needed.

[1685] 3. Output:

[1686] Coordinated learning curriculum.

[1687] Step 9: Select and configure your virtual project

[1688] 1. Input:

[1689] User: Select a virtual project.

[1690] 2. Processing:

[1691] On your device: Encrypt the project selection information and send it to the server.

[1692] Server: Sets up the metaverse virtual environment based on the selected project and loads the project.

[1693] Server: Encrypts the configuration information and sends it to the device.

[1694] 3. Output:

[1695] Virtual environment configuration data.

[1696] Step 10: Connect to your virtual environment

[1697] 1. Input:

[1698] Virtual environment configuration data.

[1699] 2. Processing:

[1700] Terminal: Connects the user to the virtual environment based on the received configuration data.

[1701] 3. Output:

[1702] The user is connected to the virtual environment.

[1703] Step 11: Submit project work progress and generate feedback

[1704] 1. Input:

[1705] User: Working on a project in a virtual environment.

[1706] 2. Processing:

[1707] Terminal: Sends work progress information to the server in real time.

[1708] Server: Analyzes progress information and generates optimal feedback using generative AI models.

[1709] Server: Encrypts the feedback and sends it to the device.

[1710] 3. Output:

[1711] Real-time feedback.

[1712] Step 12: Viewing feedback

[1713] 1. Input:

[1714] Encrypted real-time feedback.

[1715] 2. Processing:

[1716] Terminal: Decodes the feedback and displays it to the user.

[1717] 3. Output:

[1718] Appropriate feedback is provided to the user.

[1719] The above are the specific processing steps of this system's program, and we have explained in detail how the data is input at each step, the data processing and calculations performed, and how the output is obtained. By combining these steps, it is possible to provide the user with an optimal learning experience.

[1720] (Application example 2)

[1721] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1722] Conventional learning systems have difficulty adjusting in real time to the user's learning progress and emotions, and tend to provide a one-way curriculum. As a result, they are unable to flexibly respond to the user's level of understanding and motivation. Furthermore, even in methods that provide practical learning experiences using virtual environments, the lack of feedback based on the user's emotions and progress can reduce learning effectiveness. Furthermore, there is room for improvement in the operability and accuracy of real-time feedback when using devices such as smartphones and head-mounted displays.

[1723] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1724] In this invention, the server includes a means for analyzing the user's facial expressions and voice using an emotion engine and adjusting the curriculum based on real-time feedback, a means for the smartphone or head-mounted display to acquire the user's emotion data and send it to the server, and a means for the server to monitor the user's progress based on the emotion data and provide real-time feedback using generative AI, thereby making it possible to provide a customized learning experience according to the user's individual progress and emotions.

[1725] A "terminal" is a device such as a computer, smartphone, tablet, or head-mounted display that a user uses as an interface.

[1726] The "server" is a central processing unit that manages user login information, learning data, and emotional data, and generates and transmits profile data, learning curriculum, feedback, etc.

[1727] An "emotion engine" is software or a system that analyzes a user's voice and facial expression data to identify their emotional state.

[1728] "Generative AI" refers to artificial intelligence that uses machine learning and natural language processing to generate and analyze data.

[1729] "Login information" refers to identification information, such as a username and password, required for a user to access a system.

[1730] "Profile data" refers to data including the user's personal information, learning history, setting information, and the like.

[1731] A "learning curriculum" is a customized learning content and schedule based on a user's chosen field or topic.

[1732] A "Virtual Project" is a virtual learning or work assignment or project that a user can experience within the Metaverse.

[1733] A "virtual environment" is a three-dimensional virtual space constructed using metaverse technology in which users can have virtual experiences.

[1734] "Real-time feedback" is a function that responds immediately to the user's learning progress and emotional state, providing appropriate advice and curriculum adjustments.

[1735] A "smartphone" is a type of mobile device, an advanced portable terminal that can be used to install and use communication functions and a variety of applications.

[1736] A "head-mounted display" is a display device worn on a user's head and is used to provide visual information.

[1737] The system of the present invention combines generative AI, an emotion engine, and metaverse technology to provide users with a customized learning experience. This system is composed of three main components: a server, a terminal, and a user, and is realized through the interaction of these components. A specific embodiment of this system is described below.

[1738] First, the user accesses the login screen using a device (smartphone, tablet, head-mounted display, etc.) and enters their username and password. The device encrypts this login information and sends it to the server. The server compares the received login information with user data in a database and performs authentication. If authentication is successful, the server obtains the profile data, encrypts it, and sends it back to the device. The device decrypts the received profile data and displays it on the user's screen.

[1739] Next, the user selects the field or topic they want to study on their device. The device encrypts the selection and sends it to the server. The server uses generative AI to generate a custom learning curriculum based on the user's selection. The generated learning curriculum is sent from the server to the device, where it is decrypted and displayed to the user.

[1740] As the user progresses with their learning, the device uses cameras and sensors to capture changes in the user's emotions (happiness, confusion, anger, etc.) in real time. The captured data is sent to an emotion engine, which analyzes the user's emotional state. Based on the analyzed data, the server adjusts the necessary parts of the learning curriculum and provides feedback to the user.

[1741] When a user selects a virtual project, the information is sent from the device to the server, which then uses metaverse technology to configure the virtual environment and load the selected project content. The server then sends the environment configuration information to the device, which then connects the user to the virtual environment.

[1742] To achieve these functions, the following hardware and software are used:

[1743] Hardware: smartphones, tablets, head-mounted displays, cameras, sensors

[1744] Software: Python, OpenCV (image processing library), transformers (natural language processing library), Requests (library for API requests)

[1745] For example, if a user selects that they want to learn data science, the device sends the selection information to the server. The server uses generative AI to generate a custom curriculum, which the device then displays to the user. Furthermore, if the user shows a confused expression during the project, the device captures facial expression data through the camera, and the emotion engine detects this as "confusion." Based on this information, the server can provide additional explanations and hints.

[1746] Example prompts to input to a generative AI model:

[1747] A user is confused about: [machine learning algorithm parameter tuning]

[1748] Example of feedback generated based on this prompt:

[1749] "In this step, it's important to understand the role of each parameter. Parameter A adjusts the scale of the data, and parameter B controls the learning rate. For example..."

[1750] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1751] Step 1:

[1752] The user accesses the login screen using a terminal and enters their username and password.

[1753] Input: Username, Password

[1754] Output: Encrypted login information

[1755] Specific behavior:

[1756] The user enters their username and password into the login screen of their device, which then encrypts the information using the AES encryption algorithm and sends it to the server.

[1757] Step 2:

[1758] The server receives the encrypted login information, decrypts it, and authenticates it by checking it against user data in a database.

[1759] Input: Encrypted login information

[1760] Output: Authentication results, profile data

[1761] Specific behavior:

[1762] The server decrypts the received login information using the AES decryption algorithm and compares it with the user data in its database. If authentication is successful, it retrieves the profile data, encrypts it, and sends it back to the device.

[1763] Step 3:

[1764] The terminal decrypts the profile data received from the server and displays it on the user screen.

[1765] Input: Encrypted profile data

[1766] Output: Display profile information

[1767] Specific behavior:

[1768] The device will then decrypt the received encrypted profile data using the AES decryption algorithm and display it on the user's screen, allowing the user to view their profile information.

[1769] Step 4:

[1770] The user selects the field or topic they wish to study on their device and sends the selected information to the server.

[1771] Input: Select the field or topic you want to study

[1772] Output: Encrypted selection information

[1773] Specific behavior:

[1774] Users select the fields and topics they want to learn on their device, which then encrypts this selection using the AES encryption algorithm and sends it to the server.

[1775] Step 5:

[1776] The server receives the encrypted selection information, decrypts it, and uses generative AI to generate a custom learning curriculum.

[1777] Input: Encrypted selection information

[1778] Output: Custom learning curriculum

[1779] Specific behavior:

[1780] The server decrypts the received selection information using the AES decryption algorithm and inputs it into the generative AI model, which then generates a custom learning curriculum based on the user's selection, encrypts it, and sends it to the device.

[1781] Step 6:

[1782] The terminal receives the encrypted learning curriculum, decrypts it, and displays it to the user.

[1783] Input: Encrypted learning curriculum

[1784] Output: Display of learning curriculum

[1785] Specific behavior:

[1786] The terminal decrypts the received encrypted curriculum using the AES decryption algorithm and displays it on the user's screen, allowing the user to view the customized curriculum.

[1787] Step 7:

[1788] As the user progresses with their learning, the device uses cameras and sensors to capture facial and voice data in real time and sends it to the emotion engine.

[1789] Input: User's facial expression data, voice data

[1790] Output: Emotion analysis results

[1791] Specific behavior:

[1792] The device uses cameras and sensors to capture the user's facial expressions and voice in real time and transmits the data to the emotion engine, which analyzes this data to detect the user's emotional state.

[1793] Step 8:

[1794] The server adjusts the necessary parts of the learning curriculum based on the emotional data received from the emotion engine, generates real-time feedback, and sends it to the terminal.

[1795] Input: Sentiment analysis results, learning progress data

[1796] Output: Curriculum adjustments, real-time feedback

[1797] Specific behavior:

[1798] The server generates customized feedback based on the emotion analysis results and learning progress data. The feedback content is created using a generative AI model, encrypted, and sent to the device.

[1799] Step 9:

[1800] The terminal receives the encrypted feedback, decrypts it and displays it on the user's screen.

[1801] Input: Encrypted feedback

[1802] Output: Display of feedback content

[1803] Specific behavior:

[1804] The device uses the AES decryption algorithm to decrypt the received encrypted feedback and displays it on the user's screen, where the user can see specific advice and tips from the generated AI model.

[1805] Step 10:

[1806] The user selects a virtual project and the terminal transmits the information to the server.

[1807] Input: Virtual project selection information

[1808] Output: Encrypted selection information

[1809] Specific behavior:

[1810] The user selects a virtual project on the terminal, which encrypts the selection information using the AES encryption algorithm and sends it to the server.

[1811] Step 11:

[1812] The server receives the encrypted selection information and uses metaverse technology to set up a virtual environment and load the project content.

[1813] Input: Encrypted selection information

[1814] Output: Virtual environment configuration information

[1815] Specific behavior:

[1816] The server decrypts the received selection information using the AES decryption algorithm, sets up a virtual environment using metaverse technology, loads the selected project content, and sends the configuration information to the terminal.

[1817] Step 12:

[1818] The terminal receives the setting information of the virtual environment and connects the user to the virtual environment.

[1819] Input: Virtual environment configuration information

[1820] Output: Virtual environment connected

[1821] Specific behavior:

[1822] The device decodes the received virtual environment configuration information and connects the user to the virtual environment, allowing the user to participate in virtual projects and progress in their learning.

[1823] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1824] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1825] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1826] [Fourth embodiment]

[1827] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1828] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1829] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1830] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1831] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1832] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1833] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1834] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1835] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1836] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1837] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1838] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1839] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1840] This invention is a new learning system that combines generative AI and the Metaverse, aiming to provide users with customized learning experiences and practical project experiences.

[1841] System configuration and program processing

[1842] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[1843] 1. User login and profile settings

[1844] User: Uses a terminal to access the system login screen and enters a username and password.

[1845] On the device: The login information entered by the user is encrypted and sent to the server.

[1846] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[1847] Server: Encrypts the profile data and sends it back to the device.

[1848] Terminal: Decodes the received profile data and displays it on the screen.

[1849] Examples:

[1850] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[1851] 2. Customize your learning curriculum

[1852] Users: Choose the subject or topic they want to learn about.

[1853] Terminal: Encrypts the selected information and sends it to the server.

[1854] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[1855] Server: Encrypts the generated learning curriculum and sends it to the device.

[1856] Terminal: Decodes the received curriculum and displays it to the user.

[1857] Examples:

[1858] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[1859] 3. Setting up and participating in virtual projects

[1860] Users: Select a virtual project within the curriculum.

[1861] On your device: Encrypt the project selection information and send it to the server.

[1862] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[1863] Server: Encrypts the virtual environment configuration information and sends it to the device.

[1864] Terminal: Connects the user to the virtual environment based on the received configuration information.

[1865] Examples:

[1866] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[1867] 4. Personalized instruction and real-time feedback

[1868] Users: Work on projects in a virtual environment.

[1869] Device: Sends completed tasks and progress information to the server in real time.

[1870] Server: Analyzes progress information and uses generative AI to generate feedback appropriate for the user.

[1871] Server: Encrypts the generated feedback and sends it to the device.

[1872] Terminal: Display feedback on the user's screen.

[1873] Examples:

[1874] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[1875] As described above, this system leverages generative AI and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to grow.

[1876] The processing flow will be explained below.

[1877] Step 1:

[1878] The user uses a terminal to access the system's login screen and enters their username and password.

[1879] Step 2:

[1880] The terminal encrypts the entered login information and sends it to the server.

[1881] Step 3:

[1882] The server compares the received login information with the user data in the database and performs authentication.

[1883] Step 4:

[1884] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[1885] Step 5:

[1886] The device decodes the received profile data and displays it on the screen.

[1887] Step 6:

[1888] Users select the field or topic they want to learn on their device.

[1889] Step 7:

[1890] The terminal encrypts the selected information and sends it to the server.

[1891] Step 8:

[1892] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[1893] Step 9:

[1894] The learning curriculum generated by the server is encrypted and sent to the terminal.

[1895] Step 10:

[1896] The terminal decodes the received curriculum and displays it to the user.

[1897] Step 11:

[1898] A user selects a virtual project within a curriculum.

[1899] Step 12:

[1900] The terminal encrypts the project selection information and transmits it to the server.

[1901] Step 13:

[1902] The server sets up the metaverse virtual environment and loads the selected project contents.

[1903] Step 14:

[1904] The server encrypts the virtual environment configuration information and sends it to the terminal.

[1905] Step 15:

[1906] The terminal connects the user to the virtual environment based on the configuration information received.

[1907] Step 16:

[1908] Users work on projects in a virtual environment.

[1909] Step 17:

[1910] The device sends progress information to the server in real time.

[1911] Step 18:

[1912] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[1913] Step 19:

[1914] The server generates feedback that is encrypted and sent to the terminal.

[1915] Step 20:

[1916] The terminal displays the received feedback to the user and provides specific advice.

[1917] Example 1

[1918] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1919] Conventional learning systems often offer only a standard curriculum, making it difficult to customize to meet individual user needs. They also struggle to provide real-time feedback and practical project experiences. Providing a learning environment utilizing a virtual reality environment, in particular, poses numerous technical challenges and is difficult to implement.

[1920] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1921] In this invention, the server includes means for authenticating login information and obtaining user profile data, means for generating a learning curriculum using a generative AI model based on the selection information, means for configuring a metaverse environment and loading the selected projects, and means for monitoring completed tasks and progress information in real time and providing real-time feedback using the generative AI model, thereby providing a personalized learning experience and hands-on project experience with real-time feedback.

[1922] "User" refers to the person who operates the system and experiences the learning curriculum and virtual projects.

[1923] "Device" refers to a computer or mobile device on which a user performs operations such as entering login information, selecting a course of study, or participating in a virtual project.

[1924] "Server" refers to a central processing unit that authenticates user login information, obtains profile data, generates learning curricula using generative AI models, sets up the metaverse environment, and generates real-time feedback.

[1925] "Generative AI model" refers to an artificial intelligence algorithm used to generate personalized curriculum based on user selections and provide real-time feedback.

[1926] "Login Information" refers to authentication information, such as a username and password, entered by a User to access a System.

[1927] "Profile Data" refers to data that includes personal information such as a user's name, learning history, and settings.

[1928] A "learning curriculum" refers to a series of learning content and assignments generated based on the field or topic a user wants to learn.

[1929] A "virtual project" refers to a specific challenge or task that a user participates in and completes within a virtual reality environment.

[1930] A "metaverse environment" refers to a virtual work or learning environment built using virtual reality technology.

[1931] "Real-time feedback" refers to instantaneous evaluation and advice provided to users based on their progress.

[1932] This invention is a new learning system that combines generative AI models and the Metaverse. The purpose of this system is to provide users with customized learning experiences and practical project experiences. Below, we will explain how to implement this system in detail.

[1933] This system consists of three main components: a server, a terminal, and a user.

[1934] Hardware and software used

[1935] Server: Serves as a central processing unit, authenticating users, managing profile data, generating learning curricula using generative AI models (e.g., GPT-4), configuring the metaverse environment, and generating real-time feedback. Implemented in a server system that also includes databases and AI algorithms.

[1936] Terminal: A device operated by a user, such as a computer, smartphone, or tablet. The software on the terminal provides the user interface, performs encryption / decryption processing, communicates with the server, and displays information to the user.

[1937] User: An individual who uses the learning system. The user operates the device and progresses with the learning.

[1938] Program processing

[1939] User login and profile settings

[1940] A user uses a terminal to access the system's login screen and enters their username and password. The terminal encrypts this login information (e.g., AES encryption) and sends it to the server. The server decrypts the received login information and authenticates it by comparing it with information in a database. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it to the terminal. The terminal decrypts the data and displays it to the user.

[1941] Customized learning curriculum

[1942] The user selects the field or topic they wish to study. The device encrypts the selection information (e.g., RSA encryption) and sends it to the server. The server analyzes the received selection information using a generative AI model (e.g., GPT-4) and generates a learning curriculum optimal for the user. The generated curriculum is encrypted and sent to the device. The device decrypts the curriculum and displays it to the user.

[1943] Examples:

[1944] The user selects that they want to learn machine learning and data science. The device RSA-encrypts this information and sends it to the server. The server uses GPT-4 to generate a customized curriculum and sends it to the device. The device then displays the customized curriculum on the user's screen.

[1945] Setting up and participating in virtual projects

[1946] The user selects a virtual project within the curriculum. The device encrypts the selection information and sends it to the server. The server analyzes the received information and sets up the virtual project within the metaverse environment. The set virtual environment information is then encrypted and sent to the device. The device decrypts it and connects the user to the virtual environment.

[1947] Examples:

[1948] The user selects a virtual data science project. The device encrypts this selection information and sends it to the server. The server configures the metaverse environment and loads the project content. The server encrypts the environment configuration information and sends it to the device, which then connects the user to the virtual environment.

[1949] Personalized instruction and real-time feedback

[1950] Users work on projects in a virtual environment. The device sends completed tasks and progress information to the server in real time. The server analyzes the progress information and generates appropriate feedback for the user using a generative AI model (e.g., Chat GPT). The generated feedback is encrypted and sent to the device, which then displays the feedback on the user's screen.

[1951] Examples:

[1952] The user performs data analysis in a virtual environment. The device sends progress information to the server in real time. The server uses this information to generate feedback using a generative AI model and sends it to the device. The device then displays the feedback on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[1953] Prompt Sentence Examples

[1954] "Users want to learn machine learning from the basics to advanced applications. Please generate a customized learning curriculum."

[1955] As described above, the system leverages generative AI models and the Metaverse to provide users with customized learning experiences and hands-on projects, allowing them to hone their skills in a realistic virtual environment and receive relevant feedback in real time to help them grow.

[1956] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1957] Step 1:

[1958] A user accesses the login screen of the system using a terminal and enters a user name and password, which become the input information.

[1959] The terminal encrypts the entered login information using the AES encryption algorithm and sends it to the server, where it outputs the encrypted login information.

[1960] Step 2:

[1961] The server decrypts the received encrypted login information using the AES decryption algorithm.

[1962] The server compares the decrypted login information with the user data in the database and performs authentication. The user data from the database is used as input information. If authentication is successful, the user's profile data is retrieved.

[1963] The profile data is encrypted using the AES encryption algorithm and sent to the terminal, and this encrypted profile data is the output.

[1964] Step 3:

[1965] The terminal decrypts the received encrypted profile data using the AES decryption algorithm.

[1966] The decrypted profile data is displayed on the user's screen, which is the output.

[1967] Step 4:

[1968] The user selects the field or topic they want to learn about, and this selection information becomes the input information.

[1969] The terminal encrypts the user's selection information using the RSA encryption algorithm and sends it to the server, where the encrypted selection information is output.

[1970] Step 5:

[1971] The server decrypts the received encrypted selection information using the RSA decryption algorithm.

[1972] The server generates a custom learning curriculum using a generative AI model (e.g., GPT-4) based on the decoded selection information. This selection information serves as input information.

[1973] The generated learning curriculum is encrypted using the AES encryption algorithm and sent to the terminal. This encrypted learning curriculum is the output.

[1974] Step 6:

[1975] The terminal decrypts the received encrypted learning curriculum using the AES decryption algorithm.

[1976] The decoded learning curriculum is displayed on the user's screen, which is the output.

[1977] Step 7:

[1978] The user selects a virtual project within the curriculum, and this selection information becomes the input information.

[1979] The device hashes the project selection information using the SHA-256 hashing algorithm and sends it to the server, where the hashed selection information is the output.

[1980] Step 8:

[1981] The server parses the received hashed project selection information.

[1982] The server configures the metaverse environment and loads the selected project contents. This configuration information becomes the input information.

[1983] The configured virtual environment information is hashed using SHA-256 and sent to the terminal. This hashed configuration information is output.

[1984] Step 9:

[1985] The terminal connects the user to the metaverse virtual project based on the received hashed virtual environment information.

[1986] The output is that the user is connected to the virtual environment.

[1987] Step 10:

[1988] Users work on projects within a virtual environment, and their actions within this environment provide input information.

[1989] The terminal transmits completed tasks and progress information to the server in real time using the WebSocket protocol, and this progress information is the output.

[1990] Step 11:

[1991] The server analyzes the received progress information and generates optimal feedback for the user using a generative AI model (e.g., Chat GPT). The progress information is used as input.

[1992] The generated feedback is encrypted via WebSocket and sent to the terminal. This encrypted feedback becomes the output.

[1993] Step 12:

[1994] The terminal decrypts the received encrypted feedback.

[1995] The decoded feedback is displayed on the user's screen, which is the output.

[1996] In this way, by performing specific operations and processing and calculating data at each step, it is possible to provide the user with a customized learning experience.

[1997] (Application example 1)

[1998] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1999] Conventional learning systems have struggled to provide customized learning tailored to individual users' needs or practical project experiences in virtual environments. They also struggled to provide optimal feedback on users' learning progress in real time. Especially in distance learning and online education, there is a lack of individualized instruction, which can lead to insufficient learning outcomes.

[2000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2001] In this invention, the server includes a means for generating customized advice based on the user's learning progress and displaying it to the user on the terminal, a means for providing real-time feedback using generative AI, and a means for providing a project experience in a virtual reality environment using the metaverse, allowing users to enjoy customized learning and practical project experience tailored to their individual needs and receive optimal feedback in real time.

[2002] "User" refers to someone who uses the system to study or work on a project.

[2003] "Terminal" refers to a device operated by a user, used to input login information and selection information, display profile data and learning curriculum, connect to a virtual environment, and so on.

[2004] "Server" refers to a device that receives and processes information sent from a terminal and has functions such as user authentication, acquisition of profile data, generation of learning curriculum, setting up a virtual environment, and generation of real-time feedback.

[2005] "Login Information" refers to the authentication information required for a user to access a system, and typically consists of a username and password.

[2006] "Profile data" refers to data that includes information such as the user's personal information, learning history, and skill level, and is acquired by the server and returned to the terminal when authentication is successful.

[2007] "Learning curriculum" refers to the learning content or study plan for a specific field or topic that is generated by the server based on user selection information.

[2008] A "virtual project" refers to a specific assignment or task selected by a user and executed in a virtual environment within the metaverse.

[2009] The "metaverse" is a virtual environment constructed using virtual reality technology, and refers to a virtual space in which users can interact with each other.

[2010] "Generative AI" refers to artificial intelligence technology that generates appropriate advice and feedback based on a user's learning progress and project content.

[2011] "Feedback" refers to the evaluation and advice provided in real time by generative AI for tasks or projects that users are currently working on.

[2012] "Customized advice" refers to personalized advice that suggests specific content and methods based on the user's learning progress and skill level.

[2013] Three main components are required to implement this invention: a server, a terminal, and a user. The system provides users with a customized learning experience and project experience in a virtual environment.

[2014] First, a user uses a terminal to access the system's login screen. Here, they enter their username and password. The terminal encrypts the entered login information and sends it to the server. The server compares the received login information with the user data in its database and performs authentication. If authentication is successful, the server obtains the user's profile data, encrypts it, and sends it back to the terminal. The terminal decrypts the profile data and displays it to the user.

[2015] Next, the learning curriculum is customized. The user selects the fields and topics they want to study. The device encrypts these selections and sends them to the server. The server uses a generative AI model to generate a custom learning curriculum based on the user's selections. The generated curriculum is then encrypted again and sent to the device. The device decrypts the curriculum data and displays it to the user.

[2016] The user then selects a virtual project within the curriculum. The device encrypts the project selection information and sends it to the server. The server then configures the metaverse virtual environment and loads the selected project content. The virtual environment configuration information is then encrypted and sent to the device. The device then connects the user to the virtual environment based on the configuration information.

[2017] As a user works on a project in a virtual environment, the device sends completed tasks and progress information to the server in real time. The server analyzes this information and uses a generative AI model to generate appropriate feedback for the user. The generated feedback is encrypted and sent to the device, which then decrypts it and displays it to the user. This feedback can include specific advice, such as "Try a different regression algorithm for this step."

[2018] The hardware used includes smartphones and head-mounted displays (HMDs), and the software includes Unity (for building the virtual environment), AWS (for servers and databases), and OpenAI's GPT-4 generative AI.

[2019] Below are some examples of prompt sentences.

[2020] "User wants feedback on data analysis. Please provide specific advice."

[2021] The system allows users to receive customized learning tailored to their individual needs and improve their skills through hands-on project experience in a virtual environment, with real-time feedback providing a significant learning experience.

[2022] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2023] Step 1:

[2024] The user accesses the login screen of the learning system using a smartphone or head-mounted display (HMD). The user enters their username and password. This is used as input data to proceed to the next step.

[2025] Step 2:

[2026] The terminal encrypts the login information entered by the user. This encrypted data is sent to the server. The input is the login information, and the output is the encrypted login data.

[2027] Step 3:

[2028] The server decrypts the encrypted login information it receives and authenticates it by comparing it with the user data in the database. If authentication is successful, it obtains the user's profile data. The input is the encrypted login data, and the output is the user's profile data.

[2029] Step 4:

[2030] The server re-encrypts the acquired profile data and sends it to the terminal. The input is the profile data, and the output is the encrypted profile data.

[2031] Step 5:

[2032] The device decrypts the received profile data and displays it to the user, allowing the user to check their own profile information. The input is encrypted profile data, and the output is decrypted profile data.

[2033] Step 6:

[2034] The user selects the field or topic they want to study, and with this as input, they move on to the next step.

[2035] Step 7:

[2036] The terminal encrypts information about the learning field or topic selected by the user and sends it to the server. The input is the selected information, and the output is the encrypted selected information.

[2037] Step 8:

[2038] The server decodes the received selection information and generates a custom learning curriculum using a generative AI model. The input is the selection information and the output is the custom learning curriculum.

[2039] Step 9:

[2040] The server encrypts the generated curriculum and sends it to the terminal. The input is the custom curriculum, and the output is the encrypted curriculum.

[2041] Step 10:

[2042] The terminal decrypts the received curriculum and displays it to the user. The user can then view the customized curriculum. The input is the encrypted curriculum, and the output is the decrypted curriculum.

[2043] Step 11:

[2044] The user selects a virtual project within the learning curriculum, which is used as input data to proceed to the next step.

[2045] Step 12:

[2046] The terminal encrypts the project selection information and sends it to the server. The input is the project selection information, and the output is the encrypted project selection information.

[2047] Step 13:

[2048] The server decodes the received project selection information, configures the metaverse virtual environment, and loads the selected project content. The input is the project selection information, and the output is the virtual environment configuration information.

[2049] Step 14:

[2050] The server encrypts the virtual environment setting information and sends it to the terminal. The input is the virtual environment setting information, and the output is the encrypted virtual environment setting information.

[2051] Step 15:

[2052] The terminal decrypts the received virtual environment setting information and connects the user to the virtual environment. The input is the encrypted virtual environment setting information, and the output is the connection to the virtual environment.

[2053] Step 16:

[2054] A user works on a project in a virtual environment. The device sends completed tasks and progress information to the server in real time. The input is tasks and progress information, and the output is encrypted task and progress information.

[2055] Step 17:

[2056] The server interprets the received progress information and uses a generative AI model to generate feedback appropriate for the user. The input is progress information, and the output is customized feedback.

[2057] Step 18:

[2058] The server generates and encrypts the feedback and sends it to the device. The input is the customized feedback, and the output is the encrypted feedback.

[2059] Step 19:

[2060] The device decrypts the received feedback and displays it to the user, allowing them to receive specific advice based on their progress. The input is encrypted feedback, and the output is decrypted feedback.

[2061] Through this process, users can receive personalized learning experiences and optimal feedback in real time. The entire system is powered by a combination of hardware such as smartphones and head-mounted displays, and software such as Unity, AWS, and generative AI models.

[2062] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2063] This invention is a new learning system that combines generative AI, the Metaverse, and an emotion engine that recognizes users' emotions. This system provides users with customized learning experiences and practical projects, and provides real-time feedback and curriculum adjustments based on the user's emotions.

[2064] System configuration and program processing

[2065] This system consists of three main components: the server, the terminal, and the user. The roles and specific operations of each component are described below.

[2066] 1. User login and profile settings

[2067] User: Uses a terminal to access the system login screen and enters a username and password.

[2068] On the device: The login information entered by the user is encrypted and sent to the server.

[2069] Server: The received login information is verified against the user data in the database for authentication. If authentication is successful, the user's profile data is retrieved.

[2070] Server: Encrypts the profile data and sends it back to the device.

[2071] Terminal: Decodes the received profile data and displays it on the screen.

[2072] Examples:

[2073] The user accesses the login screen of the device and enters their username and password. The device encrypts the entered information and sends it to the server. The server verifies it against the information in the database and, if successful, sends the profile data to the device. The device then displays the profile data on the user's screen.

[2074] 2. Customize your learning curriculum

[2075] Users: Select the subject or topic they want to learn on their device.

[2076] Terminal: Encrypts the selected information and sends it to the server.

[2077] Server: Uses generative AI to generate a custom learning curriculum based on user selections.

[2078] Server: Encrypts the generated learning curriculum and sends it to the device.

[2079] Terminal: Decodes the received curriculum and displays it to the user.

[2080] Examples:

[2081] A user selects that they want to learn machine learning and data science. The device encrypts this information and sends it to the server. The server uses generative AI to generate a curriculum tailored to the user and sends it to the device. The device then displays the customized curriculum on the user's screen.

[2082] 3. Emotion recognition and curriculum adjustment using an emotion engine

[2083] Users: Experience changes in emotions (e.g., joy, confusion, anger, etc.) as they progress through the learning process.

[2084] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[2085] Terminal: The acquired data is sent to the emotion engine and the user's emotions are analyzed.

[2086] Server: Based on the analyzed emotional data, the learning curriculum and project content are adjusted as needed.

[2087] Examples:

[2088] If a user shows a confused expression during the project, the device captures the user's facial expression through the camera, and the emotion engine analyzes it to detect "confusion." The server uses this information to adjust the curriculum, for example, by providing additional explanations or hints.

[2089] 4. Setting up and participating in virtual projects

[2090] Users: Select a virtual project within the curriculum.

[2091] On your device: Encrypt the project selection information and send it to the server.

[2092] Server: Sets up the metaverse virtual environment and loads the selected project contents.

[2093] Server: Encrypts the virtual environment configuration information and sends it to the device.

[2094] Terminal: Connects the user to the virtual environment based on the received configuration information.

[2095] Examples:

[2096] The user selects a data science project. The device encrypts this selection and sends it to the server. The server configures a metaverse virtual environment and loads the project content. The server then sends the configuration information to the device, which then connects the user to the virtual environment.

[2097] 5. Personalized instruction and real-time feedback

[2098] Users: Work on projects in a virtual environment.

[2099] Device: Sends progress information to the server in real time.

[2100] Server: Analyzes progress information and uses generative AI to generate appropriate feedback for the user.

[2101] Server: Encrypts the generated feedback and sends it to the device.

[2102] Terminal: Display feedback on the user's screen.

[2103] Examples:

[2104] The user performs data analysis work in a virtual environment. The device sends information about the progress of the work to the server in real time. The server uses generative AI based on this information to generate feedback and sends it to the device. The device then displays the feedback information on the user's screen and provides specific advice, such as "Try a specific algorithm for this step."

[2105] As described above, this system utilizes generative AI, metaverse technology, and an emotion engine to provide users with customized learning experiences and hands-on project experiences. Users can hone their skills in a realistic virtual environment and grow while receiving appropriate feedback in real time. Furthermore, the emotion engine enables instruction and curriculum to be adjusted based on the user's emotions, creating a more personalized learning experience.

[2106] The processing flow will be explained below.

[2107] Step 1:

[2108] The user uses a terminal to access the system's login screen and enters their username and password.

[2109] Step 2:

[2110] The terminal encrypts the entered login information and sends it to the server.

[2111] Step 3:

[2112] The server compares the received login information with the user data in the database and performs authentication.

[2113] Step 4:

[2114] If the server is successful in the authentication, it obtains the user's profile data, encrypts it, and returns it to the terminal.

[2115] Step 5:

[2116] The device decodes the received profile data and displays it on the screen.

[2117] Step 6:

[2118] Users select the field or topic they want to learn through the device interface.

[2119] Step 7:

[2120] The terminal encrypts the selection information and transmits it to the server.

[2121] Step 8:

[2122] The server uses generative AI to generate a custom learning curriculum based on the user's selections.

[2123] Step 9:

[2124] The learning curriculum generated by the server is encrypted and sent to the terminal.

[2125] Step 10:

[2126] The terminal decodes the received curriculum and displays it to the user.

[2127] Step 11:

[2128] As the user progresses through the curriculum, the device uses a camera and microphone to capture facial expressions and voice in real time.

[2129] Step 12:

[2130] The device sends the acquired emotional data (facial expressions, voice, etc.) to the emotion engine, which analyzes the user's emotions.

[2131] Step 13:

[2132] The server adjusts the learning curriculum and project content as needed based on the analyzed emotional data.

[2133] Step 14:

[2134] A user selects a virtual project within a curriculum.

[2135] Step 15:

[2136] The terminal encrypts the project selection information and transmits it to the server.

[2137] Step 16:

[2138] The server sets up the metaverse virtual environment and loads the selected project contents.

[2139] Step 17:

[2140] The server encrypts the virtual environment configuration information and sends it to the terminal.

[2141] Step 18:

[2142] The terminal connects the user to the virtual environment based on the configuration information received.

[2143] Step 19:

[2144] Users work on projects in a virtual environment.

[2145] Step 20:

[2146] The device sends progress information to the server in real time.

[2147] Step 21:

[2148] The server analyzes the progress information and uses generative AI to generate feedback appropriate for the user.

[2149] Step 22:

[2150] The server generates feedback that is encrypted and sent to the terminal.

[2151] Step 23:

[2152] The terminal displays the received feedback to the user and provides specific advice.

[2153] Example 2

[2154] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2155] Conventional learning systems have difficulty dynamically adjusting learning curriculum and feedback based on the user's individual emotions and progress, resulting in problems such as reduced learning efficiency and motivation. Furthermore, project experiences using virtual reality are limited, and a more realistic learning environment is needed.

[2156] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2157] In this invention, the server includes a means for analyzing a user's emotional data and adjusting the curriculum, a means for providing customized real-time feedback using a generative AI model, and a means for providing a project experience using virtual reality technology, thereby personalizing the user's learning experience and enabling appropriate feedback and curriculum adjustment according to emotions and progress.

[2158] "User" refers to an individual who uses this system to learn or experience a project.

[2159] "Terminal" refers to an information processing device used by a user, such as a computer, tablet, or smartphone.

[2160] "Server" refers to the central computer device that performs the main processing and data management of the System.

[2161] "Login Information" means authentication information, such as a username and password, required for a User to access a System.

[2162] "Profile data" refers to data about a user, such as personal information, history, and learning progress.

[2163] "Learning curriculum" refers to the learning plans and materials created based on the field or topic selected by the user.

[2164] An "emotion analysis system" refers to a system that analyzes acquired facial expressions and voice data of a user and recognizes their emotions.

[2165] "Virtual environment" refers to a simulated environment constructed using virtual reality technology.

[2166] A "generative AI model" refers to an artificial intelligence model that generates learning curriculum and feedback based on user input data and situations.

[2167] "Real-time feedback" refers to advice and information provided instantly to users based on their progress and actions.

[2168] The system of the present invention is composed of three main components: a user, a terminal, and a server. The roles and operations of each component are described below.

[2169] System Overview

[2170] This system comprehensively handles everything from login authentication to learning curriculum generation, real-time feedback, virtual project setup and participation, and even curriculum adjustment based on sentiment analysis. By communicating with each other, the server, terminals, and users provide the optimal learning experience for users.

[2171] Hardware and software used

[2172] Device: Computer, tablet, smartphone, etc.

[2173] Server: A high-performance computing device that works in conjunction with a database.

[2174] Generative AI models: Artificial intelligence models such as GPT-3 and ChatGPT.

[2175] Sentiment analysis system: Emotion detection API (e.g. Microsoft Azure Emotion API).

[2176] Virtual reality technology: Virtual environment engines that support the Metaverse (e.g., Unity, Unreal Engine).

[2177] Specific examples of program processing

[2178] 1. User login and profile settings

[2179] Examples:

[2180] User: Enter "user123" and "password" on the login screen.

[2181] Terminal: Encrypt "user123" and "password" with AES256 and send to the server.

[2182] Server: Search for "user123" in the database and match the password. If successful, retrieve the user's profile (e.g., name, email address).

[2183] Server: The acquired profile data is encrypted using AES256 and sent to the device.

[2184] Terminal: Decrypts the encrypted data and displays "Hello, user123" on the user's screen.

[2185] Example prompt sentence:

[2186] "Please enter your username and password to log in."

[2187] 2. Customize your learning curriculum

[2188] Examples:

[2189] User: Choose Machine Learning and Data Science.

[2190] Terminal: Encrypts the selected information and sends it to the server.

[2191] Server: Generates a curriculum for "Machine Learning and Data Science" using GPT-3.

[2192] Server: Encrypts the curriculum and sends it to the device.

[2193] Terminal: Decodes the curriculum and displays it to the user (e.g., "Chapter 1: Data Preprocessing, Chapter 2: Basic Algorithms").

[2194] Example prompt sentence:

[2195] "Choose the field you want to study. Example: Machine Learning, Data Science."

[2196] 3. Emotion recognition and curriculum adjustment using an emotion engine

[2197] Examples:

[2198] User: Shows confused expression while studying.

[2199] Device: Capture confused expressions through camera and microphone.

[2200] Terminal: Sends data to an emotion analysis system to recognize "confusion."

[2201] Server: Uses generative AI models to generate additional hints and explanations in response to puzzles and adjust the learning curriculum.

[2202] Example prompt sentence:

[2203] "Confused expression detected, show hint."

[2204] 4. Setting up and participating in virtual projects

[2205] Examples:

[2206] User: Choose a data science project.

[2207] Terminal: This selection information is encrypted and sent to the server.

[2208] Server: Sets up the metaverse virtual environment and loads project content (e.g., analyzing data, building models).

[2209] Server: Encrypts the configuration information and sends it to the device.

[2210] Terminal: Connects the user to the virtual environment.

[2211] Example prompt sentence:

[2212] "Choose a hypothetical project. Example: Data Analysis Project."

[2213] 5. Personalized instruction and real-time feedback

[2214] Examples:

[2215] User: Work on data analysis within the virtual environment.

[2216] Terminal: Sends work progress information (e.g., data preprocessing completed) to the server in real time.

[2217] Server: Parses progress information and generates feedback using ChatGPT (e.g., "Try a specific algorithm at this step").

[2218] Server: Encrypts the feedback information and sends it to the device.

[2219] Terminal: Display feedback information on the user's screen.

[2220] Example prompt sentence:

[2221] "We are sending you project status updates, please wait."

[2222] Through these steps, the system provides users with a customized, efficient, and meaningful learning experience. By combining generative AI models, sentiment analysis systems, and virtual reality technology, it is possible to monitor users' progress in real time and provide optimal feedback.

[2223] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2224] The flow of this system's program processing

[2225] Step 1: Enter user login information

[2226] 1. Input:

[2227] User: Enter your username and password on the login screen.

[2228] 2. Processing:

[2229] Terminal: The entered username and password are encrypted using the AES256 algorithm.

[2230] 3. Output:

[2231] Encrypted login information is generated.

[2232] Step 2: Submit and authenticate login information

[2233] 1. Input:

[2234] Encrypted login information.

[2235] 2. Processing:

[2236] On your device: Sends encrypted login information to the server.

[2237] Server: receives the encrypted login information and authenticates it against the user data in the database (using the SHA-256 algorithm).

[2238] 3. Output:

[2239] The authentication result (success or failure).

[2240] On success, the user's profile data.

[2241] Step 3: Submit and view your profile data

[2242] 1. Input:

[2243] Authentication result and profile data (if successful).

[2244] 2. Processing:

[2245] Server: If authentication is successful, the profile data is encrypted with AES256 and sent to the device.

[2246] Terminal: Decrypts the received profile data and displays it on the user's screen.

[2247] 3. Output:

[2248] The user's profile will be displayed on the screen.

[2249] Step 4: Choose a subject or topic you want to study

[2250] 1. Input:

[2251] Users: Choose the field or topic they want to learn about (e.g., machine learning, data science).

[2252] 2. Processing:

[2253] Terminal: Encrypts the selected information and sends it to the server.

[2254] 3. Output:

[2255] Encrypted selection information.

[2256] Step 5: Generate and submit a custom learning curriculum

[2257] 1. Input:

[2258] Encrypted selection information.

[2259] 2. Processing:

[2260] Server: Decodes the selection information and generates a custom learning curriculum using a generative AI model (e.g., GPT-3).

[2261] Server: Encrypts the generated curriculum with AES256 and sends it to the device.

[2262] 3. Output:

[2263] Encrypted custom learning curriculum.

[2264] Step 6: Decipher and display the curriculum

[2265] 1. Input:

[2266] Encrypted custom learning curriculum.

[2267] 2. Processing:

[2268] Terminal: Decodes the received curriculum and displays it to the user.

[2269] 3. Output:

[2270] A customized learning curriculum is displayed to the user.

[2271] Step 7: Acquire and analyze emotion data

[2272] 1. Input:

[2273] User: Facial expressions and voice changes while learning.

[2274] 2. Processing:

[2275] Device: Uses cameras and sensors to capture the user's facial expressions and voice in real time.

[2276] Terminal: The acquired data is sent to the emotion analysis system for analysis.

[2277] 3. Output:

[2278] Sentiment analysis results (e.g., happy, confused, angry).

[2279] Step 8: Adjust the curriculum

[2280] 1. Input:

[2281] Emotion analysis results.

[2282] 2. Processing:

[2283] Server: Based on the analyzed emotional data, the learning curriculum is adjusted as needed.

[2284] 3. Output:

[2285] Coordinated learning curriculum.

[2286] Step 9: Select and configure your virtual project

[2287] 1. Input:

[2288] User: Select a virtual project.

[2289] 2. Processing:

[2290] On your device: Encrypt the project selection information and send it to the server.

[2291] Server: Sets up the metaverse virtual environment based on the selected project and loads the project.

[2292] Server: Encrypts the configuration information and sends it to the device.

[2293] 3. Output:

[2294] Virtual environment configuration data.

[2295] Step 10: Connect to your virtual environment

[2296] 1. Input:

[2297] Virtual environment configuration data.

[2298] 2. Processing:

[2299] Terminal: Connects the user to the virtual environment based on the received configuration data.

[2300] 3. Output:

[2301] The user is connected to the virtual environment.

[2302] Step 11: Submit project work progress and generate feedback

[2303] 1. Input:

[2304] User: Working on a project in a virtual environment.

[2305] 2. Processing:

[2306] Terminal: Sends work progress information to the server in real time.

[2307] Server: Analyzes progress information and generates optimal feedback using generative AI models.

[2308] Server: Encrypts the feedback and sends it to the device.

[2309] 3. Output:

[2310] Real-time feedback.

[2311] Step 12: Viewing feedback

[2312] 1. Input:

[2313] Encrypted real-time feedback.

[2314] 2. Processing:

[2315] Terminal: Decodes the feedback and displays it to the user.

[2316] 3. Output:

[2317] Appropriate feedback is provided to the user.

[2318] The above are the specific processing steps of this system's program, and we have explained in detail how the data is input at each step, the data processing and calculations performed, and how the output is obtained. By combining these steps, it is possible to provide the user with an optimal learning experience.

[2319] (Application example 2)

[2320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2321] Conventional learning systems have difficulty adjusting in real time to the user's learning progress and emotions, and tend to provide a one-way curriculum. As a result, they are unable to flexibly respond to the user's level of understanding and motivation. Furthermore, even in methods that provide practical learning experiences using virtual environments, the lack of feedback based on the user's emotions and progress can reduce learning effectiveness. Furthermore, there is room for improvement in the operability and accuracy of real-time feedback when using devices such as smartphones and head-mounted displays.

[2322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2323] In this invention, the server includes a means for analyzing the user's facial expressions and voice using an emotion engine and adjusting the curriculum based on real-time feedback, a means for the smartphone or head-mounted display to acquire the user's emotion data and send it to the server, and a means for the server to monitor the user's progress based on the emotion data and provide real-time feedback using generative AI, thereby making it possible to provide a customized learning experience according to the user's individual progress and emotions.

[2324] A "terminal" is a device such as a computer, smartphone, tablet, or head-mounted display that a user uses as an interface.

[2325] The "server" is a central processing unit that manages user login information, learning data, and emotional data, and generates and transmits profile data, learning curriculum, feedback, etc.

[2326] An "emotion engine" is software or a system that analyzes a user's voice and facial expression data to identify their emotional state.

[2327] "Generative AI" refers to artificial intelligence that uses machine learning and natural language processing to generate and analyze data.

[2328] "Login information" refers to identification information, such as a username and password, required for a user to access a system.

[2329] "Profile data" refers to data including the user's personal information, learning history, setting information, and the like.

[2330] A "learning curriculum" is a customized learning content and schedule based on a user's chosen field or topic.

[2331] A "Virtual Project" is a virtual learning or work assignment or project that a user can experience within the Metaverse.

[2332] A "virtual environment" is a three-dimensional virtual space constructed using metaverse technology in which users can have virtual experiences.

[2333] "Real-time feedback" is a function that responds immediately to the user's learning progress and emotional state, providing appropriate advice and curriculum adjustments.

[2334] A "smartphone" is a type of mobile device, an advanced portable terminal that can be used to install and use communication functions and a variety of applications.

[2335] A "head-mounted display" is a display device worn on a user's head and is used to provide visual information.

[2336] The system of the present invention combines generative AI, an emotion engine, and metaverse technology to provide users with a customized learning experience. This system is composed of three main components: a server, a terminal, and a user, and is realized through the interaction of these components. A specific embodiment of this system is described below.

[2337] First, the user accesses the login screen using a device (smartphone, tablet, head-mounted display, etc.) and enters their username and password. The device encrypts this login information and sends it to the server. The server compares the received login information with user data in a database and performs authentication. If authentication is successful, the server obtains the profile data, encrypts it, and sends it back to the device. The device decrypts the received profile data and displays it on the user's screen.

[2338] Next, the user selects the field or topic they want to study on their device. The device encrypts the selection and sends it to the server. The server uses generative AI to generate a custom learning curriculum based on the user's selection. The generated learning curriculum is sent from the server to the device, where it is decrypted and displayed to the user.

[2339] As the user progresses with their learning, the device uses cameras and sensors to capture changes in the user's emotions (happiness, confusion, anger, etc.) in real time. The captured data is sent to an emotion engine, which analyzes the user's emotional state. Based on the analyzed data, the server adjusts the necessary parts of the learning curriculum and provides feedback to the user.

[2340] When a user selects a virtual project, the information is sent from the device to the server, which then uses metaverse technology to configure the virtual environment and load the selected project content. The server then sends the environment configuration information to the device, which then connects the user to the virtual environment.

[2341] To achieve these functions, the following hardware and software are used:

[2342] Hardware: smartphones, tablets, head-mounted displays, cameras, sensors

[2343] Software: Python, OpenCV (image processing library), transformers (natural language processing library), Requests (library for API requests)

[2344] For example, if a user selects that they want to learn data science, the device sends the selection information to the server. The server uses generative AI to generate a custom curriculum, which the device then displays to the user. Furthermore, if the user shows a confused expression during the project, the device captures facial expression data through the camera, and the emotion engine detects this as "confusion." Based on this information, the server can provide additional explanations and hints.

[2345] Example prompts to input to a generative AI model:

[2346] A user is confused about: [machine learning algorithm parameter tuning]

[2347] Example of feedback generated based on this prompt:

[2348] "In this step, it's important to understand the role of each parameter. Parameter A adjusts the scale of the data, and parameter B controls the learning rate. For example..."

[2349] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2350] Step 1:

[2351] The user accesses the login screen using a terminal and enters their username and password.

[2352] Input: Username, Password

[2353] Output: Encrypted login information

[2354] Specific behavior:

[2355] The user enters their username and password into the login screen of their device, which then encrypts the information using the AES encryption algorithm and sends it to the server.

[2356] Step 2:

[2357] The server receives the encrypted login information, decrypts it, and authenticates it by checking it against user data in a database.

[2358] Input: Encrypted login information

[2359] Output: Authentication results, profile data

[2360] Specific behavior:

[2361] The server decrypts the received login information using the AES decryption algorithm and compares it with the user data in its database. If authentication is successful, it retrieves the profile data, encrypts it, and sends it back to the device.

[2362] Step 3:

[2363] The terminal decrypts the profile data received from the server and displays it on the user screen.

[2364] Input: Encrypted profile data

[2365] Output: Display profile information

[2366] Specific behavior:

[2367] The device will then decrypt the received encrypted profile data using the AES decryption algorithm and display it on the user's screen, allowing the user to view their profile information.

[2368] Step 4:

[2369] The user selects the field or topic they wish to study on their device and sends the selected information to the server.

[2370] Input: Select the field or topic you want to study

[2371] Output: Encrypted selection information

[2372] Specific behavior:

[2373] Users select the fields and topics they want to learn on their device, which then encrypts this selection using the AES encryption algorithm and sends it to the server.

[2374] Step 5:

[2375] The server receives the encrypted selection information, decrypts it, and uses generative AI to generate a custom learning curriculum.

[2376] Input: Encrypted selection information

[2377] Output: Custom learning curriculum

[2378] Specific behavior:

[2379] The server decrypts the received selection information using the AES decryption algorithm and inputs it into the generative AI model, which then generates a custom learning curriculum based on the user's selection, encrypts it, and sends it to the device.

[2380] Step 6:

[2381] The terminal receives the encrypted learning curriculum, decrypts it, and displays it to the user.

[2382] Input: Encrypted learning curriculum

[2383] Output: Display of learning curriculum

[2384] Specific behavior:

[2385] The terminal decrypts the received encrypted curriculum using the AES decryption algorithm and displays it on the user's screen, allowing the user to view the customized curriculum.

[2386] Step 7:

[2387] As the user progresses with their learning, the device uses cameras and sensors to capture facial and voice data in real time and sends it to the emotion engine.

[2388] Input: User's facial expression data, voice data

[2389] Output: Emotion analysis results

[2390] Specific behavior:

[2391] The device uses cameras and sensors to capture the user's facial expressions and voice in real time and transmits the data to the emotion engine, which analyzes this data to detect the user's emotional state.

[2392] Step 8:

[2393] The server adjusts the necessary parts of the learning curriculum based on the emotional data received from the emotion engine, generates real-time feedback, and sends it to the terminal.

[2394] Input: Sentiment analysis results, learning progress data

[2395] Output: Curriculum adjustments, real-time feedback

[2396] Specific behavior:

[2397] The server generates customized feedback based on the emotion analysis results and learning progress data. The feedback content is created using a generative AI model, encrypted, and sent to the device.

[2398] Step 9:

[2399] The terminal receives the encrypted feedback, decrypts it and displays it on the user's screen.

[2400] Input: Encrypted feedback

[2401] Output: Display of feedback content

[2402] Specific behavior:

[2403] The device uses the AES decryption algorithm to decrypt the received encrypted feedback and displays it on the user's screen, where the user can see specific advice and tips from the generated AI model.

[2404] Step 10:

[2405] The user selects a virtual project and the terminal transmits the information to the server.

[2406] Input: Virtual project selection information

[2407] Output: Encrypted selection information

[2408] Specific behavior:

[2409] The user selects a virtual project on the terminal, which encrypts the selection information using the AES encryption algorithm and sends it to the server.

[2410] Step 11:

[2411] The server receives the encrypted selection information and uses metaverse technology to set up a virtual environment and load the project content.

[2412] Input: Encrypted selection information

[2413] Output: Virtual environment configuration information

[2414] Specific behavior:

[2415] The server decrypts the received selection information using the AES decryption algorithm, sets up a virtual environment using metaverse technology, loads the selected project content, and sends the configuration information to the terminal.

[2416] Step 12:

[2417] The terminal receives the setting information of the virtual environment and connects the user to the virtual environment.

[2418] Input: Virtual environment configuration information

[2419] Output: Virtual environment connected

[2420] Specific behavior:

[2421] The device decodes the received virtual environment configuration information and connects the user to the virtual environment, allowing the user to participate in virtual projects and progress in their learning.

[2422] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2424] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2425] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2426] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2427] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2428] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2429] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2430] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2431] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an exa...

Claims

1. a means for a user to enter login information; A means for transmitting the input login information to a server by the terminal; a means by which the server authenticates the login information and obtains the user's profile data; a means for the server to transmit profile data to the terminal upon successful authentication; a means for the device to display profile information; A way for users to select the areas and topics they want to learn about; means for the terminal to transmit the selection information to the server; A means for the server to generate a learning curriculum based on the selection information; A means for the server to transmit the generated learning curriculum to the terminal; a means by which the device displays the curriculum; a means for a user to select a virtual project; means for the terminal to transmit project selection information to the server; a means for the server to set up a virtual environment and load the selected project; A means for the server to transmit environmental data to the terminal; a means by which the terminal connects the user to the virtual environment; A server monitors the user's progress and provides real-time feedback using generative AI. a means for the device to display feedback; A system including:

2. 2. The system according to claim 1, further comprising means for the server to generate customized advice based on the user's learning progress and for the terminal to display the advice to the user.

3. 2. The system of claim 1, wherein the server uses the metaverse to provide a project experience within a virtual reality environment, and the terminal includes means for connecting the user to the virtual environment.

Images