System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving a question input by a user from a terminal; means for sending the question to a natural language processing model to obtain an answer; means for sending the answer to the terminal to display to the user; a server comprising the natural language processing model for generating the answer to the question; a virtual reality space simulator connected to the server; and the terminal comprising an interface for operating the simulator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional learning systems have made it difficult for working adults to efficiently and practically acquire new skills and techniques. Time and environmental constraints often prevent independent reskilling. In particular, it has been difficult to provide an interactive learning environment or practical project experience, resulting in low learning effectiveness. It has also been difficult to realize customized learning tailored to the needs of individual learners. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving a question entered by a user from a terminal; means for transmitting the question to a natural language processing model to obtain an answer; means for transmitting the answer to the terminal and displaying it to the user; a server including a natural language processing model that generates an answer to the question; a virtual reality space simulation means connected to the server; and a terminal including an interface for operating the simulation means. The system stores multiple virtual reality space simulation projects in a database and provides users with randomly selected project information, thereby enabling working professionals to efficiently advance their reskilling. Furthermore, the system can improve learning effectiveness by analyzing users' learning history and performance data and providing individually optimized learning content.

[0006] A "question" is information that a user inputs via a terminal and requests an answer from a natural language processing model.

[0007] "Means for receiving" refers to the functions and mechanisms for receiving questions and requests from the terminal.

[0008] A "natural language processing model" is an artificial intelligence technology for generating answers to user questions.

[0009] The "means for obtaining an answer" refers to a function or mechanism for receiving an answer generated from a natural language processing model.

[0010] "Terminal" means a device operated by a user and equipped with an interface for inputting and sending questions and requests.

[0011] A "server" is a computer system that includes a natural language processing model and a virtual reality space simulation means and provides these functions.

[0012] A "virtual reality space simulation means" is an environment that uses virtual reality technology to allow a user to experience a virtual project.

[0013] The "interface" is a user interface that allows a user to operate a terminal and communicate with a server.

[0014] A "database" is an information storage system for storing information on multiple projects and providing it as needed.

[0015] "Study history" is data that records a user's learning activities and progress information.

[0016] "Performance Data" means data about a user's performance or achievement in a virtual reality simulation or other learning activity. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[0039] System configuration

[0040] 1. Terminal

[0041] The terminal has an input device and a user interface for users to input questions and project simulation requests, and has the ability to send requests to a server via the Internet.

[0042] 2. Server

[0043] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. Furthermore, the server manages a database, storing and analyzing learning history and performance data.

[0044] 3. Virtual Reality Space Simulation Method

[0045] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[0046] Program processing explanation

[0047] ChatGPT part

[0048] 1. Receiving and processing inquiries

[0049] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[0050] The server receives this request and extracts the question.

[0051] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[0052] The server receives the response generated by ChatGPT and sends it to the device.

[0053] The terminal receives the response and displays it to the user.

[0054] Metaverse Project Part

[0055] 1. Request and provide project simulation

[0056] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[0057] The server receives the request and extracts random or specified project information from the database.

[0058] The server sends details of the selected project to the terminal.

[0059] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[0060] Specific examples

[0061] Example 1: A user asks, "I want to know about data analysis in Python."

[0062] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0063] 2. The device sends this question as a POST request to the server's API endpoint.

[0064] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0065] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0066] 5. The server sends this response to the terminal and displays it to the user.

[0067] Example 2: User requests a new project

[0068] 1. The user clicks the project simulation request button on the terminal.

[0069] 2. The device sends a GET request to the server.

[0070] 3. The server receives the request and randomly selects a project from the database.

[0071] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[0072] 5. The device displays this information to the user and provides a project simulation in a virtual reality space.

[0073] As described above, the system of the present invention provides users with an interactive learning environment and practical project experience, thereby enhancing learning effectiveness and creating an environment where working professionals can efficiently acquire new skills and techniques.

[0074] The processing flow will be explained below.

[0075] ChatGPT part

[0076] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[0077] Step 1:

[0078] The user types a question into the terminal and clicks the send button.

[0079] Step 2:

[0080] The terminal sends the entered question to the server as a POST request.

[0081] Step 3:

[0082] The server receives the request and extracts the question.

[0083] Step 4:

[0084] The server sends the extracted questions to the natural language processing model (ChatGPT).

[0085] Step 5:

[0086] ChatGPT generates a response to the question and sends it back to the server.

[0087] Step 6:

[0088] The server receives the response sent back from ChatGPT, converts it to JSON format, and sends it back to the device.

[0089] Step 7:

[0090] The terminal displays the response received from the server to the user.

[0091] Metaverse Project Part

[0092] Example 2: Processing steps for a project simulation request

[0093] Step 1:

[0094] The user clicks the Request Project Simulation button on the terminal.

[0095] Step 2:

[0096] The terminal sends a project simulation request to the server as a GET request.

[0097] Step 3:

[0098] The server receives the request and randomly selects a project from the database.

[0099] Step 4:

[0100] The server extracts the details of the selected project.

[0101] Step 5:

[0102] The server converts the project details into JSON format and sends it back to the device.

[0103] Step 6:

[0104] The terminal displays detailed information about the project received from the server to the user.

[0105] Step 7:

[0106] The user accesses the virtual reality space simulation means via a terminal and experiences the project.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] In today's educational environment, learners are expected to efficiently acquire new skills and knowledge. However, existing educational systems struggle to provide personalized instruction and practical experiences that are tailored to learners' specific needs. Furthermore, traditional methods struggle to support learning through real-time dialogue or simulated learning using virtual reality. This hinders the improvement of learning outcomes and the acquisition of practical skills.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for receiving data input by a user from an input / output device, means for transmitting the data to a natural language processing algorithm to obtain a response, and means for transmitting the response to the input / output device and displaying it to the user. This allows users to learn through question and answer sessions using a natural language processing algorithm in real time. The server also includes means for storing multiple virtual space simulation projects in a storage device and providing randomly selected project information to the user, enabling learning through practical project experience. Furthermore, the server includes means for analyzing a user's learning history and performance data and providing individually optimized learning content, enabling personalized education and significantly improving learning effectiveness.

[0112] 1. An "input / output device" is a device that allows a user to input data and display information received from a server.

[0113] 2. "Natural language processing algorithms" are machine learning models for understanding text data and generating appropriate responses.

[0114] 3. "Information Processing Device" means a computer system for receiving and processing data and generating and transmitting responses.

[0115] 4. "Virtual space simulation device" means a device that uses virtual reality technology to provide users with a simulation experience.

[0116] 5. "Interface" means the means by which a user operates a virtual reality simulation device.

[0117] 6. "Storage device" means a hardware device for storing data and retrieving that data as needed.

[0118] 7. "Project Information" means the specific scenarios and data used in the virtual world simulation.

[0119] 8. "Learning History" means a record of a user's previous learning activities.

[0120] 9. "Performance Data" means information that indicates a User's learning results and progress.

[0121] A specific system and its operation that are the subject of the present invention will be described below. The system is an interactive educational platform where users learn through input / output devices, and includes an information processing device and a virtual space simulation device.

[0122] System configuration

[0123] 1. Input / Output Devices

[0124] An input / output device is a device that allows users to input data and display information. Examples include personal computers, tablets, and smartphones. An input / output device has the ability to send requests to an information processing device via the Internet.

[0125] 2. Information Processing Device

[0126] It includes a natural language processing algorithm and generates responses to user questions. Specifically, generative AI models such as ChatGPT are used. The information processing device is connected to the virtual space simulation device and manages the virtual project simulation. The information processing device also manages the storage device, accumulating and analyzing learning history and performance data.

[0127] 3. Virtual space simulation device

[0128] It uses virtual reality technology to provide an environment in which users can simulate projects. Specifically, a VR headset is used. Multiple virtual projects are stored in a storage device and are selected randomly or provided according to the user's preferences.

[0129] Program processing explanation

[0130] Question-answering part of natural language processing

[0131] 1. Enter and submit your question

[0132] The user enters a question into the input / output device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and submits it.

[0133] 2. Receiving and processing inquiries

[0134] The input / output device transmits the input question to the information processing device as a POST request.

[0135] An information processing device receives the request, extracts the question content, sends it to a natural language processing algorithm, and generates a response.

[0136] 3. Receiving and Displaying Responses

[0137] The information processing device sends the generated response to the input / output device, which receives the response and displays it to the user. For example, it displays a response such as, "The pandas library is often used for data analysis in Python..."

[0138] Requesting and providing virtual space simulations

[0139] 1. Request and submit a simulation

[0140] A user clicks a project simulation request button on an input / output device. For example, the user thinks, "I want to simulate a new project," and presses the corresponding button.

[0141] 2. Obtaining project information

[0142] The input / output device transmits a request to the information processing device as a GET request.

[0143] The information processing device receives the request and extracts random or specified project information from the storage device, for example, selecting the "Virtual Store Development" project.

[0144] 3. Sending and viewing project information

[0145] The information processing device transmits details of the selected project to the input / output device.

[0146] The input / output device displays the received project details to the user, and provides an environment in which the project can be executed through a virtual space simulation device. For example, a development scenario for a virtual store is displayed on a VR headset.

[0147] Specific examples

[0148] Example 1: A user asks, "I want to know about data analysis in Python."

[0149] 1. The user enters "I want to know about data analysis using Python" into the input form on the input / output device and clicks the submit button.

[0150] 2. The input / output device sends this question as a POST request to the API endpoint of the information processing device.

[0151] 3. An information processor receives the question and sends it to a natural language processing algorithm to generate a response.

[0152] 4. The information processing device receives a response from ChatGPT such as, "The pandas library is often used for data analysis in Python..."

[0153] 5. The information processing device sends this response to the input / output device and displays it to the user.

[0154] Example 2: User requests a new project

[0155] 1. The user clicks the project simulation request button on the I / O device.

[0156] 2. The input / output device sends the data as a GET request to the information processing device.

[0157] 3. The information processing device receives the request and randomly selects a project from the storage device.

[0158] 4. The information processing device selects "Development of a virtual store" and sends the detailed information to the input / output device.

[0159] 5. The input / output device displays this information to the user, providing a project simulation in the virtual space simulation device.

[0160] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, enhancing learning effectiveness and enabling them to efficiently acquire new skills and techniques.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] The user enters a question into the input form of the I / O device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and then presses the "Submit" button. The input data is stored in the memory of the I / O device and used for subsequent processing.

[0164] Step 2:

[0165] The device sends the entered question to the server as a POST request. Specifically, it sends the entered text data as an HTTP request to the API endpoint. The output is the HTTP request received by the server.

[0166] Step 3:

[0167] The server receives the request and extracts the question. It extracts pure text data from the HTTP request body. This process analyzes the question from the received request and converts the request content into a format suitable for natural language processing algorithms. The output is the extracted text data.

[0168] Step 4:

[0169] The server sends the extracted question to a natural language processing algorithm (ChatGPT) to generate a response. Specifically, the extracted question is passed to ChatGPT as a prompt sentence, which is then processed using natural language processing. The generated response data is the output.

[0170] Step 5:

[0171] The server receives the generated response and sends a reply to the input / output device. Specifically, it converts the response from the natural language processing algorithm into an HTTP response format and sends it to the input / output device. The output is an HTTP response to the input / output device.

[0172] Step 6:

[0173] The terminal receives the response and displays it to the user. Specifically, it analyzes the received HTTP response data and renders the text data on the screen. It must be displayed in a format that is easy for the user to read. The output is text that is displayed on the user's screen.

[0174] Step 7:

[0175] A user clicks a project simulation request button on an input / output device. For example, if the user thinks, "I want to simulate a new project," he / she presses the corresponding button. The input data is the user's request.

[0176] Step 8:

[0177] The terminal sends a request to the server as a GET request. Specifically, it sends the request data generated based on the user's action as an HTTP GET request. The output is an HTTP request received by the server.

[0178] Step 9:

[0179] The server receives the request and extracts random or specified project information from storage. It performs a database query to retrieve the appropriate project information. For example, select "Virtual Store Development." The output is the extracted project information.

[0180] Step 10:

[0181] The server sends the details of the selected project to the terminal. Specifically, the server forms the project information as an HTTP response and sends it to the input / output device. The output is an HTTP response to the input / output device.

[0182] Step 11:

[0183] The terminal displays the received project details to the user, providing an environment in which the project can be executed through a virtual space simulation device. Specifically, the received details are analyzed and displayed on the screen. The terminal also launches a virtual space simulation to create an environment in which the project can be simulated. For example, a development scenario for a virtual store can be displayed on a VR headset. The output is a simulation environment that the user can directly experience.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] Currently, when users learn, it is difficult for them to acquire specific knowledge or acquire practical operational skills through experience. Furthermore, there is a lack of systems that provide effective feedback and individually optimized learning content quickly. In particular, there is a need for a platform that integrates operational simulation functions in a virtual space to provide a learning environment that is close to real-life experience, allowing users to efficiently acquire skills.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes means for providing an operation simulation in a virtual space based on questions and requests input by a user from a terminal, means for displaying the results of the virtual space simulation to the user, and means for analyzing the data and generating optimized feedback based on the user's learning history and performance. This allows the user to experience a simulation in a virtual space that is close to an actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[0189] "User" means an individual or corporation that uses this system.

[0190] A "terminal" is a device, such as a smartphone, tablet, or computer, through which a user inputs a question or request.

[0191] "Question" refers to a learning-related inquiry or doubt that a user inputs via a terminal.

[0192] A "natural language processing model" is a type of artificial intelligence that generates appropriate answers to user questions or input.

[0193] An "answer" is a response to a user's question generated by a natural language processing model.

[0194] The "server" is a device that includes a natural language processing model and manages answers to user questions and simulation results in cooperation with the virtual reality space simulation means.

[0195] "Virtual reality space simulation" is a system that uses virtual reality technology to allow users to experience virtual environments and scenarios.

[0196] "Simulation means" refers to a device or software for executing a virtual reality space simulation.

[0197] "Interface" refers to a user interface that allows a user to operate the simulation means.

[0198] "Operation simulation" refers to a program that allows users to learn through experience how to operate and manage a store in a virtual space.

[0199] "Virtual space" refers to a virtual environment provided by virtual reality technology.

[0200] Analyzing "data" refers to the process of analyzing collected user learning history and performance data to find trends and patterns.

[0201] "Learning history" refers to historical data of the learning activities and simulations that a user has performed up to now.

[0202] "Performance Data" refers to data on the results or outcomes achieved by a user through simulation or learning.

[0203] "Optimized feedback" refers to specific advice and suggestions for next learning that are individually tailored based on the user's learning history and performance.

[0204] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[0205] System configuration

[0206] 1. Terminal

[0207] The device includes an input device and user interface for users to enter questions and project simulation requests. The terminal has the ability to send requests to the server via the Internet. Examples of such devices include smartphones, tablets, and computers.

[0208] 2. Server

[0209] It includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. In addition, the server manages a database to store and analyze learning history and performance data. Specific examples that can be used include Amazon Web Services (AWS) and Google Cloud Platform.

[0210] 3. Virtual Reality Space Simulation Method

[0211] It uses virtual reality technology to provide an environment where users can simulate projects. It stores multiple virtual projects, which are selected randomly or provided according to the user's preferences. It provides 3D simulations using Unity and Unreal Engine.

[0212] Program processing explanation

[0213] Receiving and processing questions

[0214] When a user enters a question into the device and clicks the send button, the server receives the question from the device. The server extracts the content of the question and sends it to a natural language processing model (for example, OpenAI's GPT-3) to generate a response. The generated response is sent from the server to the device and displayed to the user. As a concrete example, a user enters, "What is the best way to manage inventory?"

[0215] Request and provide project simulations

[0216] When a user clicks the project simulation request button on their device, the server receives the request and extracts random or specified project information from the database. The extracted project details are sent from the server to the device and displayed to the user. The user can then perform a project simulation in a virtual reality space. A specific example is when the user selects a "seasonal product campaign" and simulates product placement and advertising in a virtual space.

[0217] Data analysis and feedback

[0218] The server collects and analyzes the user's learning history and performance data, and based on this data, provides optimized feedback and suggests next learning content, allowing users to efficiently acquire new skills and techniques.

[0219] Examples of specific examples and prompts

[0220] For example, if a user asks, "What is the best way to manage inventory?", an example prompt sentence might be:

[0221] Example prompt sentence:

[0222] What is the best way to manage inventory?

[0223] This system allows users to experience simulations in a virtual space that closely resembles the actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The user inputs a question or a project simulation request into the terminal and clicks the send button. The terminal then sends the input question or request to the server.

[0227] Input: Questions or requests typed by the user into the device

[0228] Output: Request data from the terminal to the server

[0229] Step 2:

[0230] The server receives questions and requests sent from the device at a specified API endpoint, extracts the content of the question, and sends it to a natural language processing model to generate an answer.

[0231] Input: Request data sent from the terminal

[0232] Output: Question data sent to the natural language processing model and generated answer data

[0233] Step 3:

[0234] The server receives the response from the natural language processing model and extracts the answer, which it then sends back to the terminal and displays to the user.

[0235] Input: Answer data generated from a natural language processing model

[0236] Output: Answer data sent to the user's device

[0237] Step 4:

[0238] When a user clicks the request button of a project simulation, the terminal sends the request to the server as a GET request.

[0239] Input: Project Simulation Request

[0240] Output: GET request sent to the server

[0241] Step 5:

[0242] The server receives a request for a project simulation, extracts relevant project information from the database randomly or based on specified conditions, and transmits the extracted project information to the terminal.

[0243] Input: Project simulation request and project information from the database

[0244] Output: Project details sent to the device

[0245] Step 6:

[0246] The terminal presents the project information received from the server to the user and starts a virtual reality space simulation, and the user experiences the simulation using a virtual reality device.

[0247] Input: Project details sent from the server

[0248] Output: Project details and simulation environment presented to the user

[0249] Step 7:

[0250] During the simulation, the server collects user operation and performance data, which is then saved as a learning history and analyzed later.

[0251] Input: User operation data and performance data

[0252] Output: Learning history data and performance data

[0253] Step 8:

[0254] The server analyzes the collected learning history and performance data and generates personalized feedback, which is provided to the user and includes suggestions for next learning content.

[0255] Input: Learning history data and performance data

[0256] Output: Optimized feedback and learning suggestions provided to the user

[0257] Step 9:

[0258] The user can then view the optimized feedback and suggestions for the next learning activity on their device, and begin a new learning activity. The system repeats this process, helping the user improve their skills.

[0259] Input: Optimized feedback and learning suggestions

[0260] Output: User's new learning activity

[0261] Through these steps, users can efficiently acquire skills in an interactive learning environment.

[0262] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0263] As an embodiment of the present invention, a specific system incorporating an emotion engine and its operation will be described below. This system is an interactive educational platform where users learn through their terminals, and includes a server, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[0264] System configuration

[0265] 1. Terminal

[0266] The terminal is equipped with an input device and a user interface for users to input questions and project simulation requests. The terminal is equipped with an emotion engine and has the ability to recognize emotions from the user's facial expressions, tone of voice, etc.

[0267] 2. Server

[0268] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. The server also receives emotion information sent from the emotion engine and uses it to adjust answers and provide optimal learning content.

[0269] 3. Emotion Engine

[0270] It recognizes the user's emotions and extracts emotional information in real time from the user's facial expressions, tone of voice, etc. The emotion engine is installed on the device, and the data is sent to a server.

[0271] 4. Virtual Reality Space Simulation Method

[0272] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[0273] Program processing explanation

[0274] ChatGPT part

[0275] 1. Receiving and processing inquiries

[0276] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[0277] The server receives this request and extracts the question.

[0278] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[0279] The server receives the response generated by ChatGPT and sends it to the device.

[0280] The terminal receives the response and displays it to the user.

[0281] 2. Acquiring and using emotional information

[0282] While the user is typing a question, the device's emotion engine recognizes the user's emotions in real time.

[0283] The device sends emotional information to the server, which then adjusts the content and tone of the response based on this information.

[0284] The server sends the adjusted answer to the terminal and displays it to the user.

[0285] Metaverse Project Part

[0286] 1. Request and provide project simulation

[0287] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[0288] The server receives the request and randomly selects a project from the database.

[0289] The server extracts detailed information about the selected project and sends it to the device.

[0290] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[0291] 2. Use of emotional information

[0292] The emotion engine recognizes the user's emotions in real time during the project simulation.

[0293] Based on the emotional information, the server dynamically adjusts the difficulty of the project and the learning content provided.

[0294] For example, if the user is feeling stressed, the server will adjust the difficulty level by lowering it or adding explanations.

[0295] Specific examples

[0296] Example 1: A user asks, "I want to know about data analysis in Python."

[0297] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0298] 2. The device sends this question as a POST request to the server's API endpoint.

[0299] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0300] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0301] 5. The server sends this response to the terminal and displays it to the user.

[0302] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[0303] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[0304] Example 2: User requests a new project

[0305] 1. The user clicks the project simulation request button on the terminal.

[0306] 2. The device sends a GET request to the server.

[0307] 3. The server receives the request and randomly selects a project from the database.

[0308] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[0309] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[0310] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[0311] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[0312] As described above, the system of the present invention provides users with an integrated interactive learning environment and practical project experience, and further optimizes learning effectiveness by utilizing users' emotional information. This creates an environment in which working professionals can efficiently acquire new skills and techniques.

[0313] The processing flow will be explained below.

[0314] ChatGPT part

[0315] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[0316] Step 1:

[0317] The user types a question into the terminal, saying "I want to know about data analysis using Python," and clicks the submit button.

[0318] Step 2:

[0319] The device sends the entered question as a POST request to the server's API endpoint.

[0320] Step 3:

[0321] The server receives the POST request and extracts the question from the request data.

[0322] Step 4:

[0323] The server sends the extracted questions to a natural language processing model (ChatGPT).

[0324] Step 5:

[0325] ChatGPT generates answers to questions and sends the answers back to the server.

[0326] Step 6:

[0327] The server receives the response sent back from ChatGPT, formats the response appropriately, and sends it to the device.

[0328] Step 7:

[0329] The terminal displays the received response to the user.

[0330] Use of emotional information

[0331] Acquiring and using emotional information with an emotion engine

[0332] Step 1:

[0333] While the user types their question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[0334] Step 2:

[0335] The emotion information recognized by the device's emotion engine is sent to the server.

[0336] Step 3:

[0337] The server receives the emotional information and adjusts the content and tone of the answers to questions.

[0338] Step 4:

[0339] The server sends the adjusted answer to the terminal and displays it to the user.

[0340] Metaverse Project Part

[0341] Example 2: Processing steps for a project simulation request

[0342] Step 1:

[0343] The user clicks the Request Project Simulation button on the terminal.

[0344] Step 2:

[0345] The terminal sends a project simulation request to the server as a GET request.

[0346] Step 3:

[0347] The server receives a GET request and randomly selects multiple project information from the database.

[0348] Step 4:

[0349] The server extracts detailed information about the selected project.

[0350] Step 5:

[0351] The server converts the project details into JSON format and sends it to the terminal.

[0352] Step 6:

[0353] The terminal displays the received project details information to the user.

[0354] Step 7:

[0355] The user accesses the virtual reality space simulation means via the terminal and experiences the selected project.

[0356] Use of emotional information

[0357] Using emotional information during project simulation with an emotion engine

[0358] Step 1:

[0359] During the project simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[0360] Step 2:

[0361] The emotion information recognized by the device's emotion engine is sent to the server.

[0362] Step 3:

[0363] The server receives emotional information and dynamically adjusts the difficulty of the project and the learning content provided.

[0364] Step 4:

[0365] The server sends the adjusted learning content and support information to the terminal and displays it to the user.

[0366] Example 2

[0367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] Conventional interactive educational systems have struggled to provide learning content that takes into account the user's emotional state. This can lead to problems such as reduced learning efficiency and increased stress. Furthermore, even with project simulations using virtual reality technology, optimal learning outcomes cannot be achieved because the content is not dynamically adjusted based on the user's progress or emotions.

[0369] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means including an emotion engine that recognizes the user's emotions while the user is entering or viewing a question and extracts emotional information in real time, a means for transmitting the emotional information to the server and adjusting the content and tone of the answer, and a means for transmitting the adjusted answer to the terminal and displaying it. This makes it possible to provide learning content tailored to the user's emotional state. In addition, by storing multiple virtual reality space simulation projects in a database, recognizing the user's emotions in real time during the simulation, and dynamically adjusting the content and difficulty of the simulation based on the emotional information, it is possible to provide the user with an optimized learning environment.

[0370] A "terminal" is a device that has an interface and input devices through which a user can enter questions and project simulation requests.

[0371] The "server" is a central device that contains the natural language processing model, generates answers based on user questions, and adjusts learning based on emotional and other data.

[0372] A "natural language processing model" is a software system that includes algorithms that analyze natural language questions entered by users and generate appropriate answers to those questions.

[0373] An "emotion engine" is a device that recognizes a user's facial expressions and tone of voice in real time and extracts emotional information.

[0374] A "virtual reality space simulation means" is a means and device that uses virtual reality technology to provide an environment in which a user can experience a virtual project.

[0375] The "means for adjusting the content and tone of the answer" refers to a mechanism or algorithm for adjusting the answer generated by the natural language processing model based on the emotional information sent from the emotion engine in accordance with the user's emotional state.

[0376] A "project simulation request" is request information that a user sends from a terminal to the system when the user desires to perform a project simulation in a virtual reality space.

[0377] "Study history" is data that records the user's past learning activities and their progress.

[0378] "Performance data" is data that indicates the results and efficiency of a user's learning activities.

[0379] A "database" is a storage device that stores multiple virtual reality space simulation projects and related information and can be accessed as needed.

[0380] The present invention is a system in which a user inputs a question via a terminal, a natural language processing model generates an answer based on the input, and further provides optimized learning content using the user's emotional information. The system includes a server, a terminal, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[0381] System configuration

[0382] Terminal

[0383] The terminal is equipped with an interface and an input device for users to input questions and project simulation requests. The terminal is equipped with an emotion engine that recognizes the user's facial expressions and tone of voice in real time and extracts emotional information.

[0384] server

[0385] The server receives questions entered by users from their devices and sends them to a natural language processing model to generate answers. The generated answers are then sent to the devices and displayed to the users. The server also receives emotional information sent from the emotion engine and adjusts the content and tone of the answers based on this information. The server is also connected to a virtual reality space simulation means and manages the simulation of the virtual project.

[0386] Emotion Engine

[0387] The emotion engine is installed on the device and extracts emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to the server, which then adjusts the content and tone of the answer based on this information.

[0388] Virtual reality space simulation tool

[0389] The virtual reality space simulation means provides an environment in which a user can simulate a project using virtual reality technology. A plurality of virtual projects are stored in a database and are selected randomly or provided according to the user's request.

[0390] Specific examples

[0391] Specific example of question processing

[0392] If a user asks, "I want to know about data analysis using Python," the specific process is as follows:

[0393] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0394] 2. The device sends this question as a POST request to the server's API endpoint.

[0395] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0396] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0397] 5. The server sends this response to the terminal and displays it to the user.

[0398] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[0399] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[0400] Example prompt sentence:

[0401] User: "I want to learn about data analysis using Python."

[0402] Model: "The pandas library is often used for data analysis in Python. Other popular libraries include numpy and matplotlib."

[0403] Project Simulation Examples

[0404] As an example of a user requesting a new project, the "Virtual Store Development" project is selected.

[0405] 1. The user clicks the project simulation request button on the terminal.

[0406] 2. The device sends a GET request to the server.

[0407] 3. The server receives the request and randomly selects a project from the database.

[0408] 4. The server selects "Development of a virtual store" and sends the details to the terminal.

[0409] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[0410] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[0411] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[0412] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, and utilizes users' emotional information to optimize learning effectiveness, creating an environment where users can efficiently acquire new skills and techniques.

[0413] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0414] Step 1: User enters question into terminal

[0415] The user enters a question into the input form on the terminal and clicks the submit button. The input text "I want to know about data analysis using Python" is sent to the terminal.

[0416] Step 2: The device sends the question to the server

[0417] The device sends the user's question as a POST request to the server's specified API endpoint. This request includes the question entered by the user. The server then receives the request.

[0418] Step 3: The server sends the question to the natural language processing model

[0419] The server sends the received question to a natural language processing model (e.g., ChatGPT) to generate an answer. The server inputs the question into the model and receives the generated answer as output. Specifically, the answer generated is "The pandas library is often used for data analysis in Python..."

[0420] Step 4: The server sends the generated answer to the device

[0421] The server sends the generated answer, which includes text obtained from the natural language processing model, to the device, which then receives the answer.

[0422] Step 5: The device displays the answer to the user

[0423] The device displays the answer it receives to the user. This display is the text sent from the server and displayed in the interface. Specifically, the answer displayed is, "The pandas library is often used for data analysis in Python..."

[0424] Step 6: The device recognizes the user's emotions

[0425] While the user is typing a question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is surprised."

[0426] Step 7: The device sends the emotion information to the server.

[0427] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[0428] Step 8: The server adjusts the answer based on the emotional information

[0429] The server adjusts the content and tone of the generated answer based on the emotional information received. In this process, the emotional information is used as input to modify the content and expression of the answer text. For example, if the user is feeling stressed, the server may adjust the answer by adding a gentler tone.

[0430] Step 9: The server sends the adjusted answer to the device.

[0431] The server sends the adjusted answer to the device, which includes the text modified based on the sentiment, and the device receives the adjusted answer.

[0432] Step 10: The device displays the adjusted answer to the user

[0433] The terminal displays the adjusted answer to the user. This display shows the corrected text sent from the server in the interface. Specifically, the answer is displayed in a friendly tone: "The pandas library is often used for data analysis in Python. Try it yourself."

[0434] Step 11: User requests project simulation

[0435] The user clicks the project simulation request button on the terminal. As an input, a request for project simulation is sent to the terminal.

[0436] Step 12: The device sends a request to the server

[0437] The terminal sends the user's project simulation request to the server as a GET request, which includes a request for the project simulation. The server then receives the request.

[0438] Step 13: The server randomly selects a project and extracts its details

[0439] Based on the request received by the server, a virtual project is randomly selected from the database. The server extracts the project details and generates them as output. Specifically, the project "Development of a Virtual Store" is selected.

[0440] Step 14: The server sends the details to the device

[0441] The server transmits the extracted project details to the terminal, including the project description and procedures, and the terminal receives the details.

[0442] Step 15: The terminal displays detailed information to the user and provides a simulation environment

[0443] The terminal displays the received project details to the user, providing a simulation environment in virtual reality space. This display shows the project information sent from the server on the interface. Specifically, details of the "virtual store development" are displayed, and the user experiences the project in virtual reality space.

[0444] Step 16: The device recognizes the user's emotions during the simulation.

[0445] During the simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is excited."

[0446] Step 17: The device sends the emotion information to the server.

[0447] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[0448] Step 18: The server adjusts the content and difficulty of the simulation based on the emotional information.

[0449] The content and difficulty of the simulation are dynamically adjusted based on the emotional information received by the server. In this process, the content of the simulation is dynamically changed using the emotional information as input. For example, if the user feels the simulation is difficult, adjustments such as simplifying the next step are made.

[0450] Step 19: The server sends the adjustments to the device

[0451] The server transmits the adjusted simulation content to the terminal, which includes the simulation data modified based on the emotion. The adjusted content is then received by the terminal.

[0452] Step 20: The device reflects the adjustments in the simulation environment and displays them to the user.

[0453] The terminal reflects the adjusted simulation content in the virtual reality space simulation environment and displays it to the user. This display reflects the corrected simulation data sent from the server on the interface. Specifically, the "Virtual Store Development" simulation with the adjusted difficulty level is displayed.

[0454] Through these processing steps, users can enjoy an interactive learning environment that combines hands-on project experiences. Furthermore, by utilizing emotional information, an environment that optimizes learning effectiveness is provided.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] Conventional educational platforms provide uniform learning content without considering the user's emotions, making it difficult to maximize individual learning efficiency and comprehension. Furthermore, because they do not take into account the user's stress or fatigue, it can be difficult to continue learning. Furthermore, when providing a simulation environment using virtual reality technology, adjustments are not made based on the user's current emotional state, making it difficult to provide an optimal learning experience.

[0458] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question input by a user from a terminal, means for transmitting the question to a natural language processing model to obtain an answer, and means for transmitting the answer to the terminal and displaying it to the user. This realizes a system including emotion recognition means for recognizing a user's emotions in real time and transmitting emotional information to the server, and means for dynamically adjusting answers and learning content based on the emotional information. This makes it possible to provide optimal and personalized learning content and a simulation environment according to the user's current emotional state.

[0459] A "terminal" is a device equipped with an input device and a user interface for a user to input questions and project simulation requests and obtain emotion information.

[0460] A "natural language processing model" is an algorithm or system that generates answers to questions entered by users, and specifically refers to a model that understands natural language and generates responses.

[0461] A "server" is a central computer that transmits, receives, and processes data via a network, and in the present invention includes a natural language processing model, a virtual reality space simulation means, and a device that processes emotional information.

[0462] "Virtual reality space simulation means" refers to technology or devices that provide an environment in which users can simulate projects using virtual reality technology.

[0463] "Emotion recognition means" refers to technology or devices that extract emotional information in real time from a user's facial expressions, tone of voice, etc., and is responsible for transmitting this information to a server.

[0464] An "interface" refers to a connection means or operating device that allows a user to operate a simulation means, and is designed to allow intuitive and efficient operation.

[0465] "Learning History and Performance Data" refers to data that records a user's past learning activities and the results of those activities, and is used to provide individually optimized learning content.

[0466] "Personalized learning content" refers to learning materials and assignments that are dynamically tailored based on a user's learning history and emotional state.

[0467] "Project simulation" refers to a virtual project that simulates real-world environments and scenarios in a virtual reality space, allowing users to experience and learn from them.

[0468] A specific embodiment for carrying out the present invention includes the following system and its operation, which enables a user to receive personalized learning content and virtual reality project simulations while having their emotions recognized in real time.

[0469] System configuration

[0470] 1. Terminal

[0471] The device is equipped with an input device and a user interface for users to input questions and project simulation requests. The device is also equipped with emotion recognition means to extract emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to a server and used to dynamically adjust the learning content.

[0472] 2. Server

[0473] It includes a natural language processing model (e.g., ChatGPT) and generates answers to user questions. The server receives emotional information sent from the emotion recognition means and dynamically adjusts the answers and learning content. It is also connected to the virtual reality space simulation means and manages the simulation of the virtual project.

[0474] 3. Emotion recognition means

[0475] Emotional information is extracted in real time from the user's facial expressions, tone of voice, etc. This is installed on the device, and the data is sent to the server. Based on the emotional information, the server adjusts the answers and learning content.

[0476] 4. Virtual Reality Space Simulation Method

[0477] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored in a database, and these are provided randomly or according to the user's preferences.

[0478] Program processing explanation

[0479] The server receives the user's emotional information in real time and dynamically optimizes the learning content based on it. To achieve this, the device is equipped with emotion recognition hardware such as a camera and microphone. The software uses natural language processing models (such as ChatGPT) and emotion recognition algorithms (such as EmotionEngine). The server processes the user's questions, generates optimal answers based on the emotional information, and recommends learning content.

[0480] Specific examples

[0481] For example, if a user enters a question such as "I want to know about data analysis using Python," the system operates as follows:

[0482] 1. The user enters "I want to know about data analysis using Python" into the input field of the terminal and submits it.

[0483] 2. The device sends this question to the server.

[0484] 3. The server receives the question and generates an answer using a natural language processing model (ChatGPT).

[0485] 4. The server sends the generated answer to the terminal and displays it to the user.

[0486] 5. At the same time, the device's emotion recognition means analyzes the user's emotion information and sends it to the server.

[0487] 6. The server dynamically adjusts answers and learning content based on emotional information.

[0488] When a user requests a project simulation, the server provides the user with a randomly selected simulation project and adjusts the difficulty level appropriately based on emotional information. For example, if the project "Development of a Virtual Store" is selected and the user feels stressed during the project simulation, the system will lower the difficulty level or add an explanation.

[0489] Prompt Sentence Examples

[0490] Below are some example prompts for generative AI models:

[0491] "What are some specific ways to make effective learning content recommendations when users are fatigued?"

[0492] "Please provide detailed steps on how to personalize educational content based on emotion recognition."

[0493] As described above, the embodiments of the present invention utilize the user's emotional information to provide individually optimized learning content and virtual reality project simulation, thereby maximizing the learning effect and improving the user's learning experience.

[0494] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0495] Step 1:

[0496] The user enters a question into the input field of the terminal and clicks the send button. This input field accepts the user's question in text format. The entered question is sent from the terminal to the server. The input is text data, and the output is an HTTP POST request to the server.

[0497] Step 2:

[0498] The server receives a question sent by a user. The server sends this question to a natural language processing model (ChatGPT) to generate an answer. The input is the text data of the question sent by the user, and the output is an API request to the natural language processing model and its response text.

[0499] Step 3:

[0500] The server receives a response from the natural language processing model and sends it to the terminal. The input is the data returned from the natural language processing model, and the output is text data as an HTTP response to the terminal.

[0501] Step 4:

[0502] The terminal receives the response and displays it to the user. Specifically, it displays the response text on the user interface. In this step, the input is the response data from the server, and the output is the text displayed on the terminal screen.

[0503] Step 5:

[0504] The device's emotion recognition means analyzes the user's facial expressions and tone of voice in real time. Emotional data is acquired using a camera or microphone and analyzed by an emotion recognition algorithm. The input is raw data from the camera or microphone, and the output is analyzed emotional information.

[0505] Step 6:

[0506] The device sends the acquired emotion information to the server. The input is emotion data from the emotion recognition means, and the output is emotion information as an HTTP POST request to the server.

[0507] Step 7:

[0508] The server receives emotional information and dynamically adjusts the tone and content of the answer based on that information. For example, if the user is feeling stressed, the server can make the answer gentler or add a brief explanation. The input is emotional information, and the output is adjusted text data.

[0509] Step 8:

[0510] The server sends the adjusted answers or learning content to the terminal and displays them again to the user. The input is the adjusted text data, and the output is data as an HTTP response to the terminal.

[0511] Step 9:

[0512] A user requests a project simulation. This request is sent from the terminal to the server. The input is the simulation request from the user, and the output is an HTTP GET request to the server.

[0513] Step 10:

[0514] The server receives a request for a project simulation and selects an appropriate project from the database. The input is the simulation request and the output is the selected project information.

[0515] Step 11:

[0516] The server sends the selected project information to the terminal. The input is the project information, and the output is the simulation data as an HTTP response to the terminal.

[0517] Step 12:

[0518] The terminal receives the project information and provides it to the user in the virtual reality space. The user executes the project in the virtual reality environment. The input is the project information, and the output is a simulation display in the virtual reality environment.

[0519] Step 13:

[0520] During the project simulation, the emotion recognition means recognizes the user's emotions in real time and sends them to the server. The input is raw data from the camera and microphone, and the output is analyzed emotional information.

[0521] Step 14:

[0522] The server dynamically adjusts the project difficulty and the learning content provided based on the emotional information, and provides it to the user again. The input is the emotional information and project progress information, and the output is the adjusted simulation content.

[0523] As a result, a system is realized that provides optimal learning experiences and project simulations according to the user's emotional state.

[0524] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0525] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0526] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0527] [Second embodiment]

[0528] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0529] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0530] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0531] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0532] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0533] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0534] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0535] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0536] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0537] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0538] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0539] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0540] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[0541] System configuration

[0542] 1. Terminal

[0543] The terminal has an input device and a user interface for users to input questions and project simulation requests, and has the ability to send requests to a server via the Internet.

[0544] 2. Server

[0545] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. Furthermore, the server manages a database, storing and analyzing learning history and performance data.

[0546] 3. Virtual Reality Space Simulation Method

[0547] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[0548] Program processing explanation

[0549] ChatGPT part

[0550] 1. Receiving and processing inquiries

[0551] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[0552] The server receives this request and extracts the question.

[0553] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[0554] The server receives the response generated by ChatGPT and sends it to the device.

[0555] The terminal receives the response and displays it to the user.

[0556] Metaverse Project Part

[0557] 1. Request and provide project simulation

[0558] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[0559] The server receives the request and extracts random or specified project information from the database.

[0560] The server sends details of the selected project to the terminal.

[0561] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[0562] Specific examples

[0563] Example 1: A user asks, "I want to know about data analysis in Python."

[0564] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0565] 2. The device sends this question as a POST request to the server's API endpoint.

[0566] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0567] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0568] 5. The server sends this response to the terminal and displays it to the user.

[0569] Example 2: User requests a new project

[0570] 1. The user clicks the project simulation request button on the terminal.

[0571] 2. The device sends a GET request to the server.

[0572] 3. The server receives the request and randomly selects a project from the database.

[0573] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[0574] 5. The device displays this information to the user and provides a project simulation in a virtual reality space.

[0575] As described above, the system of the present invention provides users with an interactive learning environment and practical project experience, thereby enhancing learning effectiveness and creating an environment where working professionals can efficiently acquire new skills and techniques.

[0576] The processing flow will be explained below.

[0577] ChatGPT part

[0578] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[0579] Step 1:

[0580] The user types a question into the terminal and clicks the send button.

[0581] Step 2:

[0582] The terminal sends the entered question to the server as a POST request.

[0583] Step 3:

[0584] The server receives the request and extracts the question.

[0585] Step 4:

[0586] The server sends the extracted questions to the natural language processing model (ChatGPT).

[0587] Step 5:

[0588] ChatGPT generates a response to the question and sends it back to the server.

[0589] Step 6:

[0590] The server receives the response sent back from ChatGPT, converts it to JSON format, and sends it back to the device.

[0591] Step 7:

[0592] The terminal displays the response received from the server to the user.

[0593] Metaverse Project Part

[0594] Example 2: Processing steps for a project simulation request

[0595] Step 1:

[0596] The user clicks the Request Project Simulation button on the terminal.

[0597] Step 2:

[0598] The terminal sends a project simulation request to the server as a GET request.

[0599] Step 3:

[0600] The server receives the request and randomly selects a project from the database.

[0601] Step 4:

[0602] The server extracts the details of the selected project.

[0603] Step 5:

[0604] The server converts the project details into JSON format and sends it back to the device.

[0605] Step 6:

[0606] The terminal displays detailed information about the project received from the server to the user.

[0607] Step 7:

[0608] The user accesses the virtual reality space simulation means via a terminal and experiences the project.

[0609] Example 1

[0610] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] In today's educational environment, learners are expected to efficiently acquire new skills and knowledge. However, existing educational systems struggle to provide personalized instruction and practical experiences that are tailored to learners' specific needs. Furthermore, traditional methods struggle to support learning through real-time dialogue or simulated learning using virtual reality. This hinders the improvement of learning outcomes and the acquisition of practical skills.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0613] In this invention, the server includes means for receiving data input by a user from an input / output device, means for transmitting the data to a natural language processing algorithm to obtain a response, and means for transmitting the response to the input / output device and displaying it to the user. This allows users to learn through question and answer sessions using a natural language processing algorithm in real time. The server also includes means for storing multiple virtual space simulation projects in a storage device and providing randomly selected project information to the user, enabling learning through practical project experience. Furthermore, the server includes means for analyzing a user's learning history and performance data and providing individually optimized learning content, enabling personalized education and significantly improving learning effectiveness.

[0614] 1. An "input / output device" is a device that allows a user to input data and display information received from a server.

[0615] 2. "Natural language processing algorithms" are machine learning models for understanding text data and generating appropriate responses.

[0616] 3. "Information Processing Device" means a computer system for receiving and processing data and generating and transmitting responses.

[0617] 4. "Virtual space simulation device" means a device that uses virtual reality technology to provide users with a simulation experience.

[0618] 5. "Interface" means the means by which a user operates a virtual reality simulation device.

[0619] 6. "Storage device" means a hardware device for storing data and retrieving that data as needed.

[0620] 7. "Project Information" means the specific scenarios and data used in the virtual world simulation.

[0621] 8. "Learning History" means a record of a user's previous learning activities.

[0622] 9. "Performance Data" means information that indicates a User's learning results and progress.

[0623] A specific system and its operation that are the subject of the present invention will be described below. The system is an interactive educational platform where users learn through input / output devices, and includes an information processing device and a virtual space simulation device.

[0624] System configuration

[0625] 1. Input / Output Devices

[0626] An input / output device is a device that allows users to input data and display information. Examples include personal computers, tablets, and smartphones. An input / output device has the ability to send requests to an information processing device via the Internet.

[0627] 2. Information Processing Device

[0628] It includes a natural language processing algorithm and generates responses to user questions. Specifically, generative AI models such as ChatGPT are used. The information processing device is connected to the virtual space simulation device and manages the virtual project simulation. The information processing device also manages the storage device, accumulating and analyzing learning history and performance data.

[0629] 3. Virtual space simulation device

[0630] It uses virtual reality technology to provide an environment in which users can simulate projects. Specifically, a VR headset is used. Multiple virtual projects are stored in a storage device and are selected randomly or provided according to the user's preferences.

[0631] Program processing explanation

[0632] Question-answering part of natural language processing

[0633] 1. Enter and submit your question

[0634] The user enters a question into the input / output device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and submits it.

[0635] 2. Receiving and processing inquiries

[0636] The input / output device transmits the input question to the information processing device as a POST request.

[0637] An information processing device receives the request, extracts the question content, sends it to a natural language processing algorithm, and generates a response.

[0638] 3. Receiving and Displaying Responses

[0639] The information processing device sends the generated response to the input / output device, which receives the response and displays it to the user. For example, it displays a response such as, "The pandas library is often used for data analysis in Python..."

[0640] Requesting and providing virtual space simulations

[0641] 1. Request and submit a simulation

[0642] A user clicks a project simulation request button on an input / output device. For example, the user thinks, "I want to simulate a new project," and presses the corresponding button.

[0643] 2. Obtaining project information

[0644] The input / output device transmits a request to the information processing device as a GET request.

[0645] The information processing device receives the request and extracts random or specified project information from the storage device, for example, selecting the "Virtual Store Development" project.

[0646] 3. Sending and viewing project information

[0647] The information processing device transmits details of the selected project to the input / output device.

[0648] The input / output device displays the received project details to the user, and provides an environment in which the project can be executed through a virtual space simulation device. For example, a development scenario for a virtual store is displayed on a VR headset.

[0649] Specific examples

[0650] Example 1: A user asks, "I want to know about data analysis in Python."

[0651] 1. The user enters "I want to know about data analysis using Python" into the input form on the input / output device and clicks the submit button.

[0652] 2. The input / output device sends this question as a POST request to the API endpoint of the information processing device.

[0653] 3. An information processor receives the question and sends it to a natural language processing algorithm to generate a response.

[0654] 4. The information processing device receives a response from ChatGPT such as, "The pandas library is often used for data analysis in Python..."

[0655] 5. The information processing device sends this response to the input / output device and displays it to the user.

[0656] Example 2: User requests a new project

[0657] 1. The user clicks the project simulation request button on the I / O device.

[0658] 2. The input / output device sends the data as a GET request to the information processing device.

[0659] 3. The information processing device receives the request and randomly selects a project from the storage device.

[0660] 4. The information processing device selects "Development of a virtual store" and sends the detailed information to the input / output device.

[0661] 5. The input / output device displays this information to the user, providing a project simulation in the virtual space simulation device.

[0662] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, enhancing learning effectiveness and enabling them to efficiently acquire new skills and techniques.

[0663] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0664] Step 1:

[0665] The user enters a question into the input form of the I / O device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and then presses the "Submit" button. The input data is stored in the memory of the I / O device and used for subsequent processing.

[0666] Step 2:

[0667] The device sends the entered question to the server as a POST request. Specifically, it sends the entered text data as an HTTP request to the API endpoint. The output is the HTTP request received by the server.

[0668] Step 3:

[0669] The server receives the request and extracts the question. It extracts pure text data from the HTTP request body. This process analyzes the question from the received request and converts the request content into a format suitable for natural language processing algorithms. The output is the extracted text data.

[0670] Step 4:

[0671] The server sends the extracted question to a natural language processing algorithm (ChatGPT) to generate a response. Specifically, the extracted question is passed to ChatGPT as a prompt sentence, which is then processed using natural language processing. The generated response data is the output.

[0672] Step 5:

[0673] The server receives the generated response and sends a reply to the input / output device. Specifically, it converts the response from the natural language processing algorithm into an HTTP response format and sends it to the input / output device. The output is an HTTP response to the input / output device.

[0674] Step 6:

[0675] The terminal receives the response and displays it to the user. Specifically, it analyzes the received HTTP response data and renders the text data on the screen. It must be displayed in a format that is easy for the user to read. The output is text that is displayed on the user's screen.

[0676] Step 7:

[0677] A user clicks a project simulation request button on an input / output device. For example, the user thinks, "I want to simulate a new project," and presses the corresponding button. The input data is the user's request.

[0678] Step 8:

[0679] The terminal sends a request to the server as a GET request. Specifically, it sends the request data generated based on the user's action as an HTTP GET request. The output is an HTTP request received by the server.

[0680] Step 9:

[0681] The server receives the request and extracts random or specified project information from storage. It performs a database query to retrieve the appropriate project information. For example, select "Virtual Store Development." The output is the extracted project information.

[0682] Step 10:

[0683] The server sends the details of the selected project to the terminal. Specifically, the server forms the project information as an HTTP response and sends it to the input / output device. The output is an HTTP response to the input / output device.

[0684] Step 11:

[0685] The terminal displays the received project details to the user, providing an environment in which the project can be executed through a virtual space simulation device. Specifically, the received details are analyzed and displayed on the screen. The terminal also launches a virtual space simulation to create an environment in which the project can be simulated. For example, a development scenario for a virtual store can be displayed on a VR headset. The output is a simulation environment that the user can directly experience.

[0686] (Application example 1)

[0687] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] Currently, when users learn, it is difficult for them to acquire specific knowledge or acquire practical operational skills through experience. Furthermore, there is a lack of systems that provide effective feedback and individually optimized learning content quickly. In particular, there is a need for a platform that integrates operational simulation functions in a virtual space to provide a learning environment that is close to real-life experience, allowing users to efficiently acquire skills.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0690] In this invention, the server includes means for providing an operation simulation in a virtual space based on questions and requests input by a user from a terminal, means for displaying the results of the virtual space simulation to the user, and means for analyzing the data and generating optimized feedback based on the user's learning history and performance. This allows the user to experience a simulation in a virtual space that is close to an actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[0691] "User" means an individual or corporation that uses this system.

[0692] A "terminal" is a device, such as a smartphone, tablet, or computer, through which a user inputs a question or request.

[0693] "Question" refers to a learning-related inquiry or doubt that a user inputs via a terminal.

[0694] A "natural language processing model" is a type of artificial intelligence that generates appropriate answers to user questions or input.

[0695] An "answer" is a response to a user's question generated by a natural language processing model.

[0696] The "server" is a device that includes a natural language processing model and manages answers to user questions and simulation results in cooperation with the virtual reality space simulation means.

[0697] "Virtual reality space simulation" is a system that uses virtual reality technology to allow users to experience virtual environments and scenarios.

[0698] "Simulation means" refers to a device or software for executing a virtual reality space simulation.

[0699] "Interface" refers to a user interface that allows a user to operate the simulation means.

[0700] "Operation simulation" refers to a program that allows users to learn through experience how to operate and manage a store in a virtual space.

[0701] "Virtual space" refers to a virtual environment provided by virtual reality technology.

[0702] Analyzing "data" refers to the process of analyzing collected user learning history and performance data to find trends and patterns.

[0703] "Learning history" refers to historical data of the learning activities and simulations that a user has performed up to now.

[0704] "Performance Data" refers to data on the results or outcomes achieved by a user through simulation or learning.

[0705] "Optimized feedback" refers to specific advice and suggestions for next learning that are individually tailored based on the user's learning history and performance.

[0706] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[0707] System configuration

[0708] 1. Terminal

[0709] The device includes an input device and user interface for users to enter questions and project simulation requests. The terminal has the ability to send requests to the server via the Internet. Examples of such devices include smartphones, tablets, and computers.

[0710] 2. Server

[0711] It includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. In addition, the server manages a database to store and analyze learning history and performance data. Specific examples that can be used include Amazon Web Services (AWS) and Google Cloud Platform.

[0712] 3. Virtual Reality Space Simulation Method

[0713] It uses virtual reality technology to provide an environment where users can simulate projects. It stores multiple virtual projects, which are selected randomly or provided according to the user's preferences. It provides 3D simulations using Unity and Unreal Engine.

[0714] Program processing explanation

[0715] Receiving and processing questions

[0716] When a user enters a question into the device and clicks the send button, the server receives the question from the device. The server extracts the content of the question and sends it to a natural language processing model (for example, OpenAI's GPT-3) to generate a response. The generated response is sent from the server to the device and displayed to the user. As a concrete example, a user enters, "What is the best way to manage inventory?"

[0717] Request and provide project simulations

[0718] When a user clicks the project simulation request button on their device, the server receives the request and extracts random or specified project information from the database. The extracted project details are sent from the server to the device and displayed to the user. The user can then perform a project simulation in a virtual reality space. A specific example is when the user selects a "seasonal product campaign" and simulates product placement and advertising in a virtual space.

[0719] Data analysis and feedback

[0720] The server collects and analyzes the user's learning history and performance data, and based on this data, provides optimized feedback and suggests next learning content, allowing users to efficiently acquire new skills and techniques.

[0721] Examples of specific examples and prompts

[0722] For example, if a user asks, "What is the best way to manage inventory?", an example prompt sentence might be:

[0723] Example prompt sentence:

[0724] What is the best way to manage inventory?

[0725] This system allows users to experience simulations in a virtual space that closely resembles the actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[0726] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0727] Step 1:

[0728] The user inputs a question or a project simulation request into the terminal and clicks the send button. The terminal then sends the input question or request to the server.

[0729] Input: Questions or requests typed by the user into the device

[0730] Output: Request data from the terminal to the server

[0731] Step 2:

[0732] The server receives questions and requests sent from the device at a specified API endpoint, extracts the content of the question, and sends it to a natural language processing model to generate an answer.

[0733] Input: Request data sent from the terminal

[0734] Output: Question data sent to the natural language processing model and generated answer data

[0735] Step 3:

[0736] The server receives the response from the natural language processing model and extracts the answer, which it then sends back to the terminal and displays to the user.

[0737] Input: Answer data generated from a natural language processing model

[0738] Output: Answer data sent to the user's device

[0739] Step 4:

[0740] When a user clicks the request button of a project simulation, the terminal sends the request to the server as a GET request.

[0741] Input: Project Simulation Request

[0742] Output: GET request sent to the server

[0743] Step 5:

[0744] The server receives a request for a project simulation, extracts relevant project information from the database randomly or based on specified conditions, and transmits the extracted project information to the terminal.

[0745] Input: Project simulation request and project information from the database

[0746] Output: Project details sent to the device

[0747] Step 6:

[0748] The terminal presents the project information received from the server to the user and starts a virtual reality space simulation, and the user experiences the simulation using a virtual reality device.

[0749] Input: Project details sent from the server

[0750] Output: Project details and simulation environment presented to the user

[0751] Step 7:

[0752] During the simulation, the server collects user operation and performance data, which is then saved as a learning history and analyzed later.

[0753] Input: User operation data and performance data

[0754] Output: Learning history data and performance data

[0755] Step 8:

[0756] The server analyzes the collected learning history and performance data and generates personalized feedback, which is provided to the user and includes suggestions for next learning content.

[0757] Input: Learning history data and performance data

[0758] Output: Optimized feedback and learning suggestions provided to the user

[0759] Step 9:

[0760] The user can then view the optimized feedback and suggestions for the next learning activity on their device, and begin a new learning activity. The system repeats this process, helping the user improve their skills.

[0761] Input: Optimized feedback and learning suggestions

[0762] Output: User's new learning activity

[0763] Through these steps, users can efficiently acquire skills in an interactive learning environment.

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] As an embodiment of the present invention, a specific system incorporating an emotion engine and its operation will be described below. This system is an interactive educational platform where users learn through their terminals, and includes a server, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[0766] System configuration

[0767] 1. Terminal

[0768] The terminal is equipped with an input device and a user interface for users to input questions and project simulation requests. The terminal is equipped with an emotion engine and has the ability to recognize emotions from the user's facial expressions, tone of voice, etc.

[0769] 2. Server

[0770] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. The server also receives emotion information sent from the emotion engine and uses it to adjust answers and provide optimal learning content.

[0771] 3. Emotion Engine

[0772] It recognizes the user's emotions and extracts emotional information in real time from the user's facial expressions, tone of voice, etc. The emotion engine is installed on the device, and the data is sent to a server.

[0773] 4. Virtual Reality Space Simulation Method

[0774] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[0775] Program processing explanation

[0776] ChatGPT part

[0777] 1. Receiving and processing inquiries

[0778] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[0779] The server receives this request and extracts the question.

[0780] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[0781] The server receives the response generated by ChatGPT and sends it to the device.

[0782] The terminal receives the response and displays it to the user.

[0783] 2. Acquiring and using emotional information

[0784] While the user is typing a question, the device's emotion engine recognizes the user's emotions in real time.

[0785] The device sends emotional information to the server, which then adjusts the content and tone of the response based on this information.

[0786] The server sends the adjusted answer to the terminal and displays it to the user.

[0787] Metaverse Project Part

[0788] 1. Request and provide project simulation

[0789] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[0790] The server receives the request and randomly selects a project from the database.

[0791] The server extracts detailed information about the selected project and sends it to the device.

[0792] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[0793] 2. Use of emotional information

[0794] The emotion engine recognizes the user's emotions in real time during the project simulation.

[0795] Based on the emotional information, the server dynamically adjusts the difficulty of the project and the learning content provided.

[0796] For example, if the user is feeling stressed, the server will adjust the difficulty level by lowering it or adding explanations.

[0797] Specific examples

[0798] Example 1: A user asks, "I want to know about data analysis in Python."

[0799] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0800] 2. The device sends this question as a POST request to the server's API endpoint.

[0801] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0802] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0803] 5. The server sends this response to the terminal and displays it to the user.

[0804] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[0805] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[0806] Example 2: User requests a new project

[0807] 1. The user clicks the project simulation request button on the terminal.

[0808] 2. The device sends a GET request to the server.

[0809] 3. The server receives the request and randomly selects a project from the database.

[0810] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[0811] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[0812] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[0813] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[0814] As described above, the system of the present invention provides users with an integrated interactive learning environment and practical project experience, and further optimizes learning effectiveness by utilizing users' emotional information. This creates an environment in which working professionals can efficiently acquire new skills and techniques.

[0815] The processing flow will be explained below.

[0816] ChatGPT part

[0817] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[0818] Step 1:

[0819] The user types a question into the terminal, saying "I want to know about data analysis using Python," and clicks the submit button.

[0820] Step 2:

[0821] The device sends the entered question as a POST request to the server's API endpoint.

[0822] Step 3:

[0823] The server receives the POST request and extracts the question from the request data.

[0824] Step 4:

[0825] The server sends the extracted questions to a natural language processing model (ChatGPT).

[0826] Step 5:

[0827] ChatGPT generates answers to questions and sends the answers back to the server.

[0828] Step 6:

[0829] The server receives the response sent back from ChatGPT, formats the response appropriately, and sends it to the device.

[0830] Step 7:

[0831] The terminal displays the received response to the user.

[0832] Use of emotional information

[0833] Acquiring and using emotional information with an emotion engine

[0834] Step 1:

[0835] While the user types their question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[0836] Step 2:

[0837] The emotion information recognized by the device's emotion engine is sent to the server.

[0838] Step 3:

[0839] The server receives the emotional information and adjusts the content and tone of the answers to questions.

[0840] Step 4:

[0841] The server sends the adjusted answer to the terminal and displays it to the user.

[0842] Metaverse Project Part

[0843] Example 2: Processing steps for a project simulation request

[0844] Step 1:

[0845] The user clicks the Request Project Simulation button on the terminal.

[0846] Step 2:

[0847] The terminal sends a project simulation request to the server as a GET request.

[0848] Step 3:

[0849] The server receives a GET request and randomly selects multiple project information from the database.

[0850] Step 4:

[0851] The server extracts detailed information about the selected project.

[0852] Step 5:

[0853] The server converts the project details into JSON format and sends it to the terminal.

[0854] Step 6:

[0855] The terminal displays the received project details information to the user.

[0856] Step 7:

[0857] The user accesses the virtual reality space simulation means via the terminal and experiences the selected project.

[0858] Use of emotional information

[0859] Using emotional information during project simulation with an emotion engine

[0860] Step 1:

[0861] During the project simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[0862] Step 2:

[0863] The emotion information recognized by the device's emotion engine is sent to the server.

[0864] Step 3:

[0865] The server receives emotional information and dynamically adjusts the difficulty of the project and the learning content provided.

[0866] Step 4:

[0867] The server sends the adjusted learning content and support information to the terminal and displays it to the user.

[0868] Example 2

[0869] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] Conventional interactive educational systems have struggled to provide learning content that takes into account the user's emotional state. This can lead to problems such as reduced learning efficiency and increased stress. Furthermore, even with project simulations using virtual reality technology, optimal learning outcomes cannot be achieved because the content is not dynamically adjusted based on the user's progress or emotions.

[0871] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means including an emotion engine that recognizes the user's emotions while the user is entering or viewing a question and extracts emotional information in real time, a means for transmitting the emotional information to the server and adjusting the content and tone of the answer, and a means for transmitting the adjusted answer to the terminal and displaying it. This makes it possible to provide learning content tailored to the user's emotional state. In addition, by storing multiple virtual reality space simulation projects in a database, recognizing the user's emotions in real time during the simulation, and dynamically adjusting the content and difficulty of the simulation based on the emotional information, it is possible to provide the user with an optimized learning environment.

[0872] A "terminal" is a device that has an interface and input devices through which a user can enter questions and project simulation requests.

[0873] The "server" is a central device that contains the natural language processing model, generates answers based on user questions, and adjusts learning based on emotional and other data.

[0874] A "natural language processing model" is a software system that includes algorithms that analyze natural language questions entered by users and generate appropriate answers to those questions.

[0875] An "emotion engine" is a device that recognizes a user's facial expressions and tone of voice in real time and extracts emotional information.

[0876] A "virtual reality space simulation means" is a means and device that uses virtual reality technology to provide an environment in which a user can experience a virtual project.

[0877] The "means for adjusting the content and tone of the answer" refers to a mechanism or algorithm for adjusting the answer generated by the natural language processing model based on the emotional information sent from the emotion engine in accordance with the user's emotional state.

[0878] A "project simulation request" is request information that a user sends from a terminal to the system when the user desires to perform a project simulation in a virtual reality space.

[0879] "Study history" is data that records the user's past learning activities and their progress.

[0880] "Performance data" is data that indicates the results and efficiency of a user's learning activities.

[0881] A "database" is a storage device that stores multiple virtual reality space simulation projects and related information and can be accessed as needed.

[0882] The present invention is a system in which a user inputs a question via a terminal, a natural language processing model generates an answer based on the input, and further provides optimized learning content using the user's emotional information. The system includes a server, a terminal, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[0883] System configuration

[0884] Terminal

[0885] The terminal is equipped with an interface and an input device for users to input questions and project simulation requests. The terminal is equipped with an emotion engine that recognizes the user's facial expressions and tone of voice in real time and extracts emotional information.

[0886] server

[0887] The server receives questions entered by users from their devices and sends them to a natural language processing model to generate answers. The generated answers are then sent to the devices and displayed to the users. The server also receives emotional information sent from the emotion engine and adjusts the content and tone of the answers based on this information. The server is also connected to a virtual reality space simulation means and manages the simulation of the virtual project.

[0888] Emotion Engine

[0889] The emotion engine is installed on the device and extracts emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to the server, which then adjusts the content and tone of the answer based on this information.

[0890] Virtual reality space simulation tool

[0891] The virtual reality space simulation means provides an environment in which a user can simulate a project using virtual reality technology. A plurality of virtual projects are stored in a database and are selected randomly or provided according to the user's request.

[0892] Specific examples

[0893] Specific example of question processing

[0894] If a user asks, "I want to know about data analysis using Python," the specific process is as follows:

[0895] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[0896] 2. The device sends this question as a POST request to the server's API endpoint.

[0897] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[0898] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[0899] 5. The server sends this response to the terminal and displays it to the user.

[0900] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[0901] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[0902] Example prompt sentence:

[0903] User: "I want to learn about data analysis using Python."

[0904] Model: "The pandas library is often used for data analysis in Python. Other popular libraries include numpy and matplotlib."

[0905] Project Simulation Examples

[0906] As an example of a user requesting a new project, the "Virtual Store Development" project is selected.

[0907] 1. The user clicks the project simulation request button on the terminal.

[0908] 2. The device sends a GET request to the server.

[0909] 3. The server receives the request and randomly selects a project from the database.

[0910] 4. The server selects "Development of a virtual store" and sends the details to the terminal.

[0911] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[0912] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[0913] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[0914] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, and utilizes users' emotional information to optimize learning effectiveness, creating an environment where users can efficiently acquire new skills and techniques.

[0915] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0916] Step 1: User enters question into terminal

[0917] The user enters a question into the input form on the terminal and clicks the submit button. The input text "I want to know about data analysis using Python" is sent to the terminal.

[0918] Step 2: The device sends the question to the server

[0919] The device sends the user's question as a POST request to the server's specified API endpoint. This request includes the question entered by the user. The server then receives the request.

[0920] Step 3: The server sends the question to the natural language processing model

[0921] The server sends the received question to a natural language processing model (e.g., ChatGPT) to generate an answer. The server inputs the question into the model and receives the generated answer as output. Specifically, the answer generated is "The pandas library is often used for data analysis in Python..."

[0922] Step 4: The server sends the generated answer to the device

[0923] The server sends the generated answer, which includes text obtained from the natural language processing model, to the device, which then receives the answer.

[0924] Step 5: The device displays the answer to the user

[0925] The device displays the answer it receives to the user. This display is the text sent from the server and displayed in the interface. Specifically, the answer displayed is, "The pandas library is often used for data analysis in Python..."

[0926] Step 6: The device recognizes the user's emotions

[0927] While the user is typing a question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is surprised."

[0928] Step 7: The device sends the emotion information to the server.

[0929] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[0930] Step 8: The server adjusts the answer based on the emotional information

[0931] The server adjusts the content and tone of the generated answer based on the emotional information received. In this process, the emotional information is used as input to modify the content and expression of the answer text. For example, if the user is feeling stressed, the server may adjust the answer by adding a gentler tone.

[0932] Step 9: The server sends the adjusted answer to the device.

[0933] The server sends the adjusted answer to the device, which includes the text modified based on the sentiment, and the device receives the adjusted answer.

[0934] Step 10: The device displays the adjusted answer to the user

[0935] The terminal displays the adjusted answer to the user. This display shows the corrected text sent from the server in the interface. Specifically, the answer is displayed in a friendly tone: "The pandas library is often used for data analysis in Python. Try it yourself."

[0936] Step 11: User requests project simulation

[0937] The user clicks the project simulation request button on the terminal. As an input, a request for project simulation is sent to the terminal.

[0938] Step 12: The device sends a request to the server

[0939] The terminal sends the user's project simulation request to the server as a GET request, which includes a request for the project simulation. The server then receives the request.

[0940] Step 13: The server randomly selects a project and extracts its details

[0941] Based on the request received by the server, a virtual project is randomly selected from the database. The server extracts the project details and generates them as output. Specifically, the project "Development of a Virtual Store" is selected.

[0942] Step 14: The server sends the details to the device

[0943] The server transmits the extracted project details to the terminal, including the project description and procedures, and the terminal receives the details.

[0944] Step 15: The terminal displays detailed information to the user and provides a simulation environment

[0945] The terminal displays the received project details to the user, providing a simulation environment in virtual reality space. This display shows the project information sent from the server on the interface. Specifically, details of the "virtual store development" are displayed, and the user experiences the project in virtual reality space.

[0946] Step 16: The device recognizes the user's emotions during the simulation.

[0947] During the simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is excited."

[0948] Step 17: The device sends the emotion information to the server.

[0949] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[0950] Step 18: The server adjusts the content and difficulty of the simulation based on the emotional information.

[0951] The content and difficulty of the simulation are dynamically adjusted based on the emotional information received by the server. In this process, the content of the simulation is dynamically changed using the emotional information as input. For example, if the user feels the simulation is difficult, adjustments such as simplifying the next step are made.

[0952] Step 19: The server sends the adjustments to the device

[0953] The server transmits the adjusted simulation content to the terminal, which includes the simulation data modified based on the emotion. The adjusted content is then received by the terminal.

[0954] Step 20: The device reflects the adjustments in the simulation environment and displays them to the user.

[0955] The terminal reflects the adjusted simulation content in the virtual reality space simulation environment and displays it to the user. This display reflects the corrected simulation data sent from the server on the interface. Specifically, the "Virtual Store Development" simulation with the adjusted difficulty level is displayed.

[0956] Through these processing steps, users can enjoy an interactive learning environment that combines hands-on project experiences. Furthermore, by utilizing emotional information, an environment that optimizes learning effectiveness is provided.

[0957] (Application example 2)

[0958] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0959] Conventional educational platforms provide uniform learning content without considering the user's emotions, making it difficult to maximize individual learning efficiency and comprehension. Furthermore, because they do not take into account the user's stress or fatigue, it can be difficult to continue learning. Furthermore, when providing a simulation environment using virtual reality technology, adjustments are not made based on the user's current emotional state, making it difficult to provide an optimal learning experience.

[0960] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question input by a user from a terminal, means for transmitting the question to a natural language processing model to obtain an answer, and means for transmitting the answer to the terminal and displaying it to the user. This realizes a system including emotion recognition means for recognizing a user's emotions in real time and transmitting emotional information to the server, and means for dynamically adjusting answers and learning content based on the emotional information. This makes it possible to provide optimal and personalized learning content and a simulation environment according to the user's current emotional state.

[0961] A "terminal" is a device equipped with an input device and a user interface for a user to input questions and project simulation requests and obtain emotion information.

[0962] A "natural language processing model" is an algorithm or system that generates answers to questions entered by users, and specifically refers to a model that understands natural language and generates responses.

[0963] A "server" is a central computer that transmits, receives, and processes data via a network, and in the present invention includes a natural language processing model, a virtual reality space simulation means, and a device that processes emotional information.

[0964] "Virtual reality space simulation means" refers to technology or devices that provide an environment in which users can simulate projects using virtual reality technology.

[0965] "Emotion recognition means" refers to technology or devices that extract emotional information in real time from a user's facial expressions, tone of voice, etc., and is responsible for transmitting this information to a server.

[0966] An "interface" refers to a connection means or operating device that allows a user to operate a simulation means, and is designed to allow intuitive and efficient operation.

[0967] "Learning History and Performance Data" refers to data that records a user's past learning activities and the results of those activities, and is used to provide individually optimized learning content.

[0968] "Personalized learning content" refers to learning materials and assignments that are dynamically tailored based on a user's learning history and emotional state.

[0969] "Project simulation" refers to a virtual project that simulates real-world environments and scenarios in a virtual reality space, allowing users to experience and learn from them.

[0970] A specific embodiment for carrying out the present invention includes the following system and its operation, which enables a user to receive personalized learning content and virtual reality project simulations while having their emotions recognized in real time.

[0971] System configuration

[0972] 1. Terminal

[0973] The device is equipped with an input device and a user interface for users to input questions and project simulation requests. The device is also equipped with emotion recognition means to extract emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to a server and used to dynamically adjust the learning content.

[0974] 2. Server

[0975] It includes a natural language processing model (e.g., ChatGPT) and generates answers to user questions. The server receives emotional information sent from the emotion recognition means and dynamically adjusts the answers and learning content. It is also connected to the virtual reality space simulation means and manages the simulation of the virtual project.

[0976] 3. Emotion recognition means

[0977] Emotional information is extracted in real time from the user's facial expressions, tone of voice, etc. This is installed on the device, and the data is sent to the server. Based on the emotional information, the server adjusts the answers and learning content.

[0978] 4. Virtual Reality Space Simulation Method

[0979] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored in a database, and these are provided randomly or according to the user's preferences.

[0980] Program processing explanation

[0981] The server receives the user's emotional information in real time and dynamically optimizes the learning content based on it. To achieve this, the device is equipped with emotion recognition hardware such as a camera and microphone. The software uses natural language processing models (such as ChatGPT) and emotion recognition algorithms (such as EmotionEngine). The server processes the user's questions, generates optimal answers based on the emotional information, and recommends learning content.

[0982] Specific examples

[0983] For example, if a user enters a question such as "I want to know about data analysis using Python," the system operates as follows:

[0984] 1. The user enters "I want to know about data analysis using Python" into the input field of the terminal and submits it.

[0985] 2. The device sends this question to the server.

[0986] 3. The server receives the question and generates an answer using a natural language processing model (ChatGPT).

[0987] 4. The server sends the generated answer to the terminal and displays it to the user.

[0988] 5. At the same time, the device's emotion recognition means analyzes the user's emotion information and sends it to the server.

[0989] 6. The server dynamically adjusts answers and learning content based on emotional information.

[0990] When a user requests a project simulation, the server provides the user with a randomly selected simulation project and adjusts the difficulty level appropriately based on emotional information. For example, if the project "Development of a Virtual Store" is selected and the user feels stressed during the project simulation, the system will lower the difficulty level or add an explanation.

[0991] Prompt Sentence Examples

[0992] Below are some example prompts for generative AI models:

[0993] "What are some specific ways to make effective learning content recommendations when users are fatigued?"

[0994] "Please provide detailed steps on how to personalize educational content based on emotion recognition."

[0995] As described above, the embodiments of the present invention utilize the user's emotional information to provide individually optimized learning content and virtual reality project simulation, thereby maximizing the learning effect and improving the user's learning experience.

[0996] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0997] Step 1:

[0998] The user enters a question into the input field of the terminal and clicks the send button. This input field accepts the user's question in text format. The entered question is sent from the terminal to the server. The input is text data, and the output is an HTTP POST request to the server.

[0999] Step 2:

[1000] The server receives a question sent by a user. The server sends this question to a natural language processing model (ChatGPT) to generate an answer. The input is the text data of the question sent by the user, and the output is an API request to the natural language processing model and its response text.

[1001] Step 3:

[1002] The server receives a response from the natural language processing model and sends it to the terminal. The input is the data returned from the natural language processing model, and the output is text data as an HTTP response to the terminal.

[1003] Step 4:

[1004] The terminal receives the response and displays it to the user. Specifically, it displays the response text on the user interface. In this step, the input is the response data from the server, and the output is the text displayed on the terminal screen.

[1005] Step 5:

[1006] The device's emotion recognition means analyzes the user's facial expressions and tone of voice in real time. Emotional data is acquired using a camera or microphone and analyzed by an emotion recognition algorithm. The input is raw data from the camera or microphone, and the output is analyzed emotional information.

[1007] Step 6:

[1008] The device sends the acquired emotion information to the server. The input is emotion data from the emotion recognition means, and the output is emotion information as an HTTP POST request to the server.

[1009] Step 7:

[1010] The server receives emotional information and dynamically adjusts the tone and content of the answer based on that information. For example, if the user is feeling stressed, the server can make the answer gentler or add a brief explanation. The input is emotional information, and the output is adjusted text data.

[1011] Step 8:

[1012] The server sends the adjusted answers or learning content to the terminal and displays them again to the user. The input is the adjusted text data, and the output is data as an HTTP response to the terminal.

[1013] Step 9:

[1014] A user requests a project simulation. This request is sent from the terminal to the server. The input is the simulation request from the user, and the output is an HTTP GET request to the server.

[1015] Step 10:

[1016] The server receives a request for a project simulation and selects an appropriate project from the database. The input is the simulation request and the output is the selected project information.

[1017] Step 11:

[1018] The server sends the selected project information to the terminal. The input is the project information, and the output is the simulation data as an HTTP response to the terminal.

[1019] Step 12:

[1020] The terminal receives the project information and provides it to the user in the virtual reality space. The user executes the project in the virtual reality environment. The input is the project information, and the output is a simulation display in the virtual reality environment.

[1021] Step 13:

[1022] During the project simulation, the emotion recognition means recognizes the user's emotions in real time and sends them to the server. The input is raw data from the camera and microphone, and the output is analyzed emotional information.

[1023] Step 14:

[1024] The server dynamically adjusts the project difficulty and the learning content provided based on the emotional information, and provides it to the user again. The input is the emotional information and project progress information, and the output is the adjusted simulation content.

[1025] As a result, a system is realized that provides optimal learning experiences and project simulations according to the user's emotional state.

[1026] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1027] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1028] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1029] [Third embodiment]

[1030] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1031] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1033] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1034] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1035] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1037] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1038] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1040] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1041] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1042] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[1043] System configuration

[1044] 1. Terminal

[1045] The terminal has an input device and a user interface for users to input questions and project simulation requests, and has the ability to send requests to a server via the Internet.

[1046] 2. Server

[1047] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. Furthermore, the server manages a database, storing and analyzing learning history and performance data.

[1048] 3. Virtual Reality Space Simulation Method

[1049] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[1050] Program processing explanation

[1051] ChatGPT part

[1052] 1. Receiving and processing inquiries

[1053] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[1054] The server receives this request and extracts the question.

[1055] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[1056] The server receives the response generated by ChatGPT and sends it to the device.

[1057] The terminal receives the response and displays it to the user.

[1058] Metaverse Project Part

[1059] 1. Request and provide project simulation

[1060] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[1061] The server receives the request and extracts random or specified project information from the database.

[1062] The server sends details of the selected project to the terminal.

[1063] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[1064] Specific examples

[1065] Example 1: A user asks, "I want to know about data analysis in Python."

[1066] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1067] 2. The device sends this question as a POST request to the server's API endpoint.

[1068] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1069] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1070] 5. The server sends this response to the terminal and displays it to the user.

[1071] Example 2: User requests a new project

[1072] 1. The user clicks the project simulation request button on the terminal.

[1073] 2. The device sends a GET request to the server.

[1074] 3. The server receives the request and randomly selects a project from the database.

[1075] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[1076] 5. The device displays this information to the user and provides a project simulation in a virtual reality space.

[1077] As described above, the system of the present invention provides users with an interactive learning environment and practical project experience, thereby enhancing learning effectiveness and creating an environment where working professionals can efficiently acquire new skills and techniques.

[1078] The processing flow will be explained below.

[1079] ChatGPT part

[1080] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[1081] Step 1:

[1082] The user types a question into the terminal and clicks the send button.

[1083] Step 2:

[1084] The terminal sends the entered question to the server as a POST request.

[1085] Step 3:

[1086] The server receives the request and extracts the question.

[1087] Step 4:

[1088] The server sends the extracted questions to the natural language processing model (ChatGPT).

[1089] Step 5:

[1090] ChatGPT generates a response to the question and sends it back to the server.

[1091] Step 6:

[1092] The server receives the response sent back from ChatGPT, converts it to JSON format, and sends it back to the device.

[1093] Step 7:

[1094] The terminal displays the response received from the server to the user.

[1095] Metaverse Project Part

[1096] Example 2: Processing steps for a project simulation request

[1097] Step 1:

[1098] The user clicks the Request Project Simulation button on the terminal.

[1099] Step 2:

[1100] The terminal sends a project simulation request to the server as a GET request.

[1101] Step 3:

[1102] The server receives the request and randomly selects a project from the database.

[1103] Step 4:

[1104] The server extracts the details of the selected project.

[1105] Step 5:

[1106] The server converts the project details into JSON format and sends it back to the device.

[1107] Step 6:

[1108] The terminal displays detailed information about the project received from the server to the user.

[1109] Step 7:

[1110] The user accesses the virtual reality space simulation means via a terminal and experiences the project.

[1111] Example 1

[1112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1113] In today's educational environment, learners are expected to efficiently acquire new skills and knowledge. However, existing educational systems struggle to provide personalized instruction and practical experiences that are tailored to learners' specific needs. Furthermore, traditional methods struggle to support learning through real-time dialogue or simulated learning using virtual reality. This hinders the improvement of learning outcomes and the acquisition of practical skills.

[1114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1115] In this invention, the server includes means for receiving data input by a user from an input / output device, means for transmitting the data to a natural language processing algorithm to obtain a response, and means for transmitting the response to the input / output device and displaying it to the user. This allows users to learn through question and answer sessions using a natural language processing algorithm in real time. The server also includes means for storing multiple virtual space simulation projects in a storage device and providing randomly selected project information to the user, enabling learning through practical project experience. Furthermore, the server includes means for analyzing a user's learning history and performance data and providing individually optimized learning content, enabling personalized education and significantly improving learning effectiveness.

[1116] 1. An "input / output device" is a device that allows a user to input data and display information received from a server.

[1117] 2. "Natural language processing algorithms" are machine learning models for understanding text data and generating appropriate responses.

[1118] 3. "Information Processing Device" means a computer system for receiving and processing data and generating and transmitting responses.

[1119] 4. "Virtual space simulation device" means a device that uses virtual reality technology to provide users with a simulation experience.

[1120] 5. "Interface" means the means by which a user operates a virtual reality simulation device.

[1121] 6. "Storage device" means a hardware device for storing data and retrieving that data as needed.

[1122] 7. "Project Information" means the specific scenarios and data used in the virtual world simulation.

[1123] 8. "Learning History" means a record of a user's previous learning activities.

[1124] 9. "Performance Data" means information that indicates a User's learning results and progress.

[1125] A specific system and its operation that are the subject of the present invention will be described below. The system is an interactive educational platform where users learn through input / output devices, and includes an information processing device and a virtual space simulation device.

[1126] System configuration

[1127] 1. Input / Output Devices

[1128] An input / output device is a device that allows users to input data and display information. Examples include personal computers, tablets, and smartphones. An input / output device has the ability to send requests to an information processing device via the Internet.

[1129] 2. Information Processing Device

[1130] It includes a natural language processing algorithm and generates responses to user questions. Specifically, generative AI models such as ChatGPT are used. The information processing device is connected to the virtual space simulation device and manages the virtual project simulation. The information processing device also manages the storage device, accumulating and analyzing learning history and performance data.

[1131] 3. Virtual space simulation device

[1132] It uses virtual reality technology to provide an environment in which users can simulate projects. Specifically, a VR headset is used. Multiple virtual projects are stored in a storage device and are selected randomly or provided according to the user's preferences.

[1133] Program processing explanation

[1134] Question-answering part of natural language processing

[1135] 1. Enter and submit your question

[1136] The user enters a question into the input / output device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and submits it.

[1137] 2. Receiving and processing inquiries

[1138] The input / output device transmits the input question to the information processing device as a POST request.

[1139] An information processing device receives the request, extracts the question content, sends it to a natural language processing algorithm, and generates a response.

[1140] 3. Receiving and Displaying Responses

[1141] The information processing device sends the generated response to the input / output device, which receives the response and displays it to the user. For example, it displays a response such as, "The pandas library is often used for data analysis in Python..."

[1142] Requesting and providing virtual space simulations

[1143] 1. Request and submit a simulation

[1144] A user clicks a project simulation request button on an input / output device. For example, the user thinks, "I want to simulate a new project," and presses the corresponding button.

[1145] 2. Obtaining project information

[1146] The input / output device transmits a request to the information processing device as a GET request.

[1147] The information processing device receives the request and extracts random or specified project information from the storage device, for example, selecting the "Virtual Store Development" project.

[1148] 3. Sending and viewing project information

[1149] The information processing device transmits details of the selected project to the input / output device.

[1150] The input / output device displays the received project details to the user, and provides an environment in which the project can be executed through a virtual space simulation device. For example, a development scenario for a virtual store is displayed on a VR headset.

[1151] Specific examples

[1152] Example 1: A user asks, "I want to know about data analysis in Python."

[1153] 1. The user enters "I want to know about data analysis using Python" into the input form on the input / output device and clicks the submit button.

[1154] 2. The input / output device sends this question as a POST request to the API endpoint of the information processing device.

[1155] 3. An information processor receives the question and sends it to a natural language processing algorithm to generate a response.

[1156] 4. The information processing device receives a response from ChatGPT such as, "The pandas library is often used for data analysis in Python..."

[1157] 5. The information processing device sends this response to the input / output device and displays it to the user.

[1158] Example 2: User requests a new project

[1159] 1. The user clicks the project simulation request button on the I / O device.

[1160] 2. The input / output device sends the data as a GET request to the information processing device.

[1161] 3. The information processing device receives the request and randomly selects a project from the storage device.

[1162] 4. The information processing device selects "Development of a virtual store" and sends the detailed information to the input / output device.

[1163] 5. The input / output device displays this information to the user, providing a project simulation in the virtual space simulation device.

[1164] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, enhancing learning effectiveness and enabling them to efficiently acquire new skills and techniques.

[1165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1166] Step 1:

[1167] The user enters a question into the input form of the I / O device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and then presses the "Submit" button. The input data is stored in the memory of the I / O device and used for subsequent processing.

[1168] Step 2:

[1169] The device sends the entered question to the server as a POST request. Specifically, it sends the entered text data as an HTTP request to the API endpoint. The output is the HTTP request received by the server.

[1170] Step 3:

[1171] The server receives the request and extracts the question. It extracts pure text data from the HTTP request body. This process analyzes the question from the received request and converts the request content into a format suitable for natural language processing algorithms. The output is the extracted text data.

[1172] Step 4:

[1173] The server sends the extracted question to a natural language processing algorithm (ChatGPT) to generate a response. Specifically, the extracted question is passed to ChatGPT as a prompt sentence, which is then processed using natural language processing. The generated response data is the output.

[1174] Step 5:

[1175] The server receives the generated response and sends a reply to the input / output device. Specifically, it converts the response from the natural language processing algorithm into an HTTP response format and sends it to the input / output device. The output is an HTTP response to the input / output device.

[1176] Step 6:

[1177] The terminal receives the response and displays it to the user. Specifically, it analyzes the received HTTP response data and renders the text data on the screen. It must be displayed in a format that is easy for the user to read. The output is text that is displayed on the user's screen.

[1178] Step 7:

[1179] A user clicks a project simulation request button on an input / output device. For example, if the user thinks, "I want to simulate a new project," he / she presses the corresponding button. The input data is the user's request.

[1180] Step 8:

[1181] The terminal sends a request to the server as a GET request. Specifically, it sends the request data generated based on the user's action as an HTTP GET request. The output is an HTTP request received by the server.

[1182] Step 9:

[1183] The server receives the request and extracts random or specified project information from storage. It performs a database query to retrieve the appropriate project information. For example, select "Virtual Store Development." The output is the extracted project information.

[1184] Step 10:

[1185] The server sends the details of the selected project to the terminal. Specifically, the server forms the project information as an HTTP response and sends it to the input / output device. The output is an HTTP response to the input / output device.

[1186] Step 11:

[1187] The terminal displays the received project details to the user, providing an environment in which the project can be executed through a virtual space simulation device. Specifically, the received details are analyzed and displayed on the screen. The terminal also launches a virtual space simulation to create an environment in which the project can be simulated. For example, a development scenario for a virtual store can be displayed on a VR headset. The output is a simulation environment that the user can directly experience.

[1188] (Application example 1)

[1189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1190] Currently, when users learn, it is difficult for them to acquire specific knowledge or acquire practical operational skills through experience. Furthermore, there is a lack of systems that provide effective feedback and individually optimized learning content quickly. In particular, there is a need for a platform that integrates operational simulation functions in a virtual space to provide a learning environment that is close to real-life experience, allowing users to efficiently acquire skills.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1192] In this invention, the server includes means for providing an operation simulation in a virtual space based on questions and requests input by a user from a terminal, means for displaying the results of the virtual space simulation to the user, and means for analyzing the data and generating optimized feedback based on the user's learning history and performance. This allows the user to experience a simulation in a virtual space that is close to an actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[1193] "User" means an individual or corporation that uses this system.

[1194] A "terminal" is a device, such as a smartphone, tablet, or computer, through which a user inputs a question or request.

[1195] "Question" refers to a learning-related inquiry or doubt that a user inputs via a terminal.

[1196] A "natural language processing model" is a type of artificial intelligence that generates appropriate answers to user questions or input.

[1197] An "answer" is a response to a user's question generated by a natural language processing model.

[1198] The "server" is a device that includes a natural language processing model and manages answers to user questions and simulation results in cooperation with the virtual reality space simulation means.

[1199] "Virtual reality space simulation" is a system that uses virtual reality technology to allow users to experience virtual environments and scenarios.

[1200] "Simulation means" refers to a device or software for executing a virtual reality space simulation.

[1201] "Interface" refers to a user interface that allows a user to operate the simulation means.

[1202] "Operation simulation" refers to a program that allows users to learn through experience how to operate and manage a store in a virtual space.

[1203] "Virtual space" refers to a virtual environment provided by virtual reality technology.

[1204] Analyzing "data" refers to the process of analyzing collected user learning history and performance data to find trends and patterns.

[1205] "Learning history" refers to historical data of the learning activities and simulations that a user has performed up to now.

[1206] "Performance Data" refers to data on the results or outcomes achieved by a user through simulation or learning.

[1207] "Optimized feedback" refers to specific advice and suggestions for next learning that are individually tailored based on the user's learning history and performance.

[1208] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[1209] System configuration

[1210] 1. Terminal

[1211] The device includes an input device and user interface for users to enter questions and project simulation requests. The terminal has the ability to send requests to the server via the Internet. Examples of such devices include smartphones, tablets, and computers.

[1212] 2. Server

[1213] It includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. In addition, the server manages a database to store and analyze learning history and performance data. Specific examples that can be used include Amazon Web Services (AWS) and Google Cloud Platform.

[1214] 3. Virtual Reality Space Simulation Method

[1215] It uses virtual reality technology to provide an environment where users can simulate projects. It stores multiple virtual projects, which are selected randomly or provided according to the user's preferences. It provides 3D simulations using Unity and Unreal Engine.

[1216] Program processing explanation

[1217] Receiving and processing questions

[1218] When a user enters a question into the device and clicks the send button, the server receives the question from the device. The server extracts the content of the question and sends it to a natural language processing model (for example, OpenAI's GPT-3) to generate a response. The generated response is sent from the server to the device and displayed to the user. As a concrete example, a user enters, "What is the best way to manage inventory?"

[1219] Request and provide project simulations

[1220] When a user clicks the project simulation request button on their device, the server receives the request and extracts random or specified project information from the database. The extracted project details are sent from the server to the device and displayed to the user. The user can then perform a project simulation in a virtual reality space. A specific example is when the user selects a "seasonal product campaign" and simulates product placement and advertising in a virtual space.

[1221] Data analysis and feedback

[1222] The server collects and analyzes the user's learning history and performance data, and based on this data, provides optimized feedback and suggests next learning content, allowing users to efficiently acquire new skills and techniques.

[1223] Examples of specific examples and prompts

[1224] For example, if a user asks, "What is the best way to manage inventory?", an example prompt sentence might be:

[1225] Example prompt sentence:

[1226] What is the best way to manage inventory?

[1227] This system allows users to experience simulations in a virtual space that closely resembles the actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[1228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1229] Step 1:

[1230] The user inputs a question or a project simulation request into the terminal and clicks the send button. The terminal then sends the input question or request to the server.

[1231] Input: Questions or requests typed by the user into the device

[1232] Output: Request data from the terminal to the server

[1233] Step 2:

[1234] The server receives questions and requests sent from the device at a specified API endpoint, extracts the content of the question, and sends it to a natural language processing model to generate an answer.

[1235] Input: Request data sent from the terminal

[1236] Output: Question data sent to the natural language processing model and generated answer data

[1237] Step 3:

[1238] The server receives the response from the natural language processing model and extracts the answer, which it then sends back to the terminal and displays to the user.

[1239] Input: Answer data generated from a natural language processing model

[1240] Output: Answer data sent to the user's device

[1241] Step 4:

[1242] When a user clicks the request button of a project simulation, the terminal sends the request to the server as a GET request.

[1243] Input: Project Simulation Request

[1244] Output: GET request sent to the server

[1245] Step 5:

[1246] The server receives a request for a project simulation, extracts relevant project information from the database randomly or based on specified conditions, and transmits the extracted project information to the terminal.

[1247] Input: Project simulation request and project information from the database

[1248] Output: Project details sent to the device

[1249] Step 6:

[1250] The terminal presents the project information received from the server to the user and starts a virtual reality space simulation, and the user experiences the simulation using a virtual reality device.

[1251] Input: Project details sent from the server

[1252] Output: Project details and simulation environment presented to the user

[1253] Step 7:

[1254] During the simulation, the server collects user operation and performance data, which is then saved as a learning history and analyzed later.

[1255] Input: User operation data and performance data

[1256] Output: Learning history data and performance data

[1257] Step 8:

[1258] The server analyzes the collected learning history and performance data and generates personalized feedback, which is provided to the user and includes suggestions for next learning content.

[1259] Input: Learning history data and performance data

[1260] Output: Optimized feedback and learning suggestions provided to the user

[1261] Step 9:

[1262] The user can then view the optimized feedback and suggestions for the next learning activity on their device, and begin a new learning activity. The system repeats this process, helping the user improve their skills.

[1263] Input: Optimized feedback and learning suggestions

[1264] Output: User's new learning activity

[1265] Through these steps, users can efficiently acquire skills in an interactive learning environment.

[1266] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1267] As an embodiment of the present invention, a specific system incorporating an emotion engine and its operation will be described below. This system is an interactive educational platform where users learn through their terminals, and includes a server, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[1268] System configuration

[1269] 1. Terminal

[1270] The terminal is equipped with an input device and a user interface for users to input questions and project simulation requests. The terminal is equipped with an emotion engine and has the ability to recognize emotions from the user's facial expressions, tone of voice, etc.

[1271] 2. Server

[1272] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. The server also receives emotion information sent from the emotion engine and uses it to adjust answers and provide optimal learning content.

[1273] 3. Emotion Engine

[1274] It recognizes the user's emotions and extracts emotional information in real time from the user's facial expressions, tone of voice, etc. The emotion engine is installed on the device, and the data is sent to a server.

[1275] 4. Virtual Reality Space Simulation Method

[1276] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[1277] Program processing explanation

[1278] ChatGPT part

[1279] 1. Receiving and processing inquiries

[1280] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[1281] The server receives this request and extracts the question.

[1282] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[1283] The server receives the response generated by ChatGPT and sends it to the device.

[1284] The terminal receives the response and displays it to the user.

[1285] 2. Acquiring and using emotional information

[1286] While the user is typing a question, the device's emotion engine recognizes the user's emotions in real time.

[1287] The device sends emotional information to the server, which then adjusts the content and tone of the response based on this information.

[1288] The server sends the adjusted answer to the terminal and displays it to the user.

[1289] Metaverse Project Part

[1290] 1. Request and provide project simulation

[1291] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[1292] The server receives the request and randomly selects a project from the database.

[1293] The server extracts detailed information about the selected project and sends it to the device.

[1294] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[1295] 2. Use of emotional information

[1296] The emotion engine recognizes the user's emotions in real time during the project simulation.

[1297] Based on the emotional information, the server dynamically adjusts the difficulty of the project and the learning content provided.

[1298] For example, if the user is feeling stressed, the server will adjust the difficulty level by lowering it or adding explanations.

[1299] Specific examples

[1300] Example 1: A user asks, "I want to know about data analysis in Python."

[1301] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1302] 2. The device sends this question as a POST request to the server's API endpoint.

[1303] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1304] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1305] 5. The server sends this response to the terminal and displays it to the user.

[1306] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[1307] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[1308] Example 2: User requests a new project

[1309] 1. The user clicks the project simulation request button on the terminal.

[1310] 2. The device sends a GET request to the server.

[1311] 3. The server receives the request and randomly selects a project from the database.

[1312] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[1313] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[1314] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[1315] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[1316] As described above, the system of the present invention provides users with an integrated interactive learning environment and practical project experience, and further optimizes learning effectiveness by utilizing users' emotional information. This creates an environment in which working professionals can efficiently acquire new skills and techniques.

[1317] The processing flow will be explained below.

[1318] ChatGPT part

[1319] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[1320] Step 1:

[1321] The user types a question into the terminal, saying "I want to know about data analysis using Python," and clicks the submit button.

[1322] Step 2:

[1323] The device sends the entered question as a POST request to the server's API endpoint.

[1324] Step 3:

[1325] The server receives the POST request and extracts the question from the request data.

[1326] Step 4:

[1327] The server sends the extracted questions to a natural language processing model (ChatGPT).

[1328] Step 5:

[1329] ChatGPT generates answers to questions and sends the answers back to the server.

[1330] Step 6:

[1331] The server receives the response sent back from ChatGPT, formats the response appropriately, and sends it to the device.

[1332] Step 7:

[1333] The terminal displays the received response to the user.

[1334] Use of emotional information

[1335] Acquiring and using emotional information with an emotion engine

[1336] Step 1:

[1337] While the user types their question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[1338] Step 2:

[1339] The emotion information recognized by the device's emotion engine is sent to the server.

[1340] Step 3:

[1341] The server receives the emotional information and adjusts the content and tone of the answers to questions.

[1342] Step 4:

[1343] The server sends the adjusted answer to the terminal and displays it to the user.

[1344] Metaverse Project Part

[1345] Example 2: Processing steps for a project simulation request

[1346] Step 1:

[1347] The user clicks the Request Project Simulation button on the terminal.

[1348] Step 2:

[1349] The terminal sends a project simulation request to the server as a GET request.

[1350] Step 3:

[1351] The server receives a GET request and randomly selects multiple project information from the database.

[1352] Step 4:

[1353] The server extracts detailed information about the selected project.

[1354] Step 5:

[1355] The server converts the project details into JSON format and sends it to the terminal.

[1356] Step 6:

[1357] The terminal displays the received project details information to the user.

[1358] Step 7:

[1359] The user accesses the virtual reality space simulation means via the terminal and experiences the selected project.

[1360] Use of emotional information

[1361] Using emotional information during project simulation with an emotion engine

[1362] Step 1:

[1363] During the project simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[1364] Step 2:

[1365] The emotion information recognized by the device's emotion engine is sent to the server.

[1366] Step 3:

[1367] The server receives emotional information and dynamically adjusts the difficulty of the project and the learning content provided.

[1368] Step 4:

[1369] The server sends the adjusted learning content and support information to the terminal and displays it to the user.

[1370] Example 2

[1371] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1372] Conventional interactive educational systems have struggled to provide learning content that takes into account the user's emotional state. This can lead to problems such as reduced learning efficiency and increased stress. Furthermore, even with project simulations using virtual reality technology, optimal learning outcomes cannot be achieved because the content is not dynamically adjusted based on the user's progress or emotions.

[1373] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means including an emotion engine that recognizes the user's emotions while the user is entering or viewing a question and extracts emotional information in real time, a means for transmitting the emotional information to the server and adjusting the content and tone of the answer, and a means for transmitting the adjusted answer to the terminal and displaying it. This makes it possible to provide learning content tailored to the user's emotional state. In addition, by storing multiple virtual reality space simulation projects in a database, recognizing the user's emotions in real time during the simulation, and dynamically adjusting the content and difficulty of the simulation based on the emotional information, it is possible to provide the user with an optimized learning environment.

[1374] A "terminal" is a device that has an interface and input devices through which a user can enter questions and project simulation requests.

[1375] The "server" is a central device that contains the natural language processing model, generates answers based on user questions, and adjusts learning based on emotional and other data.

[1376] A "natural language processing model" is a software system that includes algorithms that analyze natural language questions entered by users and generate appropriate answers to those questions.

[1377] An "emotion engine" is a device that recognizes a user's facial expressions and tone of voice in real time and extracts emotional information.

[1378] A "virtual reality space simulation means" is a means and device that uses virtual reality technology to provide an environment in which a user can experience a virtual project.

[1379] The "means for adjusting the content and tone of the answer" refers to a mechanism or algorithm for adjusting the answer generated by the natural language processing model based on the emotional information sent from the emotion engine in accordance with the user's emotional state.

[1380] A "project simulation request" is request information that a user sends from a terminal to the system when the user desires to perform a project simulation in a virtual reality space.

[1381] "Study history" is data that records the user's past learning activities and their progress.

[1382] "Performance data" is data that indicates the results and efficiency of a user's learning activities.

[1383] A "database" is a storage device that stores multiple virtual reality space simulation projects and related information and can be accessed as needed.

[1384] The present invention is a system in which a user inputs a question via a terminal, a natural language processing model generates an answer based on the input, and further provides optimized learning content using the user's emotional information. The system includes a server, a terminal, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[1385] System configuration

[1386] Terminal

[1387] The terminal is equipped with an interface and an input device for users to input questions and project simulation requests. The terminal is equipped with an emotion engine that recognizes the user's facial expressions and tone of voice in real time and extracts emotional information.

[1388] server

[1389] The server receives questions entered by users from their devices and sends them to a natural language processing model to generate answers. The generated answers are then sent to the devices and displayed to the users. The server also receives emotional information sent from the emotion engine and adjusts the content and tone of the answers based on this information. The server is also connected to a virtual reality space simulation means and manages the simulation of the virtual project.

[1390] Emotion Engine

[1391] The emotion engine is installed on the device and extracts emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to the server, which then adjusts the content and tone of the answer based on this information.

[1392] Virtual reality space simulation tool

[1393] The virtual reality space simulation means provides an environment in which a user can simulate a project using virtual reality technology. A plurality of virtual projects are stored in a database and are selected randomly or provided according to the user's request.

[1394] Specific examples

[1395] Specific example of question processing

[1396] If a user asks, "I want to know about data analysis using Python," the specific process is as follows:

[1397] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1398] 2. The device sends this question as a POST request to the server's API endpoint.

[1399] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1400] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1401] 5. The server sends this response to the terminal and displays it to the user.

[1402] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[1403] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[1404] Example prompt sentence:

[1405] User: "I want to learn about data analysis using Python."

[1406] Model: "The pandas library is often used for data analysis in Python. Other popular libraries include numpy and matplotlib."

[1407] Project Simulation Examples

[1408] As an example of a user requesting a new project, the "Virtual Store Development" project is selected.

[1409] 1. The user clicks the project simulation request button on the terminal.

[1410] 2. The device sends a GET request to the server.

[1411] 3. The server receives the request and randomly selects a project from the database.

[1412] 4. The server selects "Development of a virtual store" and sends the details to the terminal.

[1413] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[1414] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[1415] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[1416] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, and utilizes users' emotional information to optimize learning effectiveness, creating an environment where users can efficiently acquire new skills and techniques.

[1417] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1418] Step 1: User enters question into terminal

[1419] The user enters a question into the input form on the terminal and clicks the submit button. The input text "I want to know about data analysis using Python" is sent to the terminal.

[1420] Step 2: The device sends the question to the server

[1421] The device sends the user's question as a POST request to the server's specified API endpoint. This request includes the question entered by the user. The server then receives the request.

[1422] Step 3: The server sends the question to the natural language processing model

[1423] The server sends the received question to a natural language processing model (e.g., ChatGPT) to generate an answer. The server inputs the question into the model and receives the generated answer as output. Specifically, the answer generated is "The pandas library is often used for data analysis in Python..."

[1424] Step 4: The server sends the generated answer to the device

[1425] The server sends the generated answer, which includes text obtained from the natural language processing model, to the device, which then receives the answer.

[1426] Step 5: The device displays the answer to the user

[1427] The device displays the answer it receives to the user. This display is the text sent from the server and displayed in the interface. Specifically, the answer displayed is, "The pandas library is often used for data analysis in Python..."

[1428] Step 6: The device recognizes the user's emotions

[1429] While the user is typing a question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is surprised."

[1430] Step 7: The device sends the emotion information to the server.

[1431] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[1432] Step 8: The server adjusts the answer based on the emotional information

[1433] The server adjusts the content and tone of the generated answer based on the emotional information received. In this process, the emotional information is used as input to modify the content and expression of the answer text. For example, if the user is feeling stressed, the server may adjust the answer by adding a gentler tone.

[1434] Step 9: The server sends the adjusted answer to the device.

[1435] The server sends the adjusted answer to the device, which includes the text modified based on the sentiment, and the device receives the adjusted answer.

[1436] Step 10: The device displays the adjusted answer to the user

[1437] The terminal displays the adjusted answer to the user. This display shows the corrected text sent from the server in the interface. Specifically, the answer is displayed in a friendly tone: "The pandas library is often used for data analysis in Python. Try it yourself."

[1438] Step 11: User requests project simulation

[1439] The user clicks the project simulation request button on the terminal. As an input, a request for project simulation is sent to the terminal.

[1440] Step 12: The device sends a request to the server

[1441] The terminal sends the user's project simulation request to the server as a GET request, which includes a request for the project simulation. The server then receives the request.

[1442] Step 13: The server randomly selects a project and extracts its details

[1443] Based on the request received by the server, a virtual project is randomly selected from the database. The server extracts the project details and generates them as output. Specifically, the project "Development of a Virtual Store" is selected.

[1444] Step 14: The server sends the details to the device

[1445] The server transmits the extracted project details to the terminal, including the project description and procedures, and the terminal receives the details.

[1446] Step 15: The terminal displays detailed information to the user and provides a simulation environment

[1447] The terminal displays the received project details to the user, providing a simulation environment in virtual reality space. This display shows the project information sent from the server on the interface. Specifically, details of the "virtual store development" are displayed, and the user experiences the project in virtual reality space.

[1448] Step 16: The device recognizes the user's emotions during the simulation.

[1449] During the simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is excited."

[1450] Step 17: The device sends the emotion information to the server.

[1451] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[1452] Step 18: The server adjusts the content and difficulty of the simulation based on the emotional information.

[1453] The content and difficulty of the simulation are dynamically adjusted based on the emotional information received by the server. In this process, the content of the simulation is dynamically changed using the emotional information as input. For example, if the user feels the simulation is difficult, adjustments such as simplifying the next step are made.

[1454] Step 19: The server sends the adjustments to the device

[1455] The server transmits the adjusted simulation content to the terminal, which includes the simulation data modified based on the emotion. The adjusted content is then received by the terminal.

[1456] Step 20: The device reflects the adjustments in the simulation environment and displays them to the user.

[1457] The terminal reflects the adjusted simulation content in the virtual reality space simulation environment and displays it to the user. This display reflects the corrected simulation data sent from the server on the interface. Specifically, the "Virtual Store Development" simulation with the adjusted difficulty level is displayed.

[1458] Through these processing steps, users can enjoy an interactive learning environment that combines hands-on project experiences. Furthermore, by utilizing emotional information, an environment that optimizes learning effectiveness is provided.

[1459] (Application example 2)

[1460] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1461] Conventional educational platforms provide uniform learning content without considering the user's emotions, making it difficult to maximize individual learning efficiency and comprehension. Furthermore, because they do not take into account the user's stress or fatigue, it can be difficult to continue learning. Furthermore, when providing a simulation environment using virtual reality technology, adjustments are not made based on the user's current emotional state, making it difficult to provide an optimal learning experience.

[1462] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question input by a user from a terminal, means for transmitting the question to a natural language processing model to obtain an answer, and means for transmitting the answer to the terminal and displaying it to the user. This realizes a system including emotion recognition means for recognizing a user's emotions in real time and transmitting emotional information to the server, and means for dynamically adjusting answers and learning content based on the emotional information. This makes it possible to provide optimal and personalized learning content and a simulation environment according to the user's current emotional state.

[1463] A "terminal" is a device equipped with an input device and a user interface for a user to input questions and project simulation requests and obtain emotion information.

[1464] A "natural language processing model" is an algorithm or system that generates answers to questions entered by users, and specifically refers to a model that understands natural language and generates responses.

[1465] A "server" is a central computer that transmits, receives, and processes data via a network, and in the present invention includes a natural language processing model, a virtual reality space simulation means, and a device that processes emotional information.

[1466] "Virtual reality space simulation means" refers to technology or devices that provide an environment in which users can simulate projects using virtual reality technology.

[1467] "Emotion recognition means" refers to technology or devices that extract emotional information in real time from a user's facial expressions, tone of voice, etc., and is responsible for transmitting this information to a server.

[1468] An "interface" refers to a connection means or operating device that allows a user to operate a simulation means, and is designed to allow intuitive and efficient operation.

[1469] "Learning History and Performance Data" refers to data that records a user's past learning activities and the results of those activities, and is used to provide individually optimized learning content.

[1470] "Personalized learning content" refers to learning materials and assignments that are dynamically tailored based on a user's learning history and emotional state.

[1471] "Project simulation" refers to a virtual project that simulates real-world environments and scenarios in a virtual reality space, allowing users to experience and learn from them.

[1472] A specific embodiment for carrying out the present invention includes the following system and its operation, which enables a user to receive personalized learning content and virtual reality project simulations while having their emotions recognized in real time.

[1473] System configuration

[1474] 1. Terminal

[1475] The device is equipped with an input device and a user interface for users to input questions and project simulation requests. The device is also equipped with emotion recognition means to extract emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to a server and used to dynamically adjust the learning content.

[1476] 2. Server

[1477] It includes a natural language processing model (e.g., ChatGPT) and generates answers to user questions. The server receives emotional information sent from the emotion recognition means and dynamically adjusts the answers and learning content. It is also connected to the virtual reality space simulation means and manages the simulation of the virtual project.

[1478] 3. Emotion recognition means

[1479] Emotional information is extracted in real time from the user's facial expressions, tone of voice, etc. This is installed on the device, and the data is sent to the server. Based on the emotional information, the server adjusts the answers and learning content.

[1480] 4. Virtual Reality Space Simulation Method

[1481] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored in a database, and these are provided randomly or according to the user's preferences.

[1482] Program processing explanation

[1483] The server receives the user's emotional information in real time and dynamically optimizes the learning content based on it. To achieve this, the device is equipped with emotion recognition hardware such as a camera and microphone. The software uses natural language processing models (such as ChatGPT) and emotion recognition algorithms (such as EmotionEngine). The server processes the user's questions, generates optimal answers based on the emotional information, and recommends learning content.

[1484] Specific examples

[1485] For example, if a user enters a question such as "I want to know about data analysis using Python," the system operates as follows:

[1486] 1. The user enters "I want to know about data analysis using Python" into the input field of the terminal and submits it.

[1487] 2. The device sends this question to the server.

[1488] 3. The server receives the question and generates an answer using a natural language processing model (ChatGPT).

[1489] 4. The server sends the generated answer to the terminal and displays it to the user.

[1490] 5. At the same time, the device's emotion recognition means analyzes the user's emotion information and sends it to the server.

[1491] 6. The server dynamically adjusts answers and learning content based on emotional information.

[1492] When a user requests a project simulation, the server provides the user with a randomly selected simulation project and adjusts the difficulty level appropriately based on emotional information. For example, if the project "Development of a Virtual Store" is selected and the user feels stressed during the project simulation, the system will lower the difficulty level or add an explanation.

[1493] Prompt Sentence Examples

[1494] Below are some example prompts for generative AI models:

[1495] "What are some specific ways to make effective learning content recommendations when users are fatigued?"

[1496] "Please provide detailed steps on how to personalize educational content based on emotion recognition."

[1497] As described above, the embodiments of the present invention utilize the user's emotional information to provide individually optimized learning content and virtual reality project simulation, thereby maximizing the learning effect and improving the user's learning experience.

[1498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1499] Step 1:

[1500] The user enters a question into the input field of the terminal and clicks the send button. This input field accepts the user's question in text format. The entered question is sent from the terminal to the server. The input is text data, and the output is an HTTP POST request to the server.

[1501] Step 2:

[1502] The server receives a question sent by a user. The server sends this question to a natural language processing model (ChatGPT) to generate an answer. The input is the text data of the question sent by the user, and the output is an API request to the natural language processing model and its response text.

[1503] Step 3:

[1504] The server receives a response from the natural language processing model and sends it to the terminal. The input is the data returned from the natural language processing model, and the output is text data as an HTTP response to the terminal.

[1505] Step 4:

[1506] The terminal receives the response and displays it to the user. Specifically, it displays the response text on the user interface. In this step, the input is the response data from the server, and the output is the text displayed on the terminal screen.

[1507] Step 5:

[1508] The device's emotion recognition means analyzes the user's facial expressions and tone of voice in real time. Emotional data is acquired using a camera or microphone and analyzed by an emotion recognition algorithm. The input is raw data from the camera or microphone, and the output is analyzed emotional information.

[1509] Step 6:

[1510] The device sends the acquired emotion information to the server. The input is emotion data from the emotion recognition means, and the output is emotion information as an HTTP POST request to the server.

[1511] Step 7:

[1512] The server receives emotional information and dynamically adjusts the tone and content of the answer based on that information. For example, if the user is feeling stressed, the server can make the answer gentler or add a brief explanation. The input is emotional information, and the output is adjusted text data.

[1513] Step 8:

[1514] The server sends the adjusted answers or learning content to the terminal and displays them again to the user. The input is the adjusted text data, and the output is data as an HTTP response to the terminal.

[1515] Step 9:

[1516] A user requests a project simulation. This request is sent from the terminal to the server. The input is the simulation request from the user, and the output is an HTTP GET request to the server.

[1517] Step 10:

[1518] The server receives a request for a project simulation and selects an appropriate project from the database. The input is the simulation request and the output is the selected project information.

[1519] Step 11:

[1520] The server sends the selected project information to the terminal. The input is the project information, and the output is the simulation data as an HTTP response to the terminal.

[1521] Step 12:

[1522] The terminal receives the project information and provides it to the user in the virtual reality space. The user executes the project in the virtual reality environment. The input is the project information, and the output is a simulation display in the virtual reality environment.

[1523] Step 13:

[1524] During the project simulation, the emotion recognition means recognizes the user's emotions in real time and sends them to the server. The input is raw data from the camera and microphone, and the output is analyzed emotional information.

[1525] Step 14:

[1526] The server dynamically adjusts the project difficulty and the learning content provided based on the emotional information, and provides it to the user again. The input is the emotional information and project progress information, and the output is the adjusted simulation content.

[1527] As a result, a system is realized that provides optimal learning experiences and project simulations according to the user's emotional state.

[1528] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1529] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1530] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1531] [Fourth embodiment]

[1532] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1533] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1534] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1535] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1536] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1537] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1538] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1539] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1540] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1541] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1542] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1543] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1544] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1545] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[1546] System configuration

[1547] 1. Terminal

[1548] The terminal has an input device and a user interface for users to input questions and project simulation requests, and has the ability to send requests to a server via the Internet.

[1549] 2. Server

[1550] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. Furthermore, the server manages a database, storing and analyzing learning history and performance data.

[1551] 3. Virtual Reality Space Simulation Method

[1552] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[1553] Program processing explanation

[1554] ChatGPT part

[1555] 1. Receiving and processing inquiries

[1556] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[1557] The server receives this request and extracts the question.

[1558] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[1559] The server receives the response generated by ChatGPT and sends it to the device.

[1560] The terminal receives the response and displays it to the user.

[1561] Metaverse Project Part

[1562] 1. Request and provide project simulation

[1563] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[1564] The server receives the request and extracts random or specified project information from the database.

[1565] The server sends details of the selected project to the terminal.

[1566] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[1567] Specific examples

[1568] Example 1: A user asks, "I want to know about data analysis in Python."

[1569] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1570] 2. The device sends this question as a POST request to the server's API endpoint.

[1571] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1572] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1573] 5. The server sends this response to the terminal and displays it to the user.

[1574] Example 2: User requests a new project

[1575] 1. The user clicks the project simulation request button on the terminal.

[1576] 2. The device sends a GET request to the server.

[1577] 3. The server receives the request and randomly selects a project from the database.

[1578] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[1579] 5. The device displays this information to the user and provides a project simulation in a virtual reality space.

[1580] As described above, the system of the present invention provides users with an interactive learning environment and practical project experience, thereby enhancing learning effectiveness and creating an environment where working professionals can efficiently acquire new skills and techniques.

[1581] The processing flow will be explained below.

[1582] ChatGPT part

[1583] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[1584] Step 1:

[1585] The user types a question into the terminal and clicks the send button.

[1586] Step 2:

[1587] The terminal sends the entered question to the server as a POST request.

[1588] Step 3:

[1589] The server receives the request and extracts the question.

[1590] Step 4:

[1591] The server sends the extracted questions to the natural language processing model (ChatGPT).

[1592] Step 5:

[1593] ChatGPT generates a response to the question and sends it back to the server.

[1594] Step 6:

[1595] The server receives the response sent back from ChatGPT, converts it to JSON format, and sends it back to the device.

[1596] Step 7:

[1597] The terminal displays the response received from the server to the user.

[1598] Metaverse Project Part

[1599] Example 2: Processing steps for a project simulation request

[1600] Step 1:

[1601] The user clicks the Request Project Simulation button on the terminal.

[1602] Step 2:

[1603] The terminal sends a project simulation request to the server as a GET request.

[1604] Step 3:

[1605] The server receives the request and randomly selects a project from the database.

[1606] Step 4:

[1607] The server extracts the details of the selected project.

[1608] Step 5:

[1609] The server converts the project details into JSON format and sends it back to the device.

[1610] Step 6:

[1611] The terminal displays detailed information about the project received from the server to the user.

[1612] Step 7:

[1613] The user accesses the virtual reality space simulation means via a terminal and experiences the project.

[1614] Example 1

[1615] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1616] In today's educational environment, learners are expected to efficiently acquire new skills and knowledge. However, existing educational systems struggle to provide personalized instruction and practical experiences that are tailored to learners' specific needs. Furthermore, traditional methods struggle to support learning through real-time dialogue or simulated learning using virtual reality. This hinders the improvement of learning outcomes and the acquisition of practical skills.

[1617] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1618] In this invention, the server includes means for receiving data input by a user from an input / output device, means for transmitting the data to a natural language processing algorithm to obtain a response, and means for transmitting the response to the input / output device and displaying it to the user. This allows users to learn through question and answer sessions using a natural language processing algorithm in real time. The server also includes means for storing multiple virtual space simulation projects in a storage device and providing randomly selected project information to the user, enabling learning through practical project experience. Furthermore, the server includes means for analyzing a user's learning history and performance data and providing individually optimized learning content, enabling personalized education and significantly improving learning effectiveness.

[1619] 1. An "input / output device" is a device that allows a user to input data and display information received from a server.

[1620] 2. "Natural language processing algorithms" are machine learning models for understanding text data and generating appropriate responses.

[1621] 3. "Information Processing Device" means a computer system for receiving and processing data and generating and transmitting responses.

[1622] 4. "Virtual space simulation device" means a device that uses virtual reality technology to provide users with a simulation experience.

[1623] 5. "Interface" means the means by which a user operates a virtual reality simulation device.

[1624] 6. "Storage device" means a hardware device for storing data and retrieving that data as needed.

[1625] 7. "Project Information" means the specific scenarios and data used in the virtual world simulation.

[1626] 8. "Learning History" means a record of a user's previous learning activities.

[1627] 9. "Performance Data" means information that indicates a User's learning results and progress.

[1628] A specific system and its operation that are the subject of the present invention will be described below. The system is an interactive educational platform where users learn through input / output devices, and includes an information processing device and a virtual space simulation device.

[1629] System configuration

[1630] 1. Input / Output Devices

[1631] An input / output device is a device that allows users to input data and display information. Examples include personal computers, tablets, and smartphones. An input / output device has the ability to send requests to an information processing device via the Internet.

[1632] 2. Information Processing Device

[1633] It includes a natural language processing algorithm and generates responses to user questions. Specifically, generative AI models such as ChatGPT are used. The information processing device is connected to the virtual space simulation device and manages the virtual project simulation. The information processing device also manages the storage device, accumulating and analyzing learning history and performance data.

[1634] 3. Virtual space simulation device

[1635] It uses virtual reality technology to provide an environment in which users can simulate projects. Specifically, a VR headset is used. Multiple virtual projects are stored in a storage device and are selected randomly or provided according to the user's preferences.

[1636] Program processing explanation

[1637] Question-answering part of natural language processing

[1638] 1. Enter and submit your question

[1639] The user enters a question into the input / output device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and submits it.

[1640] 2. Receiving and processing inquiries

[1641] The input / output device transmits the input question to the information processing device as a POST request.

[1642] An information processing device receives the request, extracts the question content, sends it to a natural language processing algorithm, and generates a response.

[1643] 3. Receiving and Displaying Responses

[1644] The information processing device sends the generated response to the input / output device, which receives the response and displays it to the user. For example, it displays a response such as, "The pandas library is often used for data analysis in Python..."

[1645] Requesting and providing virtual space simulations

[1646] 1. Request and submit a simulation

[1647] A user clicks a project simulation request button on an input / output device. For example, the user thinks, "I want to simulate a new project," and presses the corresponding button.

[1648] 2. Obtaining project information

[1649] The input / output device transmits a request to the information processing device as a GET request.

[1650] The information processing device receives the request and extracts random or specified project information from the storage device, for example, selecting the "Virtual Store Development" project.

[1651] 3. Sending and viewing project information

[1652] The information processing device transmits details of the selected project to the input / output device.

[1653] The input / output device displays the received project details to the user, and provides an environment in which the project can be executed through a virtual space simulation device. For example, a development scenario for a virtual store is displayed on a VR headset.

[1654] Specific examples

[1655] Example 1: A user asks, "I want to know about data analysis in Python."

[1656] 1. The user enters "I want to know about data analysis using Python" into the input form on the input / output device and clicks the submit button.

[1657] 2. The input / output device sends this question as a POST request to the API endpoint of the information processing device.

[1658] 3. An information processor receives the question and sends it to a natural language processing algorithm to generate a response.

[1659] 4. The information processing device receives a response from ChatGPT such as, "The pandas library is often used for data analysis in Python..."

[1660] 5. The information processing device sends this response to the input / output device and displays it to the user.

[1661] Example 2: User requests a new project

[1662] 1. The user clicks the project simulation request button on the I / O device.

[1663] 2. The input / output device sends the data as a GET request to the information processing device.

[1664] 3. The information processing device receives the request and randomly selects a project from the storage device.

[1665] 4. The information processing device selects "Development of a virtual store" and sends the detailed information to the input / output device.

[1666] 5. The input / output device displays this information to the user, providing a project simulation in the virtual space simulation device.

[1667] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, enhancing learning effectiveness and enabling them to efficiently acquire new skills and techniques.

[1668] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1669] Step 1:

[1670] The user enters a question into the input form of the I / O device and clicks the "Submit" button. For example, "I want to know about data analysis using Python" and then presses the "Submit" button. The input data is stored in the memory of the I / O device and used for subsequent processing.

[1671] Step 2:

[1672] The device sends the entered question to the server as a POST request. Specifically, it sends the entered text data as an HTTP request to the API endpoint. The output is the HTTP request received by the server.

[1673] Step 3:

[1674] The server receives the request and extracts the question. It extracts pure text data from the HTTP request body. This process analyzes the question from the received request and converts the request content into a format suitable for natural language processing algorithms. The output is the extracted text data.

[1675] Step 4:

[1676] The server sends the extracted question to a natural language processing algorithm (ChatGPT) to generate a response. Specifically, the extracted question is passed to ChatGPT as a prompt sentence, which is then processed using natural language processing. The generated response data is the output.

[1677] Step 5:

[1678] The server receives the generated response and sends a reply to the input / output device. Specifically, it converts the response from the natural language processing algorithm into an HTTP response format and sends it to the input / output device. The output is an HTTP response to the input / output device.

[1679] Step 6:

[1680] The terminal receives the response and displays it to the user. Specifically, it analyzes the received HTTP response data and renders the text data on the screen. It must be displayed in a format that is easy for the user to read. The output is text that is displayed on the user's screen.

[1681] Step 7:

[1682] A user clicks a project simulation request button on an input / output device. For example, if the user thinks, "I want to simulate a new project," he / she presses the corresponding button. The input data is the user's request.

[1683] Step 8:

[1684] The terminal sends a request to the server as a GET request. Specifically, it sends the request data generated based on the user's action as an HTTP GET request. The output is an HTTP request received by the server.

[1685] Step 9:

[1686] The server receives the request and extracts random or specified project information from storage. It performs a database query to retrieve the appropriate project information. For example, select "Virtual Store Development." The output is the extracted project information.

[1687] Step 10:

[1688] The server sends the details of the selected project to the terminal. Specifically, the server forms the project information as an HTTP response and sends it to the input / output device. The output is an HTTP response to the input / output device.

[1689] Step 11:

[1690] The terminal displays the received project details to the user, providing an environment in which the project can be executed through a virtual space simulation device. Specifically, the received details are analyzed and displayed on the screen. The terminal also launches a virtual space simulation to create an environment in which the project can be simulated. For example, a development scenario for a virtual store can be displayed on a VR headset. The output is a simulation environment that the user can directly experience.

[1691] (Application example 1)

[1692] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1693] Currently, when users learn, it is difficult for them to acquire specific knowledge or acquire practical operational skills through experience. Furthermore, there is a lack of systems that provide effective feedback and individually optimized learning content quickly. In particular, there is a need for a platform that integrates operational simulation functions in a virtual space to provide a learning environment that is close to real-life experience, allowing users to efficiently acquire skills.

[1694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1695] In this invention, the server includes means for providing an operation simulation in a virtual space based on questions and requests input by a user from a terminal, means for displaying the results of the virtual space simulation to the user, and means for analyzing the data and generating optimized feedback based on the user's learning history and performance. This allows the user to experience a simulation in a virtual space that is close to an actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[1696] "User" means an individual or corporation that uses this system.

[1697] A "terminal" is a device, such as a smartphone, tablet, or computer, through which a user inputs a question or request.

[1698] "Question" refers to a learning-related inquiry or doubt that a user inputs via a terminal.

[1699] A "natural language processing model" is a type of artificial intelligence that generates appropriate answers to user questions or input.

[1700] An "answer" is a response to a user's question generated by a natural language processing model.

[1701] The "server" is a device that includes a natural language processing model and manages answers to user questions and simulation results in cooperation with the virtual reality space simulation means.

[1702] "Virtual reality space simulation" is a system that uses virtual reality technology to allow users to experience virtual environments and scenarios.

[1703] "Simulation means" refers to a device or software for executing a virtual reality space simulation.

[1704] "Interface" refers to a user interface that allows a user to operate the simulation means.

[1705] "Operation simulation" refers to a program that allows users to learn through experience how to operate and manage a store in a virtual space.

[1706] "Virtual space" refers to a virtual environment provided by virtual reality technology.

[1707] Analyzing "data" refers to the process of analyzing collected user learning history and performance data to find trends and patterns.

[1708] "Learning history" refers to historical data of the learning activities and simulations that a user has performed up to now.

[1709] "Performance Data" refers to data on the results or outcomes achieved by a user through simulation or learning.

[1710] "Optimized feedback" refers to specific advice and suggestions for next learning that are individually tailored based on the user's learning history and performance.

[1711] As an embodiment of the present invention, a specific system and its operation will be described below. The system is an interactive educational platform where users learn through terminals, and includes a server and virtual reality space simulation means.

[1712] System configuration

[1713] 1. Terminal

[1714] The device includes an input device and user interface for users to enter questions and project simulation requests. The terminal has the ability to send requests to the server via the Internet. Examples of such devices include smartphones, tablets, and computers.

[1715] 2. Server

[1716] It includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. In addition, the server manages a database to store and analyze learning history and performance data. Specific examples that can be used include Amazon Web Services (AWS) and Google Cloud Platform.

[1717] 3. Virtual Reality Space Simulation Method

[1718] It uses virtual reality technology to provide an environment where users can simulate projects. It stores multiple virtual projects, which are selected randomly or provided according to the user's preferences. It provides 3D simulations using Unity and Unreal Engine.

[1719] Program processing explanation

[1720] Receiving and processing questions

[1721] When a user enters a question into the device and clicks the send button, the server receives the question from the device. The server extracts the content of the question and sends it to a natural language processing model (for example, OpenAI's GPT-3) to generate a response. The generated response is sent from the server to the device and displayed to the user. As a concrete example, a user enters, "What is the best way to manage inventory?"

[1722] Request and provide project simulations

[1723] When a user clicks the project simulation request button on their device, the server receives the request and extracts random or specified project information from the database. The extracted project details are sent from the server to the device and displayed to the user. The user can then perform a project simulation in a virtual reality space. A specific example is when the user selects a "seasonal product campaign" and simulates product placement and advertising in a virtual space.

[1724] Data analysis and feedback

[1725] The server collects and analyzes the user's learning history and performance data, and based on this data, provides optimized feedback and suggests next learning content, allowing users to efficiently acquire new skills and techniques.

[1726] Examples of specific examples and prompts

[1727] For example, if a user asks, "What is the best way to manage inventory?", an example prompt sentence might be:

[1728] Example prompt sentence:

[1729] What is the best way to manage inventory?

[1730] This system allows users to experience simulations in a virtual space that closely resembles the actual operating environment, and by receiving individually optimized feedback, enables efficient learning.

[1731] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1732] Step 1:

[1733] The user inputs a question or a project simulation request into the terminal and clicks the send button. The terminal then sends the input question or request to the server.

[1734] Input: Questions or requests typed by the user into the device

[1735] Output: Request data from the terminal to the server

[1736] Step 2:

[1737] The server receives questions and requests sent from the device at a specified API endpoint, extracts the content of the question, and sends it to a natural language processing model to generate an answer.

[1738] Input: Request data sent from the terminal

[1739] Output: Question data sent to the natural language processing model and generated answer data

[1740] Step 3:

[1741] The server receives the response from the natural language processing model and extracts the answer, which it then sends back to the terminal and displays to the user.

[1742] Input: Answer data generated from a natural language processing model

[1743] Output: Answer data sent to the user's device

[1744] Step 4:

[1745] When a user clicks the request button of a project simulation, the terminal sends the request to the server as a GET request.

[1746] Input: Project Simulation Request

[1747] Output: GET request sent to the server

[1748] Step 5:

[1749] The server receives a request for a project simulation, extracts relevant project information from the database randomly or based on specified conditions, and transmits the extracted project information to the terminal.

[1750] Input: Project simulation request and project information from the database

[1751] Output: Project details sent to the device

[1752] Step 6:

[1753] The terminal presents the project information received from the server to the user and starts a virtual reality space simulation, and the user experiences the simulation using a virtual reality device.

[1754] Input: Project details sent from the server

[1755] Output: Project details and simulation environment presented to the user

[1756] Step 7:

[1757] During the simulation, the server collects user operation and performance data, which is then saved as a learning history and analyzed later.

[1758] Input: User operation data and performance data

[1759] Output: Learning history data and performance data

[1760] Step 8:

[1761] The server analyzes the collected learning history and performance data and generates personalized feedback, which is provided to the user and includes suggestions for next learning content.

[1762] Input: Learning history data and performance data

[1763] Output: Optimized feedback and learning suggestions provided to the user

[1764] Step 9:

[1765] The user can then view the optimized feedback and suggestions for the next learning activity on their device, and begin a new learning activity. The system repeats this process, helping the user improve their skills.

[1766] Input: Optimized feedback and learning suggestions

[1767] Output: User's new learning activity

[1768] Through these steps, users can efficiently acquire skills in an interactive learning environment.

[1769] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1770] As an embodiment of the present invention, a specific system incorporating an emotion engine and its operation will be described below. This system is an interactive educational platform where users learn through their terminals, and includes a server, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[1771] System configuration

[1772] 1. Terminal

[1773] The terminal is equipped with an input device and a user interface for users to input questions and project simulation requests. The terminal is equipped with an emotion engine and has the ability to recognize emotions from the user's facial expressions, tone of voice, etc.

[1774] 2. Server

[1775] The system includes a natural language processing model and generates answers to user questions. The server is connected to the virtual reality space simulation means and manages the simulation of the virtual project. The server also receives emotion information sent from the emotion engine and uses it to adjust answers and provide optimal learning content.

[1776] 3. Emotion Engine

[1777] It recognizes the user's emotions and extracts emotional information in real time from the user's facial expressions, tone of voice, etc. The emotion engine is installed on the device, and the data is sent to a server.

[1778] 4. Virtual Reality Space Simulation Method

[1779] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored and either randomly selected or provided according to the user's preferences.

[1780] Program processing explanation

[1781] ChatGPT part

[1782] 1. Receiving and processing inquiries

[1783] When a user enters a question into the terminal and clicks the submit button, the terminal sends the question as a POST request to the specified API endpoint on the server.

[1784] The server receives this request and extracts the question.

[1785] The server sends the extracted questions to a natural language processing model (ChatGPT) to generate answers.

[1786] The server receives the response generated by ChatGPT and sends it to the device.

[1787] The terminal receives the response and displays it to the user.

[1788] 2. Acquiring and using emotional information

[1789] While the user is typing a question, the device's emotion engine recognizes the user's emotions in real time.

[1790] The device sends emotional information to the server, which then adjusts the content and tone of the response based on this information.

[1791] The server sends the adjusted answer to the terminal and displays it to the user.

[1792] Metaverse Project Part

[1793] 1. Request and provide project simulation

[1794] When a user clicks the request button for the project simulation from a terminal, the terminal sends a GET request to the server.

[1795] The server receives the request and randomly selects a project from the database.

[1796] The server extracts detailed information about the selected project and sends it to the device.

[1797] The terminal displays the received project details information to the user and provides an environment in which the project can be carried out through a virtual reality space simulation means.

[1798] 2. Use of emotional information

[1799] The emotion engine recognizes the user's emotions in real time during the project simulation.

[1800] Based on the emotional information, the server dynamically adjusts the difficulty of the project and the learning content provided.

[1801] For example, if the user is feeling stressed, the server will adjust the difficulty level by lowering it or adding explanations.

[1802] Specific examples

[1803] Example 1: A user asks, "I want to know about data analysis in Python."

[1804] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1805] 2. The device sends this question as a POST request to the server's API endpoint.

[1806] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1807] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1808] 5. The server sends this response to the terminal and displays it to the user.

[1809] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[1810] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[1811] Example 2: User requests a new project

[1812] 1. The user clicks the project simulation request button on the terminal.

[1813] 2. The device sends a GET request to the server.

[1814] 3. The server receives the request and randomly selects a project from the database.

[1815] 4. For example, the server selects "Development of a virtual store" and sends the details to the terminal.

[1816] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[1817] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[1818] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[1819] As described above, the system of the present invention provides users with an integrated interactive learning environment and practical project experience, and further optimizes learning effectiveness by utilizing users' emotional information. This creates an environment in which working professionals can efficiently acquire new skills and techniques.

[1820] The processing flow will be explained below.

[1821] ChatGPT part

[1822] Example 1: Processing steps for the question "I want to know about data analysis using Python"

[1823] Step 1:

[1824] The user types a question into the terminal, saying "I want to know about data analysis using Python," and clicks the submit button.

[1825] Step 2:

[1826] The device sends the entered question as a POST request to the server's API endpoint.

[1827] Step 3:

[1828] The server receives the POST request and extracts the question from the request data.

[1829] Step 4:

[1830] The server sends the extracted questions to a natural language processing model (ChatGPT).

[1831] Step 5:

[1832] ChatGPT generates answers to questions and sends the answers back to the server.

[1833] Step 6:

[1834] The server receives the response sent back from ChatGPT, formats the response appropriately, and sends it to the device.

[1835] Step 7:

[1836] The terminal displays the received response to the user.

[1837] Use of emotional information

[1838] Acquiring and using emotional information with an emotion engine

[1839] Step 1:

[1840] While the user types their question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[1841] Step 2:

[1842] The emotion information recognized by the device's emotion engine is sent to the server.

[1843] Step 3:

[1844] The server receives the emotional information and adjusts the content and tone of the answers to questions.

[1845] Step 4:

[1846] The server sends the adjusted answer to the terminal and displays it to the user.

[1847] Metaverse Project Part

[1848] Example 2: Processing steps for a project simulation request

[1849] Step 1:

[1850] The user clicks the Request Project Simulation button on the terminal.

[1851] Step 2:

[1852] The terminal sends a project simulation request to the server as a GET request.

[1853] Step 3:

[1854] The server receives a GET request and randomly selects multiple project information from the database.

[1855] Step 4:

[1856] The server extracts detailed information about the selected project.

[1857] Step 5:

[1858] The server converts the project details into JSON format and sends it to the terminal.

[1859] Step 6:

[1860] The terminal displays the received project details information to the user.

[1861] Step 7:

[1862] The user accesses the virtual reality space simulation means via the terminal and experiences the selected project.

[1863] Use of emotional information

[1864] Using emotional information during project simulation with an emotion engine

[1865] Step 1:

[1866] During the project simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time.

[1867] Step 2:

[1868] The emotion information recognized by the device's emotion engine is sent to the server.

[1869] Step 3:

[1870] The server receives emotional information and dynamically adjusts the difficulty of the project and the learning content provided.

[1871] Step 4:

[1872] The server sends the adjusted learning content and support information to the terminal and displays it to the user.

[1873] Example 2

[1874] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1875] Conventional interactive educational systems have struggled to provide learning content that takes into account the user's emotional state. This can lead to problems such as reduced learning efficiency and increased stress. Furthermore, even with project simulations using virtual reality technology, optimal learning outcomes cannot be achieved because the content is not dynamically adjusted based on the user's progress or emotions.

[1876] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means including an emotion engine that recognizes the user's emotions while the user is entering or viewing a question and extracts emotional information in real time, a means for transmitting the emotional information to the server and adjusting the content and tone of the answer, and a means for transmitting the adjusted answer to the terminal and displaying it. This makes it possible to provide learning content tailored to the user's emotional state. In addition, by storing multiple virtual reality space simulation projects in a database, recognizing the user's emotions in real time during the simulation, and dynamically adjusting the content and difficulty of the simulation based on the emotional information, it is possible to provide the user with an optimized learning environment.

[1877] A "terminal" is a device that has an interface and input devices through which a user can enter questions and project simulation requests.

[1878] The "server" is a central device that contains the natural language processing model, generates answers based on user questions, and adjusts learning based on emotional and other data.

[1879] A "natural language processing model" is a software system that includes algorithms that analyze natural language questions entered by users and generate appropriate answers to those questions.

[1880] An "emotion engine" is a device that recognizes a user's facial expressions and tone of voice in real time and extracts emotional information.

[1881] A "virtual reality space simulation means" is a means and device that uses virtual reality technology to provide an environment in which a user can experience a virtual project.

[1882] The "means for adjusting the content and tone of the answer" refers to a mechanism or algorithm for adjusting the answer generated by the natural language processing model based on the emotional information sent from the emotion engine in accordance with the user's emotional state.

[1883] A "project simulation request" is request information that a user sends from a terminal to the system when the user desires to perform a project simulation in a virtual reality space.

[1884] "Study history" is data that records the user's past learning activities and their progress.

[1885] "Performance data" is data that indicates the results and efficiency of a user's learning activities.

[1886] A "database" is a storage device that stores multiple virtual reality space simulation projects and related information and can be accessed as needed.

[1887] The present invention is a system in which a user inputs a question via a terminal, a natural language processing model generates an answer based on the input, and further provides optimized learning content using the user's emotional information. The system includes a server, a terminal, a natural language processing model, an emotion engine, and a virtual reality space simulation means.

[1888] System configuration

[1889] Terminal

[1890] The terminal is equipped with an interface and an input device for users to input questions and project simulation requests. The terminal is equipped with an emotion engine that recognizes the user's facial expressions and tone of voice in real time and extracts emotional information.

[1891] server

[1892] The server receives questions entered by users from their devices and sends them to a natural language processing model to generate answers. The generated answers are then sent to the devices and displayed to the users. The server also receives emotional information sent from the emotion engine and adjusts the content and tone of the answers based on this information. The server is also connected to a virtual reality space simulation means and manages the simulation of the virtual project.

[1893] Emotion Engine

[1894] The emotion engine is installed on the device and extracts emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to the server, which then adjusts the content and tone of the answer based on this information.

[1895] Virtual reality space simulation tool

[1896] The virtual reality space simulation means provides an environment in which a user can simulate a project using virtual reality technology. A plurality of virtual projects are stored in a database and are selected randomly or provided according to the user's request.

[1897] Specific examples

[1898] Specific example of question processing

[1899] If a user asks, "I want to know about data analysis using Python," the specific process is as follows:

[1900] 1. The user enters "I want to learn about data analysis using Python" into the input form on the terminal and clicks the submit button.

[1901] 2. The device sends this question as a POST request to the server's API endpoint.

[1902] 3. The server receives the question and sends it to a natural language processing model to generate an answer.

[1903] 4. You receive a response from ChatGPT saying something like, "The pandas library is often used for data analysis in Python..."

[1904] 5. The server sends this response to the terminal and displays it to the user.

[1905] 6. At the same time, the device's emotion engine analyzes the user's facial expressions and tone of voice and sends emotional information to the server.

[1906] 7. The server adjusts the tone of the answer based on the emotional information and displays it in the most appropriate way for the user.

[1907] Example prompt sentence:

[1908] User: "I want to learn about data analysis using Python."

[1909] Model: "The pandas library is often used for data analysis in Python. Other popular libraries include numpy and matplotlib."

[1910] Project Simulation Examples

[1911] As an example of a user requesting a new project, the "Virtual Store Development" project is selected.

[1912] 1. The user clicks the project simulation request button on the terminal.

[1913] 2. The device sends a GET request to the server.

[1914] 3. The server receives the request and randomly selects a project from the database.

[1915] 4. The server selects "Development of a virtual store" and sends the details to the terminal.

[1916] 5. The terminal displays this information to the user and provides a project simulation in a virtual reality space.

[1917] 6. During the project simulation, the emotion engine recognizes the user's emotions in real time and transmits them to the server.

[1918] 7. The server dynamically adjusts the difficulty level and learning content of the project based on the emotional information and provides it to the user.

[1919] This system allows users to enjoy an interactive learning environment that combines hands-on project experiences, and utilizes users' emotional information to optimize learning effectiveness, creating an environment where users can efficiently acquire new skills and techniques.

[1920] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1921] Step 1: User enters question into terminal

[1922] The user enters a question into the input form on the terminal and clicks the submit button. The input text "I want to know about data analysis using Python" is sent to the terminal.

[1923] Step 2: The device sends the question to the server

[1924] The device sends the user's question as a POST request to the server's specified API endpoint. This request includes the question entered by the user. The server then receives the request.

[1925] Step 3: The server sends the question to the natural language processing model

[1926] The server sends the received question to a natural language processing model (e.g., ChatGPT) to generate an answer. The server inputs the question into the model and receives the generated answer as output. Specifically, the answer generated is "The pandas library is often used for data analysis in Python..."

[1927] Step 4: The server sends the generated answer to the device

[1928] The server sends the generated answer, which includes text obtained from the natural language processing model, to the terminal, which then receives the answer.

[1929] Step 5: The device displays the answer to the user

[1930] The device displays the answer it receives to the user. This display is the text sent from the server and displayed in the interface. Specifically, the answer displayed is, "The pandas library is often used for data analysis in Python..."

[1931] Step 6: The device recognizes the user's emotions

[1932] While the user is typing a question, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is surprised."

[1933] Step 7: The device sends the emotion information to the server.

[1934] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[1935] Step 8: The server adjusts the answer based on the emotional information

[1936] The server adjusts the content and tone of the generated answer based on the emotional information received. In this process, the emotional information is used as input to modify the content and expression of the answer text. For example, if the user is feeling stressed, the server may adjust the answer by adding a gentler tone.

[1937] Step 9: The server sends the adjusted answer to the device.

[1938] The server sends the adjusted answer to the device, which includes the text modified based on the sentiment, and the device receives the adjusted answer.

[1939] Step 10: The device displays the adjusted answer to the user

[1940] The terminal displays the adjusted answer to the user. This display shows the corrected text sent from the server in the interface. Specifically, the answer is displayed in a friendly tone: "The pandas library is often used for data analysis in Python. Try it yourself."

[1941] Step 11: User requests project simulation

[1942] The user clicks the project simulation request button on the terminal. As an input, a request for project simulation is sent to the terminal.

[1943] Step 12: The device sends a request to the server

[1944] The terminal sends the user's project simulation request to the server as a GET request, which includes a request for the project simulation. The server then receives the request.

[1945] Step 13: The server randomly selects a project and extracts its details

[1946] Based on the request received by the server, a virtual project is randomly selected from the database. The server extracts the project details and generates them as output. Specifically, the project "Development of a Virtual Store" is selected.

[1947] Step 14: The server sends the details to the device

[1948] The server transmits the extracted project details to the terminal, including the project description and procedures, and the terminal receives the details.

[1949] Step 15: The terminal displays detailed information to the user and provides a simulation environment

[1950] The terminal displays the received project details to the user, providing a simulation environment in virtual reality space. This display shows the project information sent from the server on the interface. Specifically, details of the "virtual store development" are displayed, and the user experiences the project in virtual reality space.

[1951] Step 16: The device recognizes the user's emotions during the simulation.

[1952] During the simulation, the device's emotion engine recognizes the user's facial expressions and tone of voice in real time. The emotion engine receives the user's facial expression data and voice as input and extracts emotional information. Specifically, it extracts emotional information such as "the user is excited."

[1953] Step 17: The device sends the emotion information to the server.

[1954] The terminal transmits the extracted emotion information to the server, which includes the user's emotional state. The server receives the emotion information.

[1955] Step 18: The server adjusts the content and difficulty of the simulation based on the emotional information.

[1956] The content and difficulty of the simulation are dynamically adjusted based on the emotional information received by the server. In this process, the content of the simulation is dynamically changed using the emotional information as input. For example, if the user feels the simulation is difficult, adjustments such as simplifying the next step are made.

[1957] Step 19: The server sends the adjustments to the device

[1958] The server transmits the adjusted simulation content to the terminal, which includes the simulation data modified based on the emotion. The adjusted content is then received by the terminal.

[1959] Step 20: The device reflects the adjustments in the simulation environment and displays them to the user.

[1960] The terminal reflects the adjusted simulation content in the virtual reality space simulation environment and displays it to the user. This display reflects the corrected simulation data sent from the server on the interface. Specifically, the "Virtual Store Development" simulation with the adjusted difficulty level is displayed.

[1961] Through these processing steps, users can enjoy an interactive learning environment that combines hands-on project experiences. Furthermore, by utilizing emotional information, an environment that optimizes learning effectiveness is provided.

[1962] (Application example 2)

[1963] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1964] Conventional educational platforms provide uniform learning content without considering the user's emotions, making it difficult to maximize individual learning efficiency and comprehension. Furthermore, because they do not take into account the user's stress or fatigue, it can be difficult to continue learning. Furthermore, when providing a simulation environment using virtual reality technology, adjustments are not made based on the user's current emotional state, making it difficult to provide an optimal learning experience.

[1965] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a question input by a user from a terminal, means for transmitting the question to a natural language processing model to obtain an answer, and means for transmitting the answer to the terminal and displaying it to the user. This realizes a system including emotion recognition means for recognizing a user's emotions in real time and transmitting emotional information to the server, and means for dynamically adjusting answers and learning content based on the emotional information. This makes it possible to provide optimal and personalized learning content and a simulation environment according to the user's current emotional state.

[1966] A "terminal" is a device equipped with an input device and a user interface for a user to input questions and project simulation requests and obtain emotion information.

[1967] A "natural language processing model" is an algorithm or system that generates answers to questions entered by users, and specifically refers to a model that understands natural language and generates responses.

[1968] A "server" is a central computer that transmits, receives, and processes data via a network, and in the present invention includes a natural language processing model, a virtual reality space simulation means, and a device that processes emotional information.

[1969] "Virtual reality space simulation means" refers to technology or devices that provide an environment in which users can simulate projects using virtual reality technology.

[1970] "Emotion recognition means" refers to technology or devices that extract emotional information in real time from a user's facial expressions, tone of voice, etc., and is responsible for transmitting this information to a server.

[1971] An "interface" refers to a connection means or operating device that allows a user to operate a simulation means, and is designed to allow intuitive and efficient operation.

[1972] "Learning History and Performance Data" refers to data that records a user's past learning activities and the results of those activities, and is used to provide individually optimized learning content.

[1973] "Personalized learning content" refers to learning materials and assignments that are dynamically tailored based on a user's learning history and emotional state.

[1974] "Project simulation" refers to a virtual project that simulates real-world environments and scenarios in a virtual reality space, allowing users to experience and learn from them.

[1975] A specific embodiment for carrying out the present invention includes the following system and its operation, which enables a user to receive personalized learning content and virtual reality project simulations while having their emotions recognized in real time.

[1976] System configuration

[1977] 1. Terminal

[1978] The device is equipped with an input device and a user interface for users to input questions and project simulation requests. The device is also equipped with emotion recognition means to extract emotional information from the user's facial expressions and tone of voice in real time. This emotional information is sent to a server and used to dynamically adjust the learning content.

[1979] 2. Server

[1980] It includes a natural language processing model (e.g., ChatGPT) and generates answers to user questions. The server receives emotional information sent from the emotion recognition means and dynamically adjusts the answers and learning content. It is also connected to the virtual reality space simulation means and manages the simulation of the virtual project.

[1981] 3. Emotion recognition means

[1982] Emotional information is extracted in real time from the user's facial expressions, tone of voice, etc. This is installed on the device, and the data is sent to the server. Based on the emotional information, the server adjusts the answers and learning content.

[1983] 4. Virtual Reality Space Simulation Method

[1984] Using virtual reality technology, the system provides an environment in which users can simulate projects. Multiple virtual projects are stored in a database, and these are provided randomly or according to the user's preferences.

[1985] Program processing explanation

[1986] The server receives the user's emotional information in real time and dynamically optimizes the learning content based on it. To achieve this, the device is equipped with emotion recognition hardware such as a camera and microphone. The software uses natural language processing models (such as ChatGPT) and emotion recognition algorithms (such as EmotionEngine). The server processes the user's questions, generates optimal answers based on the emotional information, and recommends learning content.

[1987] Specific examples

[1988] For example, if a user enters a question such as "I want to know about data analysis using Python," the system operates as follows:

[1989] 1. The user enters "I want to know about data analysis using Python" into the input field of the terminal and submits it.

[1990] 2. The device sends this question to the server.

[1991] 3. The server receives the question and generates an answer using a natural language processing model (ChatGPT).

[1992] 4. The server sends the generated answer to the terminal and displays it to the user.

[1993] 5. At the same time, the device's emotion recognition means analyzes the user's emotion information and sends it to the server.

[1994] 6. The server dynamically adjusts answers and learning content based on emotional information.

[1995] When a user requests a project simulation, the server provides the user with a randomly selected simulation project and adjusts the difficulty level appropriately based on emotional information. For example, if the project "Development of a Virtual Store" is selected and the user feels stressed during the project simulation, the system will lower the difficulty level or add an explanation.

[1996] Prompt Sentence Examples

[1997] Below are some example prompts for generative AI models:

[1998] "What are some specific ways to make effective learning content recommendations when users are fatigued?"

[1999] "Please provide detailed steps on how to personalize educational content based on emotion recognition."

[2000] As described above, the embodiments of the present invention utilize the user's emotional information to provide individually optimized learning content and virtual reality project simulation, thereby maximizing the learning effect and improving the user's learning experience.

[2001] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2002] Step 1:

[2003] The user enters a question into the input field of the terminal and clicks the send button. This input field accepts the user's question in text format. The entered question is sent from the terminal to the server. The input is text data, and the output is an HTTP POST request to the server.

[2004] Step 2:

[2005] The server receives a question sent by a user. The server sends this question to a natural language processing model (ChatGPT) to generate an answer. The input is the text data of the question sent by the user, and the output is an API request to the natural language processing model and its response text.

[2006] Step 3:

[2007] The server receives a response from the natural language processing model and sends it to the terminal. The input is the data returned from the natural language processing model, and the output is text data as an HTTP response to the terminal.

[2008] Step 4:

[2009] The terminal receives the response and displays it to the user. Specifically, it displays the response text on the user interface. In this step, the input is the response data from the server, and the output is the text displayed on the terminal screen.

[2010] Step 5:

[2011] The device's emotion recognition means analyzes the user's facial expressions and tone of voice in real time. Emotional data is acquired using a camera or microphone and analyzed by an emotion recognition algorithm. The input is raw data from the camera or microphone, and the output is analyzed emotional information.

[2012] Step 6:

[2013] The device sends the acquired emotion information to the server. The input is emotion data from the emotion recognition means, and the output is emotion information as an HTTP POST request to the server.

[2014] Step 7:

[2015] The server receives emotional information and dynamically adjusts the tone and content of the answer based on that information. For example, if the user is feeling stressed, the server can make the answer gentler or add a brief explanation. The input is emotional information, and the output is adjusted text data.

[2016] Step 8:

[2017] The server sends the adjusted answers or learning content to the terminal and displays them again to the user. The input is the adjusted text data, and the output is data as an HTTP response to the terminal.

[2018] Step 9:

[2019] A user requests a project simulation. This request is sent from the terminal to the server. The input is the simulation request from the user, and the output is an HTTP GET request to the server.

[2020] Step 10:

[2021] The server receives a request for a project simulation and selects an appropriate project from the database. The input is the simulation request and the output is the selected project information.

[2022] Step 11:

[2023] The server sends the selected project information to the terminal. The input is the project information, and the output is the simulation data as an HTTP response to the terminal.

[2024] Step 12:

[2025] The terminal receives the project information and provides it to the user in the virtual reality space. The user executes the project in the virtual reality environment. The input is the project information, and the output is a simulation display in the virtual reality environment.

[2026] Step 13:

[2027] During the project simulation, the emotion recognition means recognizes the user's emotions in real time and sends them to the server. The input is raw data from the camera and microphone, and the output is analyzed emotional information.

[2028] Step 14:

[2029] The server dynamically adjusts the project difficulty and the learning content provided based on the emotional information, and provides it to the user again. The input is the emotional information and project progress information, and the output is the adjusted simulation content.

[2030] As a result, a system is realized that provides optimal learning experiences and project simulations according to the user's emotional state.

[2031] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2032] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2033] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2034] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2035] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2036] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2037] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2038] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2039] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2040] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reass...

Claims

1. means for receiving a question input by a user from a terminal; means for submitting the question to a natural language processing model to obtain an answer; means for transmitting the answer to a terminal and displaying it to a user; a server including a natural language processing model that generates answers to the questions; a virtual reality space simulation means connected to the server; a terminal including an interface for operating the simulation means; A system including:

2. 10. The system of claim 1, further comprising means for storing a plurality of virtual reality space simulation projects in a database and providing randomly selected project information to a user.

3. 10. The system according to claim 1, further comprising means for analyzing a user's learning history and performance data and providing individually optimized learning content.

Images