System

Abstract

To enable a user to learn while having a realistic experience as if traveling to an actual place or era.SOLUTION: A system includes: a server for generating a virtual space using a virtual reality technology; means for acquiring information from a destination database and an era database and constructing the virtual space; a terminal for a user to operate within the virtual space; generative AI for generating explanation to be provided to the user within the virtual space on a subject that is a learning target of the user; means for accepting the operation by the user via a user interface displayed on the terminal; and means for providing content including the explanation to the user in real time within the virtual space based on the accepted operation by the user.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional educational methods had the problem that it was difficult for children to have realistic experiences without immersing themselves in an actual place or time. [Means for solving the problem]

[0005] The present invention provides an educational system that utilizes virtual reality technology, thereby providing a means for children to learn while having a realistic experience of traveling to a real place or time by entering a virtual space. Specifically, the present invention includes the following means.

[0006] A virtual space is generated using a server, and the shape of spaces such as classrooms and spaceships, the placement of objects, background sounds, etc. are reproduced based on information obtained from the destination database and the era database.

[0007] It displays a virtual space through the terminal and provides means for 3D graphics and audio playback.

[0008] While the user operates the virtual space, the generative AI provides commentary on the subject.

[0009] Through these measures, children can have realistic experiences that make them feel as if they are in a real place or time without leaving the comfort of their own home, thereby enhancing the effectiveness of their education.

[0010] "Virtual space" refers to an imaginary space that recreates an actual place or time, generated using virtual reality technology.

[0011] A "destination database" refers to a database that registers information about virtual space destinations, such as classrooms, space, the deep sea, primeval forests, and inside the clouds.

[0012] A "period database" refers to a database that registers information about past eras (such as the Stone Age, Heian period, Sengoku period, Edo period, etc.) and future eras (such as 2100 and 2200).

[0013] A "server" refers to a computer system responsible for generating virtual space and processing data.

[0014] "Terminal" refers to a device (smartphone, VR headset, etc.) that allows a user to display and operate a virtual space.

[0015] "Generative AI (e.g., generative AI)" refers to AI (artificial intelligence) that uses natural language processing technology to engage in dialogue. In a virtual space, it provides explanations about the subjects that users are studying and engages in dialogue with children. [Brief description of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12] 11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. The processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0020] Examples include GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0021] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The embodiment for carrying out the present invention comprises the following elements.

[0039] 1. Server:

[0040] Responsible for generating virtual space.

[0041] Information is obtained from destination and era databases to construct a virtual space.

[0042] Executes processing related to the generation of virtual space.

[0043] 2. Terminal:

[0044] Displays a virtual space generated by the server.

[0045] It provides an interface that allows the user to perform operations within a virtual space.

[0046] It executes processing related to the display of the virtual space and user operations.

[0047] 3. User:

[0048] Users access and operate the virtual space through a terminal.

[0049] Students progress through learning by interacting with generative AI in a virtual space.

[0050] 4. Generative AI:

[0051] Explanations on subjects are given in a virtual space.

[0052] Provide appropriate information according to user operations and circumstances.

[0053] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with their learning in the virtual space. The generative AI provides explanations on subjects in the virtual space to support the user's learning. For example, if the generative AI playing the role of teacher specifies the destination "space" and the time period "2100," the server acquires space data and reproduces the future universe in 2100. In the virtual space, the children experience the sensation of being in a spaceship while the generative AI provides an explanation on the universe. For example, if the generative AI explains the characteristics of a planet in space, the children can listen to the generative AI's explanation while exploring the planet.

[0054] The process flow will be explained below.

[0055] Step 1: The server starts generating the virtual space.

[0056] The server receives destination and time indications.

[0057] The server obtains information on the designated destination from a destination database.

[0058] The server obtains information on the designated era from the era database.

[0059] Step 2: The server generates the virtual space.

[0060] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0061] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0062] The server creates the data for the generated virtual space.

[0063] Step 3: Your device displays the virtual space

[0064] The terminal receives the virtual space data generated from the server.

[0065] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0066] Step 4: The user operates in the virtual space

[0067] Users access the virtual space through their terminals.

[0068] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0069] Step 5: Generative AI provides commentary on the subject

[0070] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0071] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0072] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user can interact with the generative AI while performing operations within the virtual space. Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0074] In virtual space learning systems using virtual reality technology, the challenge is to ensure that users can access, operate, and experience appropriate educational content in real time. In particular, high real-time operability and adaptability are required for users to effectively progress with their learning in the virtual space. In addition, it is necessary for dynamic explanations by generative AI to immediately respond to the user's learning requests.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes a means for generating a virtual space, a means for acquiring information from a destination database and an era database and constructing the virtual space, a generative AI for explaining subjects in the virtual space, a means for accepting user operations via an interface displayed on the terminal, and a means for the generative AI to provide content in real time in the virtual space based on the user's input. This allows the user to access in real time and receive dynamic educational content from the generative AI while operating in the virtual space.

[0077] A "server" is a central processing unit that generates a virtual space, retrieves necessary information from a database, and transmits it to a user terminal.

[0078] The "destination database" is a database that accumulates information about locations and environments necessary for constructing a virtual space.

[0079] A "period database" is a database that stores information about history and future events relating to a particular period.

[0080] A "virtual space" is a virtual environment that is generated using virtual reality technology and in which a user can immerse themselves and interact.

[0081] A "means" is a device or method used to accomplish a particular function or purpose.

[0082] "Terminal" refers to the input devices and display devices that allow a user to access and interact with a virtual space, including, for example, a VR headset or a computer.

[0083] "Generative AI" is an artificial intelligence system that provides educational content within a virtual space and explains information in real time through interaction with the user.

[0084] An "interface" refers to a user input device and a display device that allow a user to access and operate a virtual space through a terminal.

[0085] "Providing content in real time" refers to providing educational information and explanations instantly in response to user operations or requests.

[0086] "Shape of space" refers to the shape of the 3D models of terrain and buildings within the virtual space.

[0087] "Object placement" refers to the position and layout of objects that exist within a virtual space.

[0088] "Background sound" refers to the sounds and music that are played according to the environment and scene within the virtual space.

[0089] This invention provides a system for improving the educational experience of users in a virtual space. The system is mainly composed of a server, a terminal, a user, and a generative AI. A specific embodiment will be described below.

[0090] Server Roles

[0091] The server is the central player in charge of generating the virtual space. It performs the following specific tasks:

[0092] Data Acquisition: The server acquires information from the destination database and the era database. The destination database stores information about places and environments, and the era database stores information about specific eras.

[0093] Space generation: Generate a virtual space based on the acquired data. This generation includes the shape of the space, the placement of objects, and the generation of background sounds.

[0094] Data transmission: The generated virtual space data is sent to the terminal.

[0095] Terminal Roles

[0096] The terminal is a device that allows users to access and operate the virtual space. The terminal performs the following specific operations:

[0097] Displaying the virtual space: Receive the virtual space data sent from the server and display it on a display (e.g. a VR headset).

[0098] Interface provision: Provide an interface (e.g., controllers and gesture recognition) that allows the user to operate within the virtual space.

[0099] Audio playback: Plays ambient sounds in the virtual space and generative AI commentary.

[0100] User Roles

[0101] Users access the virtual space through their devices and carry out learning activities. The users perform the following specific operations:

[0102] Startup and Access: Turn on the device and access the interface. The generative AI asks the user what topic they want to learn, and the user inputs it accordingly.

[0103] In-spatial interaction: Move freely within the virtual space and learn while listening to commentary provided by the generative AI.

[0104] The role of generative AI

[0105] Generative AI supports the user's learning experience in a virtual space. Specifically, generative AI performs the following tasks:

[0106] Providing content: Based on user input, the virtual space provides explanations of subjects, providing real-time information and answering user questions.

[0107] Dynamic changes: The commentary changes dynamically in response to user actions, and the content in the virtual space also adapts.

[0108] Examples

[0109] Example scenario:

[0110] 1. The user starts up the device and accesses the virtual space.

[0111] 2. The generative AI asks the user, “What would you like to learn about today?”

[0112] 3. The user responds, "I want to learn about planets in space."

[0113] 4. The generative AI responds, "Now, I will show you information about Mars in the future, from the year 2100."

[0114] 5. The server retrieves space information from the destination database and information about the year 2100 from the time database, and generates a virtual space of the future Mars.

[0115] 6. The device displays the generated virtual Mars space.

[0116] 7. The user explores Mars in a virtual space, and the generative AI explains the planet's characteristics and environment.

[0117] Example prompt:

[0118] "Please explain the characteristics of Mars in a virtual space of the future of Mars in the year 2100."

[0119] Based on the specific embodiment of this invention, users can effectively study in a virtual space, and the dynamic explanations provided in real time by generative AI further improve the user's learning experience.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user starts a terminal and logs in.

[0123] Enter: The user powers on the device and accesses the login screen.

[0124] Action: The user enters a username and password.

[0125] Output: The device sends the user's authentication information to the server and displays the main menu if authentication is successful.

[0126] Step 2:

[0127] A generative AI asks users what topics they would like to learn about.

[0128] Input: The user accesses the main menu.

[0129] How it works: The generative AI asks via voice or text, “What would you like to learn about today?”

[0130] Output: The user inputs a topic they want to learn about (e.g., "Information about planets in the universe in the year 2100").

[0131] Step 3:

[0132] The generative AI requests data from the server.

[0133] Input: The user inputs the topic they want to learn about into the generative AI.

[0134] How it works: A generative AI analyzes user input and identifies the required destination and era data.

[0135] Output: The generative AI requests the server to obtain destination data and era data.

[0136] Step 4:

[0137] The server retrieves the required data from the database.

[0138] Input: Data request from generative AI.

[0139] How it works: The server runs SQL queries to get information about the universe from the destination database and information about the year 2100 from the era database.

[0140] Output: The acquired data (e.g., 3D model data of the universe and scenario data for the year 2100) is stored on the server.

[0141] Step 5:

[0142] The server generates the virtual space.

[0143] Input: Acquired data (space information, future information from the year 2100).

[0144] Operation: Based on the data obtained by the server, the shape of the virtual space is formed, objects are placed, and background sounds are generated.

[0145] Output: The generated virtual space data.

[0146] Step 6:

[0147] The server transmits virtual space data to the terminal.

[0148] Input: Data of the generated virtual space.

[0149] Operation: The server divides this data into packets and sends them to the device.

[0150] Output: Virtual space data received by the device.

[0151] Step 7:

[0152] The device displays the virtual space.

[0153] Input: Received virtual space data.

[0154] How it works: The device decodes the data and displays it in a virtual world on a display (e.g. a VR headset).

[0155] Output: The user can visually recognize the virtual space.

[0156] Step 8:

[0157] The generative AI begins to explain within the virtual space.

[0158] Input: The virtual space is displayed.

[0159] How it works: A generative AI uses voice synthesis technology to begin explaining something about space.

[0160] Output: The user can hear the audio description.

[0161] Step 9:

[0162] The user performs operations within the virtual space.

[0163] Input: User interaction with the interface (e.g., use of a controller).

[0164] Actions: The user moves through the virtual space and observes objects.

[0165] Output: Changes in the screen display based on the user's viewpoint and actions.

[0166] Step 10:

[0167] Generative AI dynamically provides content in response to user questions.

[0168] Input: The user asks the generative AI a question (e.g., "What is the atmosphere of Mars like?").

[0169] How it works: A generative AI analyzes the question, searches for relevant information, and provides an explanation.

[0170] Output: Specific information provided by the generative AI (e.g., "The atmosphere of Mars in 2100 will be thin and composed primarily of carbon dioxide."). (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] In today's educational environment, it is difficult to provide users with the opportunity to learn about various exhibits without visiting the real environment. In addition, in order to learn deeply about a single exhibit, it is necessary to efficiently provide a huge amount of information and provide an interactive experience at the same time. However, it is difficult for current systems to meet these requirements.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a means for acquiring information from an exhibition information database and reflecting the exhibit information in the virtual space, and a generating AI for explaining exhibits in the virtual space. This allows users to learn about various exhibits from the comfort of their own homes and gain a deeper understanding of them.

[0175] "Virtual reality technology" refers to technology that uses computer technology to generate virtual environments or scenes that feel real and allow users to immerse themselves in them.

[0176] A "virtual space" is a three-dimensional digital space that is generated using virtual reality technology and that users can experience virtually.

[0177] A "server" is a computer system that provides and processes data via a network, and in this invention is responsible for generating and managing the virtual space.

[0178] A "destination database" is a database that accumulates information about places that a user is expected to visit.

[0179] A "period database" is a database that accumulates information about a specific era.

[0180] An "exhibition information database" is a database that accumulates information about exhibits at museums, exhibitions, etc.

[0181] A "terminal" is a device that allows a user to access and operate a virtual space, and includes smartphones, smart glasses, head-mounted displays, etc.

[0182] A "generative AI" is an agent generated using artificial intelligence technology to provide appropriate information according to the user's operations and circumstances, and in this invention, it is responsible for providing explanations within the virtual space.

[0183] An "exhibit" is an object that a user observes and learns about in a virtual museum or exhibition.

[0184] "3D graphics" is a technology for generating and displaying visual objects in three-dimensional space.

[0185] The embodiment of the present invention is composed of the following elements: a server, a terminal, a user, an exhibition information database, a destination database, an era database, and a system utilizing generation AI.

[0186] First, the server uses hardware to generate the virtual space, specifically a high-performance computer and network environment. The server acquires information from the destination database and the era database, and uses virtual reality technology to create the virtual space based on this information. It also acquires information from the exhibition information database, and reflects the information on the exhibits in the virtual space.

[0187] Next, the terminal is a device that allows the user to access and operate this virtual space. Examples include smartphones, smart glasses, and head-mounted displays (HMDs). The terminal is equipped with 3D graphics and audio playback functions, which provide the user with a realistic experience. In addition, the terminal allows the user to change the viewpoint and operate objects in the virtual space according to user operations.

[0188] Users access the virtual space through their devices and interact with the AI ​​to experience the exhibits and explanations. The AI ​​provides a wealth of information about the exhibits and gives appropriate explanations based on the user's actions and questions. This allows users to learn about the various exhibits and gain a deeper understanding from the comfort of their own home.

[0189] As a concrete example, if a user inputs a prompt such as "Tell me about the Pharaohs of Egypt" into the AI ​​generator, the AI ​​generator will respond as follows:

[0190] "Ancient Egyptian pharaohs were gods and leaders of Egyptian civilization. They oversaw the construction of pyramids and temples and accomplished many historic feats. For example, famous pharaohs are known for their golden masks."

[0191] In this way, the server generates the virtual space, the terminal displays it, and the user operates it, allowing the user to learn and experience within the virtual space. The generating AI provides commentary on the exhibits within the virtual space, supporting the user's learning.

[0192] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] The server receives a request from a user.

[0195] Specifically, a user launches the virtual museum app via a terminal and sends a request for a particular exhibit (e.g., "I want to see the Egyptian exhibit"). The server's input is the request, and it generates as output instructions for proceeding to the next step of processing.

[0196] Step 2:

[0197] The server retrieves information based on the request from a destination database and a period database.

[0198] Specifically, the server executes a database query to retrieve data about the specified destination and time period (e.g., "Egypt", "Ancient"). The input to this step is the user request, and the output is the retrieved data.

[0199] Step 3:

[0200] The server obtains information about the exhibit corresponding to the request from an exhibition information database.

[0201] The server obtains data related to the exhibit (e.g., information about "Egyptian Pharaohs") based on the destination and era data obtained in the previous step. The input is the output data of the previous step, and the output is detailed exhibit information.

[0202] Step 4:

[0203] The server generates a virtual space based on the acquired data.

[0204] Specifically, the server uses virtual reality technology to integrate information on the destination, era, and exhibits, and constructs a 3D virtual space. The input for this step is various data obtained from the database, and the output is the data for the generated virtual space.

[0205] Step 5:

[0206] The terminal receives the virtual space data transmitted from the server and displays it.

[0207] The terminal uses 3D graphics and audio playback functions to provide the user with a realistic virtual space. The input of this step is the virtual space data sent from the server, and the output is the virtual space displayed on the user interface.

[0208] Step 6:

[0209] The user operates within the virtual space through the terminal.

[0210] Specifically, the user performs operations such as clicking on virtual exhibits and changing the viewpoint. The input of this step is the displayed virtual space, and the output is the user's operation data.

[0211] Step 7:

[0212] The generative AI provides commentary about the exhibits in response to user input.

[0213] The server analyzes the user's operation data and generates and sends a prompt sentence (e.g., "Tell me about the Egyptian pharaohs") to the generation AI. The generation AI generates a response based on this prompt and provides an explanation. The input is the user's operation data and the generated prompt sentence, and the output is the explanatory text or audio data.

[0214] Step 8:

[0215] The terminal receives commentary data from the generating AI and provides it to the user.

[0216] Specifically, the device displays text or plays audio to provide an explanation to the user. The input for this step is the explanation data sent from the generation AI, and the output is the display on the user interface or audio playback.

[0217] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0218] The embodiment for implementing the present invention comprises the following elements:

[0219] 1. Server:

[0220] Responsible for generating virtual space.

[0221] Information is obtained from destination and era databases to construct a virtual space.

[0222] By combining it with an emotion engine, it recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI.

[0223] 2. Terminal:

[0224] Displays a virtual space generated by the server.

[0225] It provides an interface that allows the user to perform operations within a virtual space.

[0226] It executes processing related to the display of the virtual space and user operations.

[0227] 3. User:

[0228] Users access and operate the virtual space through a terminal.

[0229] The emotion engine recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI accordingly.

[0230] 4. Generative AI:

[0231] Explanations on subjects are given in a virtual space.

[0232] The emotion engine recognizes the user's emotional state and provides appropriate information and interactions accordingly.

[0233] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted to provide a more personalized learning experience.

[0234] The process flow will be explained below.

[0235] Step 1: The server starts generating the virtual space.

[0236] The server receives destination and time indications.

[0237] The server obtains information on the designated destination from a destination database.

[0238] The server obtains information on the designated era from the era database.

[0239] The server starts the emotion engine and prepares it to recognize the user's emotional state.

[0240] Step 2: The server generates the virtual space.

[0241] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0242] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0243] The server creates the data for the generated virtual space.

[0244] The server monitors the user's emotional state using an emotion engine and adjusts the behavior of the virtual space and generative AI.

[0245] Step 3: Your device displays the virtual space

[0246] The terminal receives the virtual space data generated from the server.

[0247] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0248] Step 4: The user operates in the virtual space

[0249] Users access the virtual space through their terminals.

[0250] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0251] The emotion engine recognizes the user's emotional state and infers emotions by analyzing the user's facial and vocal characteristics.

[0252] Step 5: Generative AI provides commentary on the subject

[0253] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0254] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0255] The emotion engine recognises the user's emotional state and adjusts the generative AI's dialogue style and expressions accordingly.

[0256] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user operates it, allowing the user to proceed with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted, providing a more personalized learning experience. Example 2

[0257] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0258] In conventional educational systems using virtual reality technology, it was difficult to provide a personalized learning experience according to the user's emotions. In addition, the dialogue and information provision using generative AI models were fixed and could not be adjusted in real time, which led to a problem that the learning effect was not fully realized.

[0259] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0260] In this invention, the server includes a means for acquiring information from a destination database and an era database and constructing a virtual space, a means for combining an emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generating AI, a terminal that the user operates in the virtual space, a generating AI that explains subjects in the virtual space, and a means for operating the generating AI model using prompt sentences that provide dialogue and information based on the user's emotional state. This enables a personalized learning experience according to the user's emotions and state.

[0261] A "server" is a computer device that generates a virtual space using virtual reality technology and builds the virtual space based on information obtained from a database.

[0262] A "destination database" is a database that accumulates information about places and destinations to be referenced within a virtual space.

[0263] A "period database" is a database that accumulates information about specific eras or historical periods that can be referenced within a virtual space.

[0264] A "virtual space" is a three-dimensional space digitally constructed using virtual reality technology.

[0265] The "emotion engine" is an engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI based on that information.

[0266] A "terminal" is a device that allows a user to access and operate a virtual space, such as a smartphone or a personal computer.

[0267] "Generative AI" is an artificial intelligence that provides explanations and dialogue on subjects within a virtual space.

[0268] A "generative AI model" is an artificial intelligence model that generates dialogue and information using prompt sentences based on the user's emotional state.

[0269] A "prompt" is a text input that can be fed into a generative AI model to generate specific dialogue or information.

[0270] The present invention provides a system consisting of the following elements. The server generates a virtual space and adjusts the behavior of the virtual space and the generative AI based on the user's emotional state. The user accesses and operates the virtual space through a terminal. A personalized learning experience is provided by combining the emotion engine and the generative AI model.

[0271] server

[0272] The server runs using cloud services such as Amazon Web Services (AWS®) and Google Cloud Platform (GCP®). The server retrieves information from the destination database and the era database, and generates the virtual space based on that information. The server uses emotion engines such as Affectiva®'s SDK and IBM Watson® to recognize the user's emotional state. This makes it possible to adjust the behavior of the virtual space and the generation AI.

[0273] Terminal

[0274] The terminals include devices such as iPhone (registered trademark), ANDROID (registered trademark) smartphones, Windows (registered trademark) PCs, and Mac (registered trademark). The terminals display the virtual space generated by the server and provide an interface for the user to operate within the virtual space. The terminals use, for example, Apple (registered trademark)'s ARKit (registered trademark) or Google (registered trademark)'s ARCore (registered trademark) to display the virtual space superimposed on the real space. The user operates objects in this environment using a touch screen or mouse.

[0275] User

[0276] Users access and interact with the virtual space through their devices. The emotion engine recognizes the user's emotional state in real time. For example, if a user is losing focus while learning about ancient Egypt, the system will use generative AI models to adjust their interactions and activities.

[0277] Generation AI

[0278] Generative AI (e.g., OpenAI® ChatGPT® or Google® BERT®) provides explanations on academic subjects in a virtual space. The user's emotional state is recognized, and appropriate explanations and dialogue are provided based on that information. Generative AI uses prompts to generate dialogue for the user.

[0279] Usage example

[0280] For example, if a user is learning about ancient Egypt, the server will retrieve information about "ancient Egypt" from the destination database and information about "3000 BC" from the period database. Based on this, the server will generate a virtual space and send it to the terminal.

[0281] The device displays this virtual space and the user begins to operate it. If the user's emotional state is determined to be "low concentration" by the emotion engine, the generative AI provides dialogue using prompt sentences such as the following:

[0282] “What interactions or activities should we suggest if a user is having trouble concentrating while learning about Ancient Egypt?”

[0283] Based on this prompt, the generative AI can provide users with interesting quizzes and stories, which can recapture the user's attention and improve their learning outcomes.

[0284] As described above, the present invention provides a personalized learning experience by having the various elements of the server, terminal, user, emotion engine, and generative AI model work in coordination.

[0285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0286] Step 1:

[0287] The server retrieves information from a destination database and a period database.

[0288] Specifically, the server executes the following queries: "SELECT FROM destination database WHERE destination='Ancient Egypt'" and "SELECT FROM period database WHERE era='3000 BC'".

[0289] Input: Destination database, era database

[0290] Output: Data set for generating virtual space (destination information, time information)

[0291] Step 2:

[0292] A virtual space is generated based on the data acquired by the server.

[0293] Specifically, the server uses the generative AI model to generate a 3D model, which is designed taking into account the user's past behavioral and emotional data.

[0294] Input: Data set (destination information, time information), user's past behavior data, emotion data

[0295] Output: 3D model of the virtual space

[0296] Step 3:

[0297] The server transmits the virtual space to the terminal.

[0298] As a specific operation, the server transmits the generated 3D model data to the terminal using the HTTP protocol.

[0299] Input: 3D model of the virtual space

[0300] Output: 3D model data sent to the device

[0301] Step 4:

[0302] The device displays the virtual space.

[0303] Specifically, the device uses Apple's (registered trademark) ARKit (registered trademark) or Google's (registered trademark) ARCore (registered trademark) to overlay the received 3D model data onto real space.

[0304] Input: 3D model data

[0305] Output: Virtual space displayed on the device

[0306] Step 5:

[0307] The user performs operations within the virtual space.

[0308] Specifically, the user operates objects and views information panels using a touch screen or a mouse.

[0309] Input: Terminal user interface

[0310] Output: User operation data

[0311] Step 6:

[0312] The server uses an emotion engine to recognize the user's emotional state.

[0313] Specifically, the server analyzes the user's facial expressions and voice through the device's camera and microphone to measure their level of concentration and excitement.

[0314] Input: User's facial expression data, voice data

[0315] Output: The user's emotional state.

[0316] Step 7:

[0317] The server uses the generative AI model to generate interactions and activities based on the user's emotional state.

[0318] Specifically, the server inputs the prompt sentence into the generative AI model to get an appropriate response. For example, prompt sentence: "If the user is losing concentration while learning about ancient Egypt, what dialogue or activity should be suggested?"

[0319] Input: prompt text, user's emotional state

[0320] Output: The generated interactions and activities

[0321] Step 8:

[0322] The terminal provides the generated interactions and activities to the user.

[0323] As a specific operation, the terminal displays the generated dialogue content on the screen and outputs it as voice.

[0324] Input: Generated interactions and activities

[0325] Output: The interactions and activities presented to the user

[0326] By following these processing steps, the system of the present invention is able to provide the user with a personalized learning experience. (Application example 2)

[0327] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] In conventional systems using virtual space, it was difficult to dynamically provide content according to the user's emotional state, and only uniform information could be provided. As a result, it was difficult to provide learning or purchasing experiences according to the user's interests, and it was not possible to provide personalized services. In addition, there were limited methods for effectively providing detailed information about products and promoting them in virtual space. This made it a challenge to improve user satisfaction.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0330] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a terminal for a user to operate in the virtual space, a generation AI for explaining a subject in the virtual space, an emotion engine for recognizing the emotional state of the user and adjusting the behavior of the virtual space and the generation AI according to the emotional state, and a means for acquiring information from a product database and providing details about products in the virtual space. This makes it possible to provide a dynamic and personalized learning experience or purchasing experience according to the emotional state of the user.

[0331] "Virtual reality technology" is a technology that uses computer technology to create a virtual space that is different from the real world, allowing users to immerse themselves in the experience.

[0332] "Virtual space" refers to a digital three-dimensional space created using virtual reality technology.

[0333] A "server" is an information processing device that generates a virtual space using virtual reality technology and handles user operations and data processing.

[0334] A "destination database" is a database that stores information about destinations for constructing a virtual space.

[0335] An "era database" is a database that stores information about a specific era, and is used when constructing a virtual space.

[0336] A "terminal" is an information device that a user uses to access and operate a virtual space.

[0337] "Generative AI" is a program that uses artificial intelligence technology to provide explanations and dialogue on subjects in a virtual space.

[0338] The "emotion engine" is a component that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI according to that emotional state.

[0339] A "product database" is a database that stores detailed information about products displayed in a virtual space.

[0340] "Means for providing product details" is a function for obtaining information from a product database and providing detailed information about the product to the user within the virtual space.

[0341] In order to carry out the invention, a system including the following elements is configured. First, the server generates a virtual space using virtual reality technology and obtains information from a destination database and an era database to construct the virtual space. This virtual space is displayed on a user terminal and can be operated by the user.

[0342] The server is equipped with an emotion engine that can recognize the user's emotional state. The emotion engine uses facial expression and voice analysis technology to grasp the user's real-time emotions and dynamically adjusts the behavior of the virtual space and the generative AI accordingly. It also has a means of retrieving information from a product database and providing details about products in the virtual space.

[0343] Generative AI can provide personalized product explanations and promotions based on the user's emotional state, providing real-time information tailored to the user's interests and preferences, and improving satisfaction within virtual stores.

[0344] The terminal is an information device that allows users to access and operate the virtual space, and is equipped with 3D graphics and audio playback functions. This allows users to experience the virtual space using both their eyes and ears. It also has a dialogue function based on the user's emotional state using an emotion engine. Specifically, when a user selects a product, the system reads the user's emotions from their facial expressions and voice, and generates an optimal product description.

[0345] As a concrete example of this system, consider a user selecting clothes in a virtual store: if the user is interested in a particular dress, the emotion engine will recognize the user's surprised expression, and the generative AI will provide a personalized description such as "This dress is from our latest collection. It's perfect for a special event!", providing a more personalized experience.

[0346] Examples of prompts for generative AI models include the following:

[0347] "Users are surprised to see a dress from your new collection. Can you help me explain it to them in a way that will keep them interested?"

[0348] In terms of hardware, the server needs to be an information processing device equipped with a high-performance processor and sufficient memory, and as user terminals, smartphones, smart glasses, or head-mounted displays that support virtual reality technology are suitable. For software, it is desirable to use libraries and SDKs (software development kits) for generating virtual spaces, and AI frameworks such as TENSORFLOW (registered trademark) and PyTorch (registered trademark) for implementing artificial intelligence. For emotion recognition, it is recommended to use OpenCV (registered trademark) or the Emotion Recognition library.

[0349] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0350] Step 1:

[0351] The server generates a virtual space using virtual reality technology. It uses information obtained from the destination database and the era database as input, and generates the shape of the virtual space, the placement of objects, and background sounds based on that information. The output is the data of the generated virtual space.

[0352] Step 2:

[0353] The server transmits the generated virtual space data to the user's terminal. The terminal receives it and displays the virtual space to the user using 3D graphics and audio. The input is the virtual space data transmitted from the server, and the output is to allow the user to experience the virtual space visually and aurally.

[0354] Step 3:

[0355] The user performs operations in the virtual space. Specifically, when the user shows interest in a particular object (e.g., a product), the user makes a selection. The input is the user's operation data, and the output is the object to be selected (e.g., the product ID).

[0356] Step 4:

[0357] The server retrieves detailed information about the selected object from the product database. The input is the ID of the object selected by the user, and the output is detailed information about that object.

[0358] Step 5:

[0359] The server uses an emotion engine to recognize the user's emotional state. To do so, the server uses the user's facial expression data and voice data as input. The emotion engine analyzes these data and outputs the user's emotional state (e.g., surprise, joy, etc.).

[0360] Step 6:

[0361] The server sends a prompt to the generation AI based on the emotional state. Specifically, it creates a prompt such as, "The user is surprised to see a dress from the new collection. Please advise how to explain it so that they will remain interested." The input is the user's emotional state and the selected product information, and the output is the prompt sent to the generation AI.

[0362] Step 7:

[0363] The generation AI generates personalized product descriptions and promotional content based on the prompt text. The input is the prompt text sent from the server, and the output is the product description or promotional content provided to the user.

[0364] Step 8:

[0365] The server transmits the product description and promotional content generated by the AI ​​to the user's device. The device receives it and displays it to the user in the virtual space. The input is the product description generated by the AI, and the output is to provide the user with the product description or promotional content visually and audibly.

[0366] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0367] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0368] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0369] [Second embodiment]

[0370] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0371] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0372] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0373] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0374] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0375] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0376] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0377] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0378] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0379] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0380] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0381] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0382] The embodiment for implementing the present invention comprises the following elements:

[0383] 1. Server:

[0384] Responsible for generating virtual space.

[0385] Information is obtained from destination and era databases to construct a virtual space.

[0386] Executes processing related to the generation of virtual space.

[0387] 2. Terminal:

[0388] Displays a virtual space generated by the server.

[0389] It provides an interface that allows the user to perform operations within a virtual space.

[0390] It executes processing related to the display of the virtual space and user operations.

[0391] 3. User:

[0392] Users access and operate the virtual space through a terminal.

[0393] Students progress through learning by interacting with generative AI in a virtual space.

[0394] 4. Generative AI:

[0395] Explanations on subjects are given in a virtual space.

[0396] Provide appropriate information according to user operations and circumstances.

[0397] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with their learning in the virtual space. The generative AI provides explanations on subjects in the virtual space to support the user's learning. For example, if the generative AI playing the role of teacher specifies the destination "space" and the time period "2100," the server acquires space data and reproduces the future universe in 2100. In the virtual space, the children experience the sensation of being in a spaceship while the generative AI provides an explanation on the universe. For example, if the generative AI explains the characteristics of a planet in space, the children can listen to the generative AI's explanation while exploring the planet.

[0398] The process flow will be explained below.

[0399] Step 1: The server starts generating the virtual space.

[0400] The server receives destination and time indications.

[0401] The server obtains information on the designated destination from a destination database.

[0402] The server obtains information on the designated era from the era database.

[0403] Step 2: The server generates the virtual space.

[0404] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0405] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0406] The server creates the data for the generated virtual space.

[0407] Step 3: Your device displays the virtual space

[0408] The terminal receives the virtual space data generated from the server.

[0409] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0410] Step 4: The user operates in the virtual space

[0411] Users access the virtual space through their terminals.

[0412] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0413] Step 5: Generative AI provides commentary on the subject

[0414] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0415] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0416] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user can interact with the generative AI while performing operations within the virtual space. Example 1

[0417] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0418] In virtual space learning systems using virtual reality technology, the challenge is to ensure that users can access, operate, and experience appropriate educational content in real time. In particular, high real-time operability and adaptability are required for users to effectively progress with their learning in the virtual space. In addition, it is necessary for dynamic explanations by generative AI to immediately respond to the user's learning requests.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0420] In this invention, the server includes a means for generating a virtual space, a means for acquiring information from a destination database and an era database and constructing the virtual space, a generative AI for explaining subjects in the virtual space, a means for accepting user operations via an interface displayed on the terminal, and a means for the generative AI to provide content in real time in the virtual space based on the user's input. This allows the user to access in real time and receive dynamic educational content from the generative AI while operating in the virtual space.

[0421] A "server" is a central processing unit that generates a virtual space, retrieves necessary information from a database, and transmits it to a user terminal.

[0422] The "destination database" is a database that accumulates information about locations and environments necessary to construct a virtual space.

[0423] A "period database" is a database that stores information about history and future events relating to a particular period.

[0424] A "virtual space" is a virtual environment that is generated using virtual reality technology and in which a user can immerse themselves and interact.

[0425] A "means" is a device or method used to accomplish a particular function or purpose.

[0426] "Terminal" refers to the input devices and display devices that allow a user to access and interact with a virtual space, including, for example, a VR headset or a computer.

[0427] "Generative AI" is an artificial intelligence system that provides educational content within a virtual space and explains information in real time through interaction with the user.

[0428] An "interface" refers to a user input device and a display device that allow a user to access and operate a virtual space through a terminal.

[0429] "Providing content in real time" refers to providing educational information and explanations instantly in response to user operations or requests.

[0430] "Shape of space" refers to the shape of the 3D models of terrain and buildings within the virtual space.

[0431] "Object placement" refers to the position and layout of objects that exist within a virtual space.

[0432] "Background sound" refers to the sounds and music that are played according to the environment and scene within the virtual space.

[0433] This invention provides a system for improving the educational experience of users in a virtual space. The system is mainly composed of a server, a terminal, a user, and a generative AI. A specific embodiment will be described below.

[0434] Server Roles

[0435] The server is the central player in charge of generating the virtual space. It performs the following specific tasks:

[0436] Data Acquisition: The server acquires information from the destination database and the era database. The destination database stores information about places and environments, and the era database stores information about specific eras.

[0437] Space generation: Generate a virtual space based on the acquired data. This generation includes the shape of the space, the placement of objects, and the generation of background sounds.

[0438] Data transmission: The generated virtual space data is sent to the terminal.

[0439] Terminal Roles

[0440] The terminal is a device that allows users to access and operate the virtual space. The terminal performs the following specific operations:

[0441] Displaying the virtual space: Receive the virtual space data sent from the server and display it on a display (e.g. a VR headset).

[0442] Interface provision: Provide an interface (e.g., controllers and gesture recognition) that allows the user to operate within the virtual space.

[0443] Audio playback: Plays ambient sounds in the virtual space and generative AI commentary.

[0444] User Roles

[0445] Users access the virtual space through their devices and carry out learning activities. The users perform the following specific operations:

[0446] Startup and Access: Turn on the device and access the interface. The generative AI asks the user what topic they want to learn, and the user inputs it accordingly.

[0447] In-spatial interaction: Move freely within the virtual space and learn while listening to commentary provided by the generative AI.

[0448] The role of generative AI

[0449] Generative AI supports the user's learning experience in a virtual space. Specifically, generative AI performs the following tasks:

[0450] Providing content: Based on user input, the virtual space provides explanations of subjects, providing real-time information and answering user questions.

[0451] Dynamic changes: The commentary changes dynamically in response to user actions, and the content in the virtual space also adapts.

[0452] Examples

[0453] Example scenario:

[0454] 1. The user starts up the device and accesses the virtual space.

[0455] 2. The generative AI asks the user, “What would you like to learn about today?”

[0456] 3. The user responds, "I want to learn about planets in space."

[0457] 4. The generative AI responds, "Now, I will show you information about Mars in the future, from the year 2100."

[0458] 5. The server retrieves space information from the destination database and information about the year 2100 from the time database, and generates a virtual space of the future Mars.

[0459] 6. The device displays the generated virtual Mars space.

[0460] 7. The user explores Mars in a virtual space, and the generative AI explains the planet's characteristics and environment.

[0461] Example prompt:

[0462] "Please explain the characteristics of Mars in a virtual space of the future of Mars in the year 2100."

[0463] Based on the specific embodiment of this invention, users can effectively study in a virtual space, and the dynamic explanations provided in real time by generative AI further improve the user's learning experience.

[0464] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0465] Step 1:

[0466] The user starts a terminal and logs in.

[0467] Enter: The user powers on the device and accesses the login screen.

[0468] Action: The user enters a username and password.

[0469] Output: The device sends the user's authentication information to the server and displays the main menu if authentication is successful.

[0470] Step 2:

[0471] A generative AI asks users what topics they would like to learn about.

[0472] Input: The user accesses the main menu.

[0473] How it works: The generative AI asks via voice or text, “What would you like to learn about today?”

[0474] Output: The user inputs a topic they want to learn about (e.g., "Information about planets in the universe in the year 2100").

[0475] Step 3:

[0476] The generative AI requests data from the server.

[0477] Input: The user inputs the topic they want to learn about into the generative AI.

[0478] How it works: A generative AI analyzes user input and identifies the required destination and era data.

[0479] Output: The generative AI requests the server to obtain destination data and era data.

[0480] Step 4:

[0481] The server retrieves the required data from the database.

[0482] Input: Data request from generative AI.

[0483] How it works: The server runs SQL queries to get information about the universe from the destination database and information about the year 2100 from the era database.

[0484] Output: The acquired data (e.g., 3D model data of the universe and scenario data for the year 2100) is stored on the server.

[0485] Step 5:

[0486] The server generates the virtual space.

[0487] Input: Acquired data (space information, future information from the year 2100).

[0488] Operation: Based on the data obtained by the server, the shape of the virtual space is formed, objects are placed, and background sounds are generated.

[0489] Output: The generated virtual space data.

[0490] Step 6:

[0491] The server transmits virtual space data to the terminal.

[0492] Input: Data of the generated virtual space.

[0493] Operation: The server divides this data into packets and sends them to the device.

[0494] Output: Virtual space data received by the device.

[0495] Step 7:

[0496] The device displays the virtual space.

[0497] Input: Received virtual space data.

[0498] How it works: The device decodes the data and displays it in a virtual world on a display (e.g. a VR headset).

[0499] Output: The user can visually recognize the virtual space.

[0500] Step 8:

[0501] The generative AI begins to explain within the virtual space.

[0502] Input: The virtual space is displayed.

[0503] How it works: A generative AI uses voice synthesis technology to begin explaining something about space.

[0504] Output: The user can hear the audio description.

[0505] Step 9:

[0506] The user performs operations within the virtual space.

[0507] Input: User interaction with the interface (e.g., use of a controller).

[0508] Actions: The user moves through the virtual space and observes objects.

[0509] Output: Changes in the screen display based on the user's viewpoint and actions.

[0510] Step 10:

[0511] Generative AI dynamically provides content in response to user questions.

[0512] Input: The user asks the generative AI a question (e.g., "What is the atmosphere of Mars like?").

[0513] How it works: A generative AI analyzes the question, searches for relevant information, and provides an explanation.

[0514] Output: Specific information provided by the generative AI (e.g., "The atmosphere of Mars in 2100 will be thin and composed primarily of carbon dioxide."). (Application example 1)

[0515] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0516] In today's educational environment, it is difficult to provide users with the opportunity to learn about various exhibits without visiting the real environment. In addition, in order to learn deeply about a single exhibit, it is necessary to efficiently provide a huge amount of information and provide an interactive experience at the same time. However, it is difficult for current systems to meet these requirements.

[0517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0518] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a means for acquiring information from an exhibition information database and reflecting the exhibit information in the virtual space, and a generating AI for explaining exhibits in the virtual space. This allows users to learn about various exhibits from the comfort of their own homes and gain a deeper understanding of them.

[0519] "Virtual reality technology" refers to technology that uses computer technology to generate virtual environments or scenes that feel real and allow users to immerse themselves in them.

[0520] A "virtual space" is a three-dimensional digital space that is generated using virtual reality technology and that users can experience virtually.

[0521] A "server" is a computer system that provides and processes data via a network, and in this invention is responsible for generating and managing the virtual space.

[0522] A "destination database" is a database that accumulates information about places that a user is expected to visit.

[0523] A "period database" is a database that accumulates information about a specific era.

[0524] An "exhibition information database" is a database that accumulates information about exhibits at museums, exhibitions, etc.

[0525] A "terminal" is a device that allows a user to access and operate a virtual space, and includes smartphones, smart glasses, head-mounted displays, etc.

[0526] A "generative AI" is an agent generated using artificial intelligence technology to provide appropriate information according to the user's operations and circumstances, and in this invention, it is responsible for providing explanations within the virtual space.

[0527] An "exhibit" is an object that a user observes and learns about in a virtual museum or exhibition.

[0528] "3D graphics" is a technology for generating and displaying visual objects in three-dimensional space.

[0529] The embodiment of the present invention is composed of the following elements: a server, a terminal, a user, an exhibition information database, a destination database, an era database, and a system utilizing generation AI.

[0530] First, the server uses hardware to generate the virtual space, specifically a high-performance computer and network environment. The server acquires information from the destination database and the era database, and uses virtual reality technology to create the virtual space based on this information. It also acquires information from the exhibition information database, and reflects the information on the exhibits in the virtual space.

[0531] Next, the terminal is a device that allows the user to access and operate this virtual space. Examples include smartphones, smart glasses, and head-mounted displays (HMDs). The terminal is equipped with 3D graphics and audio playback functions, which provide the user with a realistic experience. In addition, the terminal allows the user to change the viewpoint and operate objects in the virtual space according to user operations.

[0532] Users access the virtual space through their devices and interact with the AI ​​to experience the exhibits and explanations. The AI ​​provides a wealth of information about the exhibits and gives appropriate explanations based on the user's actions and questions. This allows users to learn about the various exhibits and gain a deeper understanding from the comfort of their own home.

[0533] As a concrete example, if a user inputs a prompt such as "Tell me about the Pharaohs of Egypt" into the AI ​​generator, the AI ​​generator will respond as follows:

[0534] "Ancient Egyptian pharaohs were gods and leaders of Egyptian civilization. They oversaw the construction of pyramids and temples and accomplished many historic feats. For example, famous pharaohs are known for their golden masks."

[0535] In this way, the server generates the virtual space, the terminal displays it, and the user operates it, allowing the user to learn and experience within the virtual space. The generating AI provides commentary on the exhibits within the virtual space, supporting the user's learning.

[0536] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0537] Step 1:

[0538] The server receives a request from a user.

[0539] Specifically, a user launches the virtual museum app via a terminal and sends a request for a particular exhibit (e.g., "I want to see the Egyptian exhibit"). The server's input is the request, and it generates as output instructions for proceeding to the next step of processing.

[0540] Step 2:

[0541] The server retrieves information based on the request from a destination database and a period database.

[0542] Specifically, the server executes a database query to retrieve data about the specified destination and time period (e.g., "Egypt", "Ancient"). The input to this step is the user request, and the output is the retrieved data.

[0543] Step 3:

[0544] The server obtains information about the exhibit corresponding to the request from an exhibition information database.

[0545] The server obtains data related to the exhibit (e.g., information about "Egyptian Pharaohs") based on the destination and era data obtained in the previous step. The input is the output data of the previous step, and the output is detailed exhibit information.

[0546] Step 4:

[0547] The server generates a virtual space based on the acquired data.

[0548] Specifically, the server uses virtual reality technology to integrate information on the destination, era, and exhibits, and constructs a 3D virtual space. The input for this step is various data obtained from the database, and the output is the data for the generated virtual space.

[0549] Step 5:

[0550] The terminal receives the virtual space data transmitted from the server and displays it.

[0551] The terminal uses 3D graphics and audio playback functions to provide the user with a realistic virtual space. The input of this step is the virtual space data sent from the server, and the output is the virtual space displayed on the user interface.

[0552] Step 6:

[0553] The user operates within the virtual space through the terminal.

[0554] Specifically, the user performs operations such as clicking on virtual exhibits and changing the viewpoint. The input of this step is the displayed virtual space, and the output is the user's operation data.

[0555] Step 7:

[0556] The generative AI provides commentary about the exhibits in response to user input.

[0557] The server analyzes the user's operation data and generates and sends a prompt sentence (e.g., "Tell me about the Egyptian pharaohs") to the generation AI. The generation AI generates a response based on this prompt and provides an explanation. The input is the user's operation data and the generated prompt sentence, and the output is the explanatory text or audio data.

[0558] Step 8:

[0559] The terminal receives commentary data from the generating AI and provides it to the user.

[0560] Specifically, the device displays text or plays audio to provide an explanation to the user. The input for this step is the explanation data sent from the generation AI, and the output is the display on the user interface or audio playback.

[0561] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0562] The embodiment for implementing the present invention comprises the following elements:

[0563] 1. Server:

[0564] Responsible for generating virtual space.

[0565] Information is obtained from destination and era databases to construct a virtual space.

[0566] By combining it with an emotion engine, it recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI.

[0567] 2. Terminal:

[0568] Displays a virtual space generated by the server.

[0569] It provides an interface that allows the user to perform operations within a virtual space.

[0570] It executes processing related to the display of the virtual space and user operations.

[0571] 3. User:

[0572] Users access and operate the virtual space through a terminal.

[0573] The emotion engine recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI accordingly.

[0574] 4. Generative AI:

[0575] Explanations on subjects are given in a virtual space.

[0576] The emotion engine recognizes the user's emotional state and provides appropriate information and interactions accordingly.

[0577] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted to provide a more personalized learning experience.

[0578] The process flow will be explained below.

[0579] Step 1: The server starts generating the virtual space.

[0580] The server receives destination and time indications.

[0581] The server obtains information on the designated destination from a destination database.

[0582] The server obtains information on the designated era from the era database.

[0583] The server starts the emotion engine and prepares it to recognize the user's emotional state.

[0584] Step 2: The server generates the virtual space.

[0585] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0586] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0587] The server creates the data for the generated virtual space.

[0588] The server monitors the user's emotional state using an emotion engine and adjusts the behavior of the virtual space and generative AI.

[0589] Step 3: Your device displays the virtual space

[0590] The terminal receives the virtual space data generated from the server.

[0591] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0592] Step 4: The user operates in the virtual space

[0593] Users access the virtual space through their terminals.

[0594] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0595] The emotion engine recognizes the user's emotional state and infers emotions by analyzing the user's facial and vocal characteristics.

[0596] Step 5: Generative AI provides commentary on the subject

[0597] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0598] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0599] The emotion engine recognises the user's emotional state and adjusts the generative AI's dialogue style and expressions accordingly.

[0600] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user operates it, allowing the user to proceed with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted, providing a more personalized learning experience. Example 2

[0601] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0602] In conventional educational systems using virtual reality technology, it was difficult to provide a personalized learning experience according to the user's emotions. In addition, the dialogue and information provision using generative AI models were fixed and could not be adjusted in real time, which led to a problem that the learning effect was not fully realized.

[0603] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0604] In this invention, the server includes a means for acquiring information from a destination database and an era database and constructing a virtual space, a means for combining an emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generating AI, a terminal that the user operates in the virtual space, a generating AI that explains subjects in the virtual space, and a means for operating the generating AI model using prompt sentences that provide dialogue and information based on the user's emotional state. This enables a personalized learning experience according to the user's emotions and state.

[0605] A "server" is a computer device that generates a virtual space using virtual reality technology and builds the virtual space based on information obtained from a database.

[0606] A "destination database" is a database that accumulates information about places and destinations to be referenced within a virtual space.

[0607] A "period database" is a database that accumulates information about specific eras or historical periods that can be referenced within a virtual space.

[0608] A "virtual space" is a three-dimensional space digitally constructed using virtual reality technology.

[0609] The "emotion engine" is an engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI based on that information.

[0610] A "terminal" is a device that allows a user to access and operate a virtual space, such as a smartphone or a personal computer.

[0611] "Generative AI" is an artificial intelligence that provides explanations and dialogue on subjects within a virtual space.

[0612] A "generative AI model" is an artificial intelligence model that generates dialogue and information using prompt sentences based on the user's emotional state.

[0613] A "prompt" is a text input that can be fed into a generative AI model to generate specific dialogue or information.

[0614] The present invention provides a system consisting of the following elements. The server generates a virtual space and adjusts the behavior of the virtual space and the generative AI based on the user's emotional state. The user accesses and operates the virtual space through a terminal. A personalized learning experience is provided by combining the emotion engine and the generative AI model.

[0615] server

[0616] The server runs using cloud services such as Amazon Web Services (AWS®) and Google Cloud Platform (GCP®). The server retrieves information from the destination database and the era database, and generates the virtual space based on that information. The server uses emotion engines such as Affectiva®'s SDK and IBM Watson® to recognize the user's emotional state. This makes it possible to adjust the behavior of the virtual space and the generation AI.

[0617] Terminal

[0618] The terminals include devices such as iPhone (registered trademark), Android (registered trademark) smartphones, Windows (registered trademark) PCs, and Mac (registered trademark). The terminals display the virtual space generated from the server and provide an interface for the user to operate within the virtual space. The terminals use, for example, Apple (registered trademark)'s ARKit (registered trademark) or Google (registered trademark)'s ARCore (registered trademark) to display the virtual space superimposed on the real space. The user operates objects in this environment using a touch screen or mouse.

[0619] User

[0620] Users access and interact with the virtual space through their devices. The emotion engine recognizes the user's emotional state in real time. For example, if a user is losing focus while learning about ancient Egypt, the system will use generative AI models to adjust their interactions and activities.

[0621] Generation AI

[0622] Generative AI (e.g., OpenAI® ChatGPT® or Google® BERT®) provides explanations on academic subjects in a virtual space. The user's emotional state is recognized, and appropriate explanations and dialogue are provided based on that information. Generative AI uses prompts to generate dialogue for the user.

[0623] Usage example

[0624] For example, if a user is learning about ancient Egypt, the server will retrieve information about "ancient Egypt" from the destination database and information about "3000 BC" from the period database. Based on this, the server will generate a virtual space and send it to the terminal.

[0625] The device displays this virtual space and the user begins to operate it. If the user's emotional state is determined to be "low concentration" by the emotion engine, the generative AI provides dialogue using prompt sentences such as the following:

[0626] “What interactions or activities should we suggest if a user is having trouble concentrating while learning about Ancient Egypt?”

[0627] Based on this prompt, the generative AI can provide users with interesting quizzes and stories, which can recapture the user's attention and improve their learning outcomes.

[0628] As described above, the present invention provides a personalized learning experience by having the various elements of the server, terminal, user, emotion engine, and generative AI model work in coordination.

[0629] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0630] Step 1:

[0631] The server retrieves information from a destination database and a period database.

[0632] Specifically, the server executes the following queries: "SELECT FROM destination database WHERE destination='Ancient Egypt'" and "SELECT FROM period database WHERE era='3000 BC'".

[0633] Input: Destination database, era database

[0634] Output: Data set for generating virtual space (destination information, time information)

[0635] Step 2:

[0636] A virtual space is generated based on the data acquired by the server.

[0637] Specifically, the server uses the generative AI model to generate a 3D model, which is designed taking into account the user's past behavioral and emotional data.

[0638] Input: Data set (destination information, time information), user's past behavior data, emotion data

[0639] Output: 3D model of the virtual space

[0640] Step 3:

[0641] The server transmits the virtual space to the terminal.

[0642] As a specific operation, the server transmits the generated 3D model data to the terminal using the HTTP protocol.

[0643] Input: 3D model of the virtual space

[0644] Output: 3D model data sent to the device

[0645] Step 4:

[0646] The device displays the virtual space.

[0647] Specifically, the device uses Apple's (registered trademark) ARKit (registered trademark) or Google's (registered trademark) ARCore (registered trademark) to overlay the received 3D model data onto real space and display it.

[0648] Input: 3D model data

[0649] Output: Virtual space displayed on the device

[0650] Step 5:

[0651] The user performs operations within the virtual space.

[0652] Specifically, the user operates objects and views information panels using a touch screen or a mouse.

[0653] Input: Terminal user interface

[0654] Output: User operation data

[0655] Step 6:

[0656] The server uses an emotion engine to recognize the user's emotional state.

[0657] Specifically, the server analyzes the user's facial expressions and voice through the device's camera and microphone to measure their level of concentration and excitement.

[0658] Input: User's facial expression data, voice data

[0659] Output: The user's emotional state.

[0660] Step 7:

[0661] The server uses the generative AI model to generate interactions and activities based on the user's emotional state.

[0662] Specifically, the server inputs the prompt sentence into the generative AI model to get an appropriate response. For example, prompt sentence: "If the user is losing concentration while learning about ancient Egypt, what dialogue or activity should be suggested?"

[0663] Input: prompt text, user's emotional state

[0664] Output: The generated interactions and activities

[0665] Step 8:

[0666] The terminal provides the generated interactions and activities to the user.

[0667] As a specific operation, the terminal displays the generated dialogue content on the screen and outputs it as voice.

[0668] Input: Generated interactions and activities

[0669] Output: The interactions and activities presented to the user

[0670] By following these processing steps, the system of the present invention is able to provide the user with a personalized learning experience. (Application example 2)

[0671] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0672] In conventional systems using virtual space, it was difficult to dynamically provide content according to the user's emotional state, and only uniform information could be provided. As a result, it was difficult to provide learning or purchasing experiences according to the user's interests, and it was not possible to provide personalized services. In addition, there were limited methods for effectively providing detailed information about products and promoting them in virtual space. This made it a challenge to improve user satisfaction.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0674] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a terminal for a user to operate in the virtual space, a generation AI for explaining a subject in the virtual space, an emotion engine for recognizing the emotional state of the user and adjusting the behavior of the virtual space and the generation AI according to the emotional state, and a means for acquiring information from a product database and providing details about products in the virtual space. This makes it possible to provide a dynamic and personalized learning experience or purchasing experience according to the emotional state of the user.

[0675] "Virtual reality technology" is a technology that uses computer technology to create a virtual space that is different from the real world, allowing users to immerse themselves in the experience.

[0676] "Virtual space" refers to a digital three-dimensional space created using virtual reality technology.

[0677] A "server" is an information processing device that generates a virtual space using virtual reality technology and handles user operations and data processing.

[0678] A "destination database" is a database that stores information about destinations for constructing a virtual space.

[0679] An "era database" is a database that stores information about a specific era, and is used when constructing a virtual space.

[0680] A "terminal" is an information device that a user uses to access and operate a virtual space.

[0681] "Generative AI" is a program that uses artificial intelligence technology to provide explanations and dialogue on subjects in a virtual space.

[0682] The "emotion engine" is a component that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI according to that emotional state.

[0683] A "product database" is a database that stores detailed information about products displayed in a virtual space.

[0684] "Means for providing product details" is a function for obtaining information from a product database and providing detailed information about the product to the user within the virtual space.

[0685] In order to carry out the invention, a system including the following elements is configured. First, the server generates a virtual space using virtual reality technology and obtains information from a destination database and an era database to construct the virtual space. This virtual space is displayed on a user terminal and can be operated by the user.

[0686] The server is equipped with an emotion engine that can recognize the user's emotional state. The emotion engine uses facial expression and voice analysis technology to grasp the user's real-time emotions and dynamically adjusts the behavior of the virtual space and the generative AI accordingly. It also has a means of retrieving information from a product database and providing details about products in the virtual space.

[0687] Generative AI can provide personalized product explanations and promotions based on the user's emotional state, providing real-time information tailored to the user's interests and preferences, and improving satisfaction within virtual stores.

[0688] The terminal is an information device that allows users to access and operate the virtual space, and is equipped with 3D graphics and audio playback functions. This allows users to experience the virtual space using both their eyes and ears. It also has a dialogue function based on the user's emotional state using an emotion engine. Specifically, when a user selects a product, the system reads the user's emotions from their facial expressions and voice, and generates an optimal product description.

[0689] As a concrete example of this system, consider a user selecting clothes in a virtual store: if the user is interested in a particular dress, the emotion engine will recognize the user's surprised expression, and the generative AI will provide a personalized description such as "This dress is from our latest collection. It's perfect for a special event!", providing a more personalized experience.

[0690] Examples of prompts for generative AI models include the following:

[0691] "Users are surprised to see a dress from your new collection. Can you help me explain it to them in a way that will keep them interested?"

[0692] In terms of hardware, the server needs to be an information processing device equipped with a high-performance processor and sufficient memory, and as user terminals, smartphones, smart glasses, or head-mounted displays that support virtual reality technology are suitable. For software, it is desirable to use libraries and SDKs (software development kits) for generating virtual spaces, and AI frameworks such as TensorFlow (registered trademark) and PyTorch (registered trademark) for implementing artificial intelligence. For emotion recognition, it is recommended to use OpenCV (registered trademark) or the Emotion Recognition library.

[0693] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0694] Step 1:

[0695] The server generates a virtual space using virtual reality technology. It uses information obtained from the destination database and the era database as input, and generates the shape of the virtual space, the placement of objects, and background sounds based on that information. The output is the data of the generated virtual space.

[0696] Step 2:

[0697] The server transmits the generated virtual space data to the user's terminal. The terminal receives it and displays the virtual space to the user using 3D graphics and audio. The input is the virtual space data transmitted from the server, and the output is to allow the user to experience the virtual space visually and aurally.

[0698] Step 3:

[0699] The user performs operations in the virtual space. Specifically, when the user shows interest in a particular object (e.g., a product), the user makes a selection. The input is the user's operation data, and the output is the object to be selected (e.g., the product ID).

[0700] Step 4:

[0701] The server retrieves detailed information about the selected object from the product database. The input is the ID of the object selected by the user, and the output is detailed information about that object.

[0702] Step 5:

[0703] The server uses an emotion engine to recognize the user's emotional state. To do so, the server uses the user's facial expression data and voice data as input. The emotion engine analyzes these data and outputs the user's emotional state (e.g., surprise, joy, etc.).

[0704] Step 6:

[0705] The server sends a prompt to the generation AI based on the emotional state. Specifically, it creates a prompt such as, "The user is surprised to see a dress from the new collection. Please advise how to explain it so that they will remain interested." The input is the user's emotional state and the selected product information, and the output is the prompt sent to the generation AI.

[0706] Step 7:

[0707] The generation AI generates personalized product descriptions and promotional content based on the prompt text. The input is the prompt text sent from the server, and the output is the product description or promotional content provided to the user.

[0708] Step 8:

[0709] The server transmits the product description and promotional content generated by the AI ​​to the user's device. The device receives it and displays it to the user in the virtual space. The input is the product description generated by the AI, and the output is to provide the user with the product description or promotional content visually and audibly.

[0710] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0711] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0712] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0713] [Third embodiment]

[0714] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0715] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0716] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0717] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0718] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0719] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0720] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0721] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0722] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0723] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0724] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0725] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0726] The embodiment for implementing the present invention comprises the following elements:

[0727] 1. Server:

[0728] Responsible for generating virtual space.

[0729] Information is obtained from destination and era databases to construct a virtual space.

[0730] Executes processing related to the generation of virtual space.

[0731] 2. Terminal:

[0732] Displays a virtual space generated by the server.

[0733] It provides an interface that allows the user to perform operations within a virtual space.

[0734] It executes processing related to the display of the virtual space and user operations.

[0735] 3. User:

[0736] Users access and operate the virtual space through a terminal.

[0737] Students progress through learning by interacting with generative AI in a virtual space.

[0738] 4. Generative AI:

[0739] Explanations on subjects are given in a virtual space.

[0740] Provide appropriate information according to user operations and circumstances.

[0741] Based on the above format, the server generates a virtual space, and the device displays and operates it, allowing the user to proceed with their studies within the virtual space. The generative AI provides explanations on school subjects within the virtual space, supporting the user's learning. For example, if the generative AI acting as a teacher specifies the destination "space" and the time period "2100," the server will obtain space data and recreate the future universe of the year 2100. In the virtual space, children will experience the sensation of being inside a spaceship while the generative AI provides an explanation about space.

[0742] For example, if a generative AI explains the characteristics of a planet in space, children can listen to the generative AI's explanation while exploring the planet.

[0743] The process flow will be explained below.

[0744] Step 1: The server starts generating the virtual space.

[0745] The server receives destination and time indications.

[0746] The server obtains information on the designated destination from a destination database.

[0747] The server obtains information on the designated era from the era database.

[0748] Step 2: The server generates the virtual space.

[0749] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0750] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0751] The server creates the data for the generated virtual space.

[0752] Step 3: Your device displays the virtual space

[0753] The terminal receives the virtual space data generated from the server.

[0754] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0755] Step 4: The user operates in the virtual space

[0756] Users access the virtual space through their terminals.

[0757] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0758] Step 5: Generative AI provides commentary on the subject

[0759] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0760] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0761] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user can interact with the generative AI while performing operations within the virtual space. Example 1

[0762] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0763] In virtual space learning systems using virtual reality technology, the challenge is to ensure that users can access, operate, and experience appropriate educational content in real time. In particular, high real-time operability and adaptability are required for users to effectively progress with their learning in the virtual space. In addition, it is necessary for dynamic explanations by generative AI to immediately respond to the user's learning requests.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0765] In this invention, the server includes a means for generating a virtual space, a means for acquiring information from a destination database and an era database and constructing the virtual space, a generative AI for explaining subjects in the virtual space, a means for accepting user operations via an interface displayed on the terminal, and a means for the generative AI to provide content in real time in the virtual space based on the user's input. This allows the user to access in real time and receive dynamic educational content from the generative AI while operating in the virtual space.

[0766] A "server" is a central processing unit that generates a virtual space, retrieves necessary information from a database, and transmits it to a user terminal.

[0767] The "destination database" is a database that accumulates information about locations and environments necessary for constructing a virtual space.

[0768] A "period database" is a database that stores information about history and future events relating to a particular period.

[0769] A "virtual space" is a virtual environment that is generated using virtual reality technology and in which a user can immerse themselves and interact.

[0770] A "means" is a device or method used to accomplish a particular function or purpose.

[0771] "Terminal" refers to the input devices and display devices that allow a user to access and interact with a virtual space, including, for example, a VR headset or a computer.

[0772] "Generative AI" is an artificial intelligence system that provides educational content within a virtual space and explains information in real time through interaction with the user.

[0773] An "interface" refers to a user input device and a display device that allow a user to access and operate a virtual space through a terminal.

[0774] "Providing content in real time" refers to providing educational information and explanations instantly in response to user operations or requests.

[0775] "Shape of space" refers to the shape of the 3D models of terrain and buildings within the virtual space.

[0776] "Object placement" refers to the position and layout of objects that exist within a virtual space.

[0777] "Background sound" refers to the sounds and music that are played according to the environment and scene within the virtual space.

[0778] This invention provides a system for improving the educational experience of users in a virtual space. The system is mainly composed of a server, a terminal, a user, and a generative AI. A specific embodiment will be described below.

[0779] Server Roles

[0780] The server is the central player in charge of generating the virtual space. It performs the following specific tasks:

[0781] Data Acquisition: The server acquires information from the destination database and the era database. The destination database stores information about places and environments, and the era database stores information about specific eras.

[0782] Space generation: Generate a virtual space based on the acquired data. This generation includes the shape of the space, the placement of objects, and the generation of background sounds.

[0783] Data transmission: The generated virtual space data is sent to the terminal.

[0784] Terminal Roles

[0785] The terminal is a device that allows users to access and operate the virtual space. The terminal performs the following specific operations:

[0786] Displaying the virtual space: Receive the virtual space data sent from the server and display it on a display (e.g. a VR headset).

[0787] Interface provision: Provide an interface (e.g., controllers and gesture recognition) that allows the user to operate within the virtual space.

[0788] Audio playback: Plays ambient sounds in the virtual space and generative AI commentary.

[0789] User Roles

[0790] Users access the virtual space through their devices and carry out learning activities. The users perform the following specific operations:

[0791] Startup and Access: Turn on the device and access the interface. The generative AI asks the user what topic they want to learn about, and the user inputs it accordingly.

[0792] In-spatial interaction: Move freely within the virtual space and learn while listening to commentary provided by the generative AI.

[0793] The role of generative AI

[0794] Generative AI supports the user's learning experience in a virtual space. Specifically, generative AI performs the following tasks:

[0795] Content provision: Based on user input, the virtual space provides explanations of subjects. It provides information in real time and answers user questions.

[0796] Dynamic changes: The commentary changes dynamically in response to user actions, and the content in the virtual space also adapts.

[0797] Examples

[0798] Example scenario:

[0799] 1. The user starts up the device and accesses the virtual space.

[0800] 2. The generative AI asks the user, “What would you like to learn about today?”

[0801] 3. The user responds, "I want to learn about planets in space."

[0802] 4. The generative AI responds, "Now, I will show you information about Mars in the future, from the year 2100."

[0803] 5. The server retrieves space information from the destination database and information about the year 2100 from the time database, and generates a virtual space of the future Mars.

[0804] 6. The device displays the generated virtual Mars space.

[0805] 7. The user explores Mars in a virtual space, and the generative AI explains the planet's characteristics and environment.

[0806] Example prompt:

[0807] "Please explain the characteristics of Mars in a virtual space of the future of Mars in the year 2100."

[0808] Based on the specific embodiment of this invention, users can effectively study in a virtual space, and the dynamic explanations provided in real time by generative AI further improve the user's learning experience.

[0809] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0810] Step 1:

[0811] The user starts a terminal and logs in.

[0812] Enter: The user powers on the device and accesses the login screen.

[0813] Action: The user enters a username and password.

[0814] Output: The device sends the user's authentication information to the server and displays the main menu if authentication is successful.

[0815] Step 2:

[0816] A generative AI asks users what topics they would like to learn about.

[0817] Input: The user accesses the main menu.

[0818] How it works: The generative AI asks via voice or text, “What would you like to learn about today?”

[0819] Output: The user inputs a topic they want to learn about (e.g., "Information about planets in the universe in the year 2100").

[0820] Step 3:

[0821] The generative AI requests data from the server.

[0822] Input: The user inputs the topic they want to learn about into the generative AI.

[0823] How it works: A generative AI analyzes user input and identifies the required destination and era data.

[0824] Output: The generative AI requests the server to obtain destination data and era data.

[0825] Step 4:

[0826] The server retrieves the required data from the database.

[0827] Input: Data request from generative AI.

[0828] How it works: The server runs SQL queries to get information about the universe from the destination database and information about the year 2100 from the era database.

[0829] Output: The acquired data (e.g., 3D model data of the universe and scenario data for the year 2100) is stored on the server.

[0830] Step 5:

[0831] The server generates the virtual space.

[0832] Input: Acquired data (space information, future information from the year 2100).

[0833] Operation: Based on the data obtained by the server, the shape of the virtual space is formed, objects are placed, and background sounds are generated.

[0834] Output: The generated virtual space data.

[0835] Step 6:

[0836] The server transmits virtual space data to the terminal.

[0837] Input: Data of the generated virtual space.

[0838] Operation: The server divides this data into packets and sends them to the device.

[0839] Output: Virtual space data received by the device.

[0840] Step 7:

[0841] The device displays the virtual space.

[0842] Input: Received virtual space data.

[0843] How it works: The device decodes the data and displays it in a virtual world on a display (e.g. a VR headset).

[0844] Output: The user can visually recognize the virtual space.

[0845] Step 8:

[0846] The generative AI begins to explain within the virtual space.

[0847] Input: The virtual space is displayed.

[0848] How it works: A generative AI uses voice synthesis technology to begin explaining something about space.

[0849] Output: The user can hear the audio description.

[0850] Step 9:

[0851] The user performs operations within the virtual space.

[0852] Input: User interaction with the interface (e.g., use of a controller).

[0853] Actions: The user moves through the virtual space and observes objects.

[0854] Output: Changes in the screen display based on the user's viewpoint and actions.

[0855] Step 10:

[0856] Generative AI dynamically provides content in response to user questions.

[0857] Input: The user asks the generative AI a question (e.g., "What is the atmosphere of Mars like?").

[0858] How it works: A generative AI analyzes the question, searches for relevant information, and provides an explanation.

[0859] Output: Specific information provided by the generative AI (e.g., "The atmosphere of Mars in 2100 will be thin and composed primarily of carbon dioxide."). (Application example 1)

[0860] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] In today's educational environment, it is difficult to provide users with the opportunity to learn about various exhibits without visiting the real environment. In addition, in order to learn deeply about a single exhibit, it is necessary to efficiently provide a huge amount of information and provide an interactive experience at the same time. However, it is difficult for current systems to meet these requirements.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0863] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a means for acquiring information from an exhibition information database and reflecting the exhibit information in the virtual space, and a generating AI for explaining exhibits in the virtual space. This allows users to learn about various exhibits from the comfort of their own homes and gain a deeper understanding of them.

[0864] "Virtual reality technology" refers to technology that uses computer technology to generate virtual environments or scenes that feel real and allow users to immerse themselves in them.

[0865] A "virtual space" is a three-dimensional digital space that is generated using virtual reality technology and that users can experience virtually.

[0866] A "server" is a computer system that provides and processes data via a network, and in this invention is responsible for generating and managing the virtual space.

[0867] A "destination database" is a database that accumulates information about places that a user is expected to visit.

[0868] A "period database" is a database that accumulates information about a specific era.

[0869] An "exhibition information database" is a database that accumulates information about exhibits at museums, exhibitions, etc.

[0870] A "terminal" is a device that allows a user to access and operate a virtual space, and includes smartphones, smart glasses, head-mounted displays, etc.

[0871] A "generative AI" is an agent generated using artificial intelligence technology to provide appropriate information according to the user's operations and circumstances, and in this invention, it is responsible for providing explanations within the virtual space.

[0872] An "exhibit" is an object that a user observes and learns about in a virtual museum or exhibition.

[0873] "3D graphics" is a technology for generating and displaying visual objects in three-dimensional space.

[0874] The embodiment of the present invention is composed of the following elements: a server, a terminal, a user, an exhibition information database, a destination database, an era database, and a system utilizing generation AI.

[0875] First, the server uses hardware to generate the virtual space, specifically a high-performance computer and network environment. The server acquires information from the destination database and the era database, and uses virtual reality technology to create the virtual space based on this information. It also acquires information from the exhibition information database, and reflects the information on the exhibits in the virtual space.

[0876] Next, the terminal is a device that allows the user to access and operate this virtual space. Examples include smartphones, smart glasses, and head-mounted displays (HMDs). The terminal is equipped with 3D graphics and audio playback functions, which provide the user with a realistic experience. In addition, the terminal allows the user to change the viewpoint and operate objects in the virtual space according to user operations.

[0877] Users access the virtual space through their devices and interact with the AI ​​to experience the exhibits and explanations. The AI ​​provides a wealth of information about the exhibits and gives appropriate explanations based on the user's actions and questions. This allows users to learn about the various exhibits and gain a deeper understanding from the comfort of their own home.

[0878] As a concrete example, if a user inputs a prompt such as "Tell me about the Pharaohs of Egypt" into the AI ​​generator, the AI ​​generator will respond as follows:

[0879] "Ancient Egyptian pharaohs were gods and leaders of Egyptian civilization. They oversaw the construction of pyramids and temples and accomplished many historic feats. For example, famous pharaohs are known for their golden masks."

[0880] In this way, the server generates the virtual space, the terminal displays it, and the user operates it, allowing the user to learn and experience within the virtual space. The generating AI provides commentary on the exhibits within the virtual space, supporting the user's learning.

[0881] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0882] Step 1:

[0883] The server receives a request from a user.

[0884] Specifically, a user launches the virtual museum app via a terminal and sends a request for a particular exhibit (e.g., "I want to see the Egyptian exhibit"). The server's input is the request, and it generates as output instructions for proceeding to the next step of processing.

[0885] Step 2:

[0886] The server retrieves information based on the request from a destination database and a period database.

[0887] Specifically, the server executes a database query to retrieve data about the specified destination and time period (e.g., "Egypt", "Ancient"). The input to this step is the user request, and the output is the retrieved data.

[0888] Step 3:

[0889] The server obtains information about the exhibit corresponding to the request from an exhibition information database.

[0890] The server obtains data related to the exhibit (e.g., information about "Egyptian Pharaohs") based on the destination and era data obtained in the previous step. The input is the output data of the previous step, and the output is detailed exhibit information.

[0891] Step 4:

[0892] The server generates a virtual space based on the acquired data.

[0893] Specifically, the server uses virtual reality technology to integrate information on the destination, era, and exhibits, and constructs a 3D virtual space. The input for this step is various data obtained from the database, and the output is the data for the generated virtual space.

[0894] Step 5:

[0895] The terminal receives the virtual space data transmitted from the server and displays it.

[0896] The terminal uses 3D graphics and audio playback functions to provide the user with a realistic virtual space. The input of this step is the virtual space data sent from the server, and the output is the virtual space displayed on the user interface.

[0897] Step 6:

[0898] The user operates within the virtual space through the terminal.

[0899] Specifically, the user performs operations such as clicking on virtual exhibits and changing the viewpoint. The input of this step is the displayed virtual space, and the output is the user's operation data.

[0900] Step 7:

[0901] The generative AI provides commentary about the exhibits in response to user input.

[0902] The server analyzes the user's operation data and generates and sends a prompt sentence (e.g., "Tell me about the Egyptian pharaohs") to the generation AI. The generation AI generates a response based on this prompt and provides an explanation. The input is the user's operation data and the generated prompt sentence, and the output is the explanatory text or audio data.

[0903] Step 8:

[0904] The terminal receives commentary data from the generating AI and provides it to the user.

[0905] Specifically, the device displays text or plays audio to provide an explanation to the user. The input for this step is the explanation data sent from the generation AI, and the output is the display on the user interface or audio playback.

[0906] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0907] The embodiment for implementing the present invention comprises the following elements:

[0908] 1. Server:

[0909] Responsible for generating virtual space.

[0910] Information is obtained from destination and era databases to construct a virtual space.

[0911] By combining it with an emotion engine, it recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI.

[0912] 2. Terminal:

[0913] Displays a virtual space generated by the server.

[0914] It provides an interface that allows the user to perform operations within a virtual space.

[0915] It executes processing related to the display of the virtual space and user operations.

[0916] 3. User:

[0917] Users access and operate the virtual space through a terminal.

[0918] The emotion engine recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI accordingly.

[0919] 4. Generative AI:

[0920] Explanations on subjects are given in a virtual space.

[0921] The emotion engine recognizes the user's emotional state and provides appropriate information and interactions accordingly.

[0922] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted to provide a more personalized learning experience.

[0923] The process flow will be explained below.

[0924] Step 1: The server starts generating the virtual space.

[0925] The server receives destination and time indications.

[0926] The server obtains information on the designated destination from a destination database.

[0927] The server obtains information on the designated era from the era database.

[0928] The server starts the emotion engine and prepares it to recognize the user's emotional state.

[0929] Step 2: The server generates the virtual space.

[0930] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[0931] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[0932] The server creates the data for the generated virtual space.

[0933] The server monitors the user's emotional state using an emotion engine and adjusts the behavior of the virtual space and generative AI.

[0934] Step 3: Your device displays the virtual space

[0935] The terminal receives the virtual space data generated from the server.

[0936] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[0937] Step 4: The user operates in the virtual space

[0938] Users access the virtual space through their terminals.

[0939] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[0940] The emotion engine recognizes the user's emotional state and infers emotions by analyzing the user's facial and vocal characteristics.

[0941] Step 5: Generative AI provides commentary on the subject

[0942] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[0943] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[0944] The emotion engine recognises the user's emotional state and adjusts the generative AI's dialogue style and expressions accordingly.

[0945] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user operates it, allowing the user to proceed with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted, providing a more personalized learning experience. Example 2

[0946] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0947] In conventional educational systems using virtual reality technology, it was difficult to provide a personalized learning experience according to the user's emotions. In addition, the dialogue and information provision using generative AI models were fixed and could not be adjusted in real time, which led to a problem that the learning effect was not fully realized.

[0948] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0949] In this invention, the server includes a means for acquiring information from a destination database and an era database and constructing a virtual space, a means for combining an emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generating AI, a terminal that the user operates in the virtual space, a generating AI that explains subjects in the virtual space, and a means for operating the generating AI model using prompt sentences that provide dialogue and information based on the user's emotional state. This enables a personalized learning experience according to the user's emotions and state.

[0950] A "server" is a computer device that generates a virtual space using virtual reality technology and builds the virtual space based on information obtained from a database.

[0951] A "destination database" is a database that accumulates information about places and destinations to be referenced within a virtual space.

[0952] A "period database" is a database that accumulates information about specific eras or historical periods that can be referenced within a virtual space.

[0953] A "virtual space" is a three-dimensional space digitally constructed using virtual reality technology.

[0954] The "emotion engine" is an engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI based on that information.

[0955] A "terminal" is a device that allows a user to access and operate a virtual space, such as a smartphone or a personal computer.

[0956] "Generative AI" is an artificial intelligence that provides explanations and dialogue on subjects within a virtual space.

[0957] A "generative AI model" is an artificial intelligence model that generates dialogue and information using prompt sentences based on the user's emotional state.

[0958] A "prompt" is a text input that can be fed into a generative AI model to generate specific dialogue or information.

[0959] The present invention provides a system consisting of the following elements. The server generates a virtual space and adjusts the behavior of the virtual space and the generative AI based on the user's emotional state. The user accesses and operates the virtual space through a terminal. A personalized learning experience is provided by combining the emotion engine and the generative AI model.

[0960] server

[0961] The server runs using cloud services such as Amazon Web Services (AWS®) and Google Cloud Platform (GCP®). The server retrieves information from the destination database and the era database, and generates the virtual space based on that information. The server uses emotion engines such as Affectiva®'s SDK and IBM Watson® to recognize the user's emotional state. This makes it possible to adjust the behavior of the virtual space and the generation AI.

[0962] Terminal

[0963] The terminals include devices such as iPhone (registered trademark), Android (registered trademark) smartphones, Windows (registered trademark) PCs, and Mac (registered trademark). The terminals display the virtual space generated from the server and provide an interface for the user to operate within the virtual space. The terminals use, for example, Apple (registered trademark)'s ARKit (registered trademark) or Google (registered trademark)'s ARCore (registered trademark) to display the virtual space superimposed on the real space. The user operates objects in this environment using a touch screen or mouse.

[0964] User

[0965] Users access and interact with the virtual space through their devices. The emotion engine recognizes the user's emotional state in real time. For example, if a user is losing focus while learning about ancient Egypt, the system will use generative AI models to adjust their interactions and activities.

[0966] Generation AI

[0967] Generative AI (e.g., OpenAI® ChatGPT® or Google® BERT®) provides explanations on academic subjects in a virtual space. The user's emotional state is recognized, and appropriate explanations and dialogue are provided based on that information. Generative AI uses prompts to generate dialogue for the user.

[0968] Usage example

[0969] For example, if a user is learning about ancient Egypt, the server will retrieve information about "ancient Egypt" from the destination database and information about "3000 BC" from the period database. Based on this, the server will generate a virtual space and send it to the terminal.

[0970] The device displays this virtual space and the user begins to operate it. If the user's emotional state is determined to be "low concentration" by the emotion engine, the generative AI provides dialogue using prompt sentences such as the following:

[0971] “What interactions or activities should we suggest if a user is having trouble concentrating while learning about Ancient Egypt?”

[0972] Based on this prompt, the generative AI can provide users with interesting quizzes and stories, which can recapture the user's attention and improve their learning outcomes.

[0973] As described above, the present invention provides a personalized learning experience by having the various elements of the server, terminal, user, emotion engine, and generative AI model work in coordination.

[0974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0975] Step 1:

[0976] The server retrieves information from a destination database and a period database.

[0977] Specifically, the server executes the following queries: "SELECT FROM destination database WHERE destination='Ancient Egypt'" and "SELECT FROM period database WHERE era='3000 BC'".

[0978] Input: Destination database, era database

[0979] Output: Data set for generating virtual space (destination information, time information)

[0980] Step 2:

[0981] A virtual space is generated based on the data acquired by the server.

[0982] Specifically, the server uses the generative AI model to generate a 3D model, which is designed taking into account the user's past behavioral and emotional data.

[0983] Input: Data set (destination information, time information), user's past behavior data, emotion data

[0984] Output: 3D model of the virtual space

[0985] Step 3:

[0986] The server transmits the virtual space to the terminal.

[0987] As a specific operation, the server transmits the generated 3D model data to the terminal using the HTTP protocol.

[0988] Input: 3D model of the virtual space

[0989] Output: 3D model data sent to the device

[0990] Step 4:

[0991] The device displays the virtual space.

[0992] Specifically, the device uses Apple's (registered trademark) ARKit (registered trademark) or Google's (registered trademark) ARCore (registered trademark) to overlay the received 3D model data onto real space.

[0993] Input: 3D model data

[0994] Output: Virtual space displayed on the device

[0995] Step 5:

[0996] The user performs operations within the virtual space.

[0997] Specifically, the user operates objects and views information panels using a touch screen or a mouse.

[0998] Input: Terminal user interface

[0999] Output: User operation data

[1000] Step 6:

[1001] The server uses an emotion engine to recognize the user's emotional state.

[1002] Specifically, the server analyzes the user's facial expressions and voice through the device's camera and microphone to measure their level of concentration and excitement.

[1003] Input: User's facial expression data, voice data

[1004] Output: The user's emotional state.

[1005] Step 7:

[1006] The server uses the generative AI model to generate interactions and activities based on the user's emotional state.

[1007] Specifically, the server inputs the prompt sentence into the generative AI model to get an appropriate response. For example, prompt sentence: "If the user is losing concentration while learning about ancient Egypt, what dialogue or activity should be suggested?"

[1008] Input: prompt text, user's emotional state

[1009] Output: The generated interactions and activities

[1010] Step 8:

[1011] The terminal provides the generated interactions and activities to the user.

[1012] As a specific operation, the terminal displays the generated dialogue content on the screen and outputs it as voice.

[1013] Input: Generated interactions and activities

[1014] Output: The interactions and activities presented to the user

[1015] By following these processing steps, the system of the present invention is able to provide the user with a personalized learning experience. (Application example 2)

[1016] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1017] In conventional systems using virtual space, it was difficult to dynamically provide content according to the user's emotional state, and only uniform information could be provided. As a result, it was difficult to provide learning or purchasing experiences according to the user's interests, and it was not possible to provide personalized services. In addition, there were limited methods for effectively providing detailed information about products and promoting them in virtual space. This made it a challenge to improve user satisfaction.

[1018] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1019] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a terminal for a user to operate in the virtual space, a generation AI for explaining a subject in the virtual space, an emotion engine for recognizing the emotional state of the user and adjusting the behavior of the virtual space and the generation AI according to the emotional state, and a means for acquiring information from a product database and providing details about products in the virtual space. This makes it possible to provide a dynamic and personalized learning experience or purchasing experience according to the emotional state of the user.

[1020] "Virtual reality technology" is a technology that uses computer technology to create a virtual space that is different from the real world, allowing users to immerse themselves in the experience.

[1021] "Virtual space" refers to a digital three-dimensional space created using virtual reality technology.

[1022] A "server" is an information processing device that generates a virtual space using virtual reality technology and handles user operations and data processing.

[1023] A "destination database" is a database that stores information about destinations for constructing a virtual space.

[1024] An "era database" is a database that stores information about a specific era, and is used when constructing a virtual space.

[1025] A "terminal" is an information device that a user uses to access and operate a virtual space.

[1026] "Generative AI" is a program that uses artificial intelligence technology to provide explanations and dialogue on subjects in a virtual space.

[1027] The "emotion engine" is a component that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI according to that emotional state.

[1028] A "product database" is a database that stores detailed information about products displayed in a virtual space.

[1029] "Means for providing product details" is a function for obtaining information from a product database and providing detailed information about the product to the user within the virtual space.

[1030] In order to carry out the invention, a system including the following elements is configured. First, the server generates a virtual space using virtual reality technology and obtains information from a destination database and an era database to construct the virtual space. This virtual space is displayed on a user terminal and can be operated by the user.

[1031] The server is equipped with an emotion engine that can recognize the user's emotional state. The emotion engine uses facial expression and voice analysis technology to grasp the user's real-time emotions and dynamically adjusts the behavior of the virtual space and the generative AI accordingly. It also has a means of retrieving information from a product database and providing details about products in the virtual space.

[1032] Generative AI can provide personalized product explanations and promotions based on the user's emotional state, providing real-time information tailored to the user's interests and preferences, and improving satisfaction within virtual stores.

[1033] The terminal is an information device that allows users to access and operate the virtual space, and is equipped with 3D graphics and audio playback functions. This allows users to experience the virtual space using both their eyes and ears. It also has a dialogue function based on the user's emotional state using an emotion engine. Specifically, when a user selects a product, the system reads the user's emotions from their facial expressions and voice, and generates an optimal product description.

[1034] As a concrete example of this system, consider a user selecting clothes in a virtual store: if the user is interested in a particular dress, the emotion engine will recognize the user's surprised expression, and the generative AI will provide a personalized description such as "This dress is from our latest collection. It's perfect for a special event!", providing a more personalized experience.

[1035] Examples of prompts for generative AI models include the following:

[1036] "Users are surprised to see a dress from your new collection. Can you help me explain it to them in a way that will keep them interested?"

[1037] In terms of hardware, the server needs to be an information processing device equipped with a high-performance processor and sufficient memory, and as user terminals, smartphones, smart glasses, or head-mounted displays that support virtual reality technology are suitable. For software, it is desirable to use libraries and SDKs (software development kits) for generating virtual spaces, and AI frameworks such as TensorFlow (registered trademark) and PyTorch (registered trademark) for implementing artificial intelligence. For emotion recognition, it is recommended to use OpenCV (registered trademark) or the Emotion Recognition library.

[1038] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1039] Step 1:

[1040] The server generates a virtual space using virtual reality technology. It uses information obtained from the destination database and the era database as input, and generates the shape of the virtual space, the placement of objects, and background sounds based on that information. The output is the data of the generated virtual space.

[1041] Step 2:

[1042] The server transmits the generated virtual space data to the user's terminal. The terminal receives it and displays the virtual space to the user using 3D graphics and audio. The input is the virtual space data transmitted from the server, and the output is to allow the user to experience the virtual space visually and aurally.

[1043] Step 3:

[1044] The user performs operations in the virtual space. Specifically, when the user shows interest in a particular object (e.g., a product), the user makes a selection. The input is the user's operation data, and the output is the object to be selected (e.g., the product ID).

[1045] Step 4:

[1046] The server retrieves detailed information about the selected object from the product database. The input is the ID of the object selected by the user, and the output is detailed information about that object.

[1047] Step 5:

[1048] The server uses an emotion engine to recognize the user's emotional state. To do so, the server uses the user's facial expression data and voice data as input. The emotion engine analyzes these data and outputs the user's emotional state (e.g., surprise, joy, etc.).

[1049] Step 6:

[1050] The server sends a prompt to the generation AI based on the emotional state. Specifically, it creates a prompt such as, "The user is surprised to see a dress from the new collection. Please advise how to explain it so that they will remain interested." The input is the user's emotional state and the selected product information, and the output is the prompt sent to the generation AI.

[1051] Step 7:

[1052] The generation AI generates personalized product descriptions and promotional content based on the prompt text. The input is the prompt text sent from the server, and the output is the product description or promotional content provided to the user.

[1053] Step 8:

[1054] The server transmits the product description and promotional content generated by the AI ​​to the user's device. The device receives it and displays it to the user in the virtual space. The input is the product description generated by the AI, and the output is to provide the user with the product description or promotional content visually and audibly.

[1055] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1056] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1057] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1058] [Fourth embodiment]

[1059] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1060] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1061] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1062] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1063] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1064] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1065] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1066] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1067] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1070] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1071] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1072] The embodiment for implementing the present invention comprises the following elements:

[1073] 1. Server:

[1074] Responsible for generating virtual space.

[1075] Information is obtained from destination and era databases to construct a virtual space.

[1076] Executes processing related to the generation of virtual space.

[1077] 2. Terminal:

[1078] Displays a virtual space generated by the server.

[1079] It provides an interface that allows the user to perform operations within a virtual space.

[1080] It executes processing related to the display of the virtual space and user operations.

[1081] 3. User:

[1082] Users access and operate the virtual space through a terminal.

[1083] Students progress through learning by interacting with generative AI in a virtual space.

[1084] 4. Generative AI:

[1085] Explanations on subjects are given in a virtual space.

[1086] Provide appropriate information according to user operations and circumstances.

[1087] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with their learning in the virtual space. The generative AI provides explanations on subjects in the virtual space to support the user's learning. For example, if the generative AI playing the role of teacher specifies the destination "space" and the time period "2100," the server acquires space data and reproduces the future universe in 2100. In the virtual space, the children experience the sensation of being in a spaceship while the generative AI provides an explanation on the universe. For example, if the generative AI explains the characteristics of a planet in space, the children can listen to the generative AI's explanation while exploring the planet.

[1088] The process flow will be explained below.

[1089] Step 1: The server starts generating the virtual space.

[1090] The server receives destination and time indications.

[1091] The server obtains information on the designated destination from a destination database.

[1092] The server obtains information on the designated era from the era database.

[1093] Step 2: The server generates the virtual space.

[1094] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[1095] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[1096] The server creates the data for the generated virtual space.

[1097] Step 3: Your device displays the virtual space

[1098] The terminal receives the virtual space data generated from the server.

[1099] Based on the data it receives, the device displays the virtual space as 3D graphics and audio.

[1100] Step 4: The user operates in the virtual space

[1101] Users access the virtual space through their terminals.

[1102] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[1103] Step 5: Generative AI provides commentary on the subject

[1104] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[1105] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[1106] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user can interact with the generative AI while performing operations within the virtual space. Example 1

[1107] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1108] In virtual space learning systems using virtual reality technology, the challenge is to ensure that users can access, operate, and experience appropriate educational content in real time. In particular, high real-time operability and adaptability are required for users to effectively progress with their learning in the virtual space. In addition, it is necessary for dynamic explanations by generative AI to immediately respond to the user's learning requests.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1110] In this invention, the server includes a means for generating a virtual space, a means for acquiring information from a destination database and an era database and constructing the virtual space, a generative AI for explaining subjects in the virtual space, a means for accepting user operations via an interface displayed on the terminal, and a means for the generative AI to provide content in real time in the virtual space based on the user's input. This allows the user to access in real time and receive dynamic educational content from the generative AI while operating in the virtual space.

[1111] A "server" is a central processing unit that generates a virtual space, retrieves necessary information from a database, and transmits it to a user terminal.

[1112] The "destination database" is a database that accumulates information about locations and environments necessary for constructing a virtual space.

[1113] A "period database" is a database that stores information about history and future events relating to a particular period.

[1114] A "virtual space" is a virtual environment that is generated using virtual reality technology and in which a user can immerse themselves and interact.

[1115] A "means" is a device or method used to accomplish a particular function or purpose.

[1116] "Terminal" refers to the input devices and display devices that allow a user to access and interact with a virtual space, including, for example, a VR headset or a computer.

[1117] "Generative AI" is an artificial intelligence system that provides educational content within a virtual space and explains information in real time through interaction with the user.

[1118] An "interface" refers to a user input device and a display device that allow a user to access and operate a virtual space through a terminal.

[1119] "Providing content in real time" refers to providing educational information and explanations instantly in response to user operations or requests.

[1120] "Shape of space" refers to the shape of the 3D models of terrain and buildings within the virtual space.

[1121] "Object placement" refers to the position and layout of objects that exist within a virtual space.

[1122] "Background sound" refers to the sounds and music that are played according to the environment and scene within the virtual space.

[1123] This invention provides a system for improving the educational experience of users in a virtual space. The system is mainly composed of a server, a terminal, a user, and a generative AI. A specific embodiment will be described below.

[1124] Server Roles

[1125] The server is the central player in charge of generating the virtual space. It performs the following specific tasks:

[1126] Data Acquisition: The server acquires information from the destination database and the era database. The destination database stores information about places and environments, and the era database stores information about specific eras.

[1127] Space generation: Generate a virtual space based on the acquired data. This generation includes the shape of the space, the placement of objects, and the generation of background sounds.

[1128] Data transmission: The generated virtual space data is sent to the terminal.

[1129] Terminal Roles

[1130] The terminal is a device that allows users to access and operate the virtual space. The terminal performs the following specific operations:

[1131] Displaying the virtual space: Receive the virtual space data sent from the server and display it on a display (e.g. a VR headset).

[1132] Interface provision: Provide an interface (e.g., controllers and gesture recognition) that allows the user to operate within the virtual space.

[1133] Audio playback: Plays ambient sounds in the virtual space and generative AI commentary.

[1134] User Roles

[1135] Users access the virtual space through their devices and carry out learning activities. The users perform the following specific operations:

[1136] Startup and Access: Turn on the device and access the interface. The generative AI asks the user what topic they want to learn about, and the user inputs it accordingly.

[1137] In-spatial interaction: Move freely within the virtual space and learn while listening to commentary provided by the generative AI.

[1138] The role of generative AI

[1139] Generative AI supports the user's learning experience in a virtual space. Specifically, generative AI performs the following tasks:

[1140] Providing content: Based on user input, the virtual space provides explanations of subjects, providing real-time information and answering user questions.

[1141] Dynamic changes: The commentary changes dynamically in response to user actions, and the content in the virtual space also adapts.

[1142] Examples

[1143] Example scenario:

[1144] 1. The user starts up the device and accesses the virtual space.

[1145] 2. The generative AI asks the user, “What would you like to learn about today?”

[1146] 3. The user responds, "I want to learn about planets in space."

[1147] 4. The generative AI responds, "Now, I will show you information about Mars in the future, from the year 2100."

[1148] 5. The server retrieves space information from the destination database and information about the year 2100 from the time database, and generates a virtual space of the future Mars.

[1149] 6. The device displays the generated virtual Mars space.

[1150] 7. The user explores Mars in a virtual space, and the generative AI explains the planet's characteristics and environment.

[1151] Example prompt:

[1152] "Please explain the characteristics of Mars in a virtual space of the future of Mars in the year 2100."

[1153] Based on the specific embodiment of this invention, users can effectively study in a virtual space, and the dynamic explanations provided in real time by generative AI further improve the user's learning experience.

[1154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] The user starts a terminal and logs in.

[1157] Enter: The user powers on the device and accesses the login screen.

[1158] Action: The user enters a username and password.

[1159] Output: The device sends the user's authentication information to the server and displays the main menu if authentication is successful.

[1160] Step 2:

[1161] A generative AI asks users what topics they would like to learn about.

[1162] Input: The user accesses the main menu.

[1163] How it works: The generative AI asks via voice or text, “What would you like to learn about today?”

[1164] Output: The user inputs a topic they want to learn about (e.g., "Information about planets in the universe in the year 2100").

[1165] Step 3:

[1166] The generative AI requests data from the server.

[1167] Input: The user inputs the topic they want to learn about into the generative AI.

[1168] How it works: A generative AI analyzes user input and identifies the required destination and era data.

[1169] Output: The generative AI requests the server to obtain destination data and era data.

[1170] Step 4:

[1171] The server retrieves the required data from the database.

[1172] Input: Data request from generative AI.

[1173] How it works: The server runs SQL queries to get information about the universe from the destination database and information about the year 2100 from the era database.

[1174] Output: The acquired data (e.g., 3D model data of the universe and scenario data for the year 2100) is stored on the server.

[1175] Step 5:

[1176] The server generates the virtual space.

[1177] Input: Acquired data (space information, future information from the year 2100).

[1178] Operation: Based on the data obtained by the server, the shape of the virtual space is formed, objects are placed, and background sounds are generated.

[1179] Output: The generated virtual space data.

[1180] Step 6:

[1181] The server transmits virtual space data to the terminal.

[1182] Input: Data of the generated virtual space.

[1183] Operation: The server divides this data into packets and sends them to the device.

[1184] Output: Virtual space data received by the device.

[1185] Step 7:

[1186] The device displays the virtual space.

[1187] Input: Received virtual space data.

[1188] How it works: The device decodes the data and displays it in a virtual world on a display (e.g. a VR headset).

[1189] Output: The user can visually recognize the virtual space.

[1190] Step 8:

[1191] The generative AI begins to explain within the virtual space.

[1192] Input: The virtual space is displayed.

[1193] How it works: A generative AI uses voice synthesis technology to begin explaining something about space.

[1194] Output: The user can hear the audio description.

[1195] Step 9:

[1196] The user performs operations within the virtual space.

[1197] Input: User interaction with the interface (e.g., use of a controller).

[1198] Actions: The user moves through the virtual space and observes objects.

[1199] Output: Changes in the screen display based on the user's viewpoint and actions.

[1200] Step 10:

[1201] Generative AI dynamically provides content in response to user questions.

[1202] Input: The user asks the generative AI a question (e.g., "What is the atmosphere of Mars like?").

[1203] How it works: A generative AI analyzes the question, searches for relevant information, and provides an explanation.

[1204] Output: Specific information provided by the generative AI (e.g., "The atmosphere of Mars in 2100 will be thin and composed primarily of carbon dioxide."). (Application example 1)

[1205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1206] In today's educational environment, it is difficult to provide users with the opportunity to learn about various exhibits without visiting the real environment. In addition, in order to learn deeply about a single exhibit, it is necessary to efficiently provide a huge amount of information and provide an interactive experience at the same time. However, it is difficult for current systems to meet these requirements.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1208] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a means for acquiring information from an exhibition information database and reflecting the exhibit information in the virtual space, and a generating AI for explaining exhibits in the virtual space. This allows users to learn about various exhibits from the comfort of their own homes and gain a deeper understanding of them.

[1209] "Virtual reality technology" refers to technology that uses computer technology to generate virtual environments or scenes that feel real and allow users to immerse themselves in them.

[1210] A "virtual space" is a three-dimensional digital space that is generated using virtual reality technology and that users can experience virtually.

[1211] A "server" is a computer system that provides and processes data via a network, and in this invention is responsible for generating and managing the virtual space.

[1212] A "destination database" is a database that accumulates information about places that a user is expected to visit.

[1213] A "period database" is a database that accumulates information about a specific era.

[1214] An "exhibition information database" is a database that accumulates information about exhibits at museums, exhibitions, etc.

[1215] A "terminal" is a device that allows a user to access and operate a virtual space, and includes smartphones, smart glasses, head-mounted displays, etc.

[1216] A "generative AI" is an agent generated using artificial intelligence technology to provide appropriate information according to the user's operations and circumstances, and in this invention, it is responsible for providing explanations within the virtual space.

[1217] An "exhibit" is an object that a user observes and learns about in a virtual museum or exhibition.

[1218] "3D graphics" is a technology for generating and displaying visual objects in three-dimensional space.

[1219] The embodiment of the present invention is composed of the following elements: a server, a terminal, a user, an exhibition information database, a destination database, an era database, and a system utilizing generation AI.

[1220] First, the server uses hardware to generate the virtual space, specifically a high-performance computer and network environment. The server acquires information from the destination database and the era database, and uses virtual reality technology to create the virtual space based on this information. It also acquires information from the exhibition information database, and reflects the information on the exhibits in the virtual space.

[1221] Next, the terminal is a device that allows the user to access and operate this virtual space. Examples include smartphones, smart glasses, and head-mounted displays (HMDs). The terminal is equipped with 3D graphics and audio playback functions, which provide the user with a realistic experience. In addition, the terminal allows the user to change the viewpoint and operate objects in the virtual space according to user operations.

[1222] Users access the virtual space through their devices and interact with the AI ​​to experience the exhibits and explanations. The AI ​​provides a wealth of information about the exhibits and gives appropriate explanations based on the user's actions and questions. This allows users to learn about the various exhibits and gain a deeper understanding from the comfort of their own home.

[1223] As a concrete example, if a user inputs a prompt such as "Tell me about the Pharaohs of Egypt" into the AI ​​generator, the AI ​​generator will respond as follows:

[1224] "Ancient Egyptian pharaohs were gods and leaders of Egyptian civilization. They oversaw the construction of pyramids and temples and accomplished many historic feats. For example, famous pharaohs are known for their golden masks."

[1225] In this way, the server generates the virtual space, the terminal displays it, and the user operates it, allowing the user to learn and experience within the virtual space. The generating AI provides commentary on the exhibits within the virtual space, supporting the user's learning.

[1226] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1227] Step 1:

[1228] The server receives a request from a user.

[1229] Specifically, a user launches the virtual museum app via a terminal and sends a request for a particular exhibit (e.g., "I want to see the Egyptian exhibit"). The server's input is the request, and it generates as output instructions for proceeding to the next step of processing.

[1230] Step 2:

[1231] The server retrieves information based on the request from a destination database and a period database.

[1232] Specifically, the server executes a database query to retrieve data about the specified destination and time period (e.g., "Egypt", "Ancient"). The input to this step is the user request, and the output is the retrieved data.

[1233] Step 3:

[1234] The server obtains information about the exhibit corresponding to the request from an exhibition information database.

[1235] The server obtains data related to the exhibit (e.g., information about "Egyptian Pharaohs") based on the destination and era data obtained in the previous step. The input is the output data of the previous step, and the output is detailed exhibit information.

[1236] Step 4:

[1237] The server generates a virtual space based on the acquired data.

[1238] Specifically, the server uses virtual reality technology to integrate information on the destination, era, and exhibits, and constructs a 3D virtual space. The input for this step is various data obtained from the database, and the output is the data for the generated virtual space.

[1239] Step 5:

[1240] The terminal receives the virtual space data transmitted from the server and displays it.

[1241] The terminal uses 3D graphics and audio playback functions to provide the user with a realistic virtual space. The input of this step is the virtual space data sent from the server, and the output is the virtual space displayed on the user interface.

[1242] Step 6:

[1243] The user operates within the virtual space through the terminal.

[1244] Specifically, the user performs operations such as clicking on virtual exhibits and changing the viewpoint. The input of this step is the displayed virtual space, and the output is the user's operation data.

[1245] Step 7:

[1246] The generative AI provides commentary about the exhibits in response to user input.

[1247] The server analyzes the user's operation data and generates and sends a prompt sentence (e.g., "Tell me about the Egyptian pharaohs") to the generation AI. The generation AI generates a response based on this prompt and provides an explanation. The input is the user's operation data and the generated prompt sentence, and the output is the explanatory text or audio data.

[1248] Step 8:

[1249] The terminal receives commentary data from the generating AI and provides it to the user.

[1250] Specifically, the device displays text or plays audio to provide an explanation to the user. The input for this step is the explanation data sent from the generation AI, and the output is the display on the user interface or audio playback.

[1251] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1252] The embodiment for implementing the present invention comprises the following elements:

[1253] 1. Server:

[1254] Responsible for generating virtual space.

[1255] Information is obtained from destination and era databases to construct a virtual space.

[1256] By combining it with an emotion engine, it recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI.

[1257] 2. Terminal:

[1258] Displays a virtual space generated by the server.

[1259] It provides an interface that allows the user to perform operations within a virtual space.

[1260] It executes processing related to the display of the virtual space and user operations.

[1261] 3. User:

[1262] Users access and operate the virtual space through a terminal.

[1263] The emotion engine recognizes the user's emotional state and adjusts the behavior of the virtual space and generative AI accordingly.

[1264] 4. Generative AI:

[1265] Explanations on subjects are given in a virtual space.

[1266] The emotion engine recognizes the user's emotional state and provides appropriate information and interactions accordingly.

[1267] Based on the above, the server generates a virtual space, the terminal displays it, and the user can operate it to progress with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted to provide a more personalized learning experience.

[1268] The process flow will be explained below.

[1269] Step 1: The server starts generating the virtual space.

[1270] The server receives destination and time indications.

[1271] The server obtains information on the designated destination from a destination database.

[1272] The server obtains information on the designated era from the era database.

[1273] The server starts the emotion engine and prepares it to recognize the user's emotional state.

[1274] Step 2: The server generates the virtual space.

[1275] The server determines the shape of the virtual space and the placement of objects based on the destination information it obtains.

[1276] The server sets the background and ambient sounds of the virtual space based on the information about the era it has acquired.

[1277] The server creates the data for the generated virtual space.

[1278] The server monitors the user's emotional state using an emotion engine and adjusts the behavior of the virtual space and generative AI.

[1279] Step 3: Your device displays the virtual space

[1280] The terminal receives the virtual space data generated from the server.

[1281] The device uses the received data to display the virtual space as 3D graphics and audio.

[1282] Show.

[1283] Step 4: The user operates in the virtual space

[1284] Users access the virtual space through their terminals.

[1285] Users can move their position and manipulate objects in the virtual space, including by recognizing hand gestures and accepting voice commands.

[1286] The emotion engine recognizes the user's emotional state and infers emotions by analyzing the user's facial and vocal characteristics.

[1287] Step 5: Generative AI provides commentary on the subject

[1288] When a user performs an operation within the virtual space, the generative AI provides an explanation accordingly.

[1289] The generative AI provides descriptions of objects and places in the virtual world, as well as subject-related information.

[1290] The emotion engine recognises the user's emotional state and adjusts the generative AI's dialogue style and expressions accordingly.

[1291] Through the above processing steps, the server generates a virtual space, the terminal displays it, and the user operates it, allowing the user to proceed with learning within the virtual space. By combining it with an emotion engine, the user's emotional state is recognized and the behavior of the virtual space and generative AI is adjusted, providing a more personalized learning experience. Example 2

[1292] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1293] In conventional educational systems using virtual reality technology, it was difficult to provide a personalized learning experience according to the user's emotions. In addition, the dialogue and information provision using generative AI models were fixed and could not be adjusted in real time, which led to a problem that the learning effect was not fully realized.

[1294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1295] In this invention, the server includes a means for acquiring information from a destination database and an era database and constructing a virtual space, a means for combining an emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generating AI, a terminal that the user operates in the virtual space, a generating AI that explains subjects in the virtual space, and a means for operating the generating AI model using prompt sentences that provide dialogue and information based on the user's emotional state. This enables a personalized learning experience according to the user's emotions and state.

[1296] A "server" is a computer device that generates a virtual space using virtual reality technology and builds the virtual space based on information obtained from a database.

[1297] A "destination database" is a database that accumulates information about places and destinations to be referenced within a virtual space.

[1298] A "period database" is a database that accumulates information about specific eras or historical periods that can be referenced within a virtual space.

[1299] A "virtual space" is a three-dimensional space digitally constructed using virtual reality technology.

[1300] The "emotion engine" is an engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI based on that information.

[1301] A "terminal" is a device that allows a user to access and operate a virtual space, such as a smartphone or a personal computer.

[1302] "Generative AI" is an artificial intelligence that provides explanations and dialogue on subjects within a virtual space.

[1303] A "generative AI model" is an artificial intelligence model that generates dialogue and information using prompt sentences based on the user's emotional state.

[1304] A "prompt" is a text input that can be fed into a generative AI model to generate specific dialogue or information.

[1305] The present invention provides a system consisting of the following elements. The server generates a virtual space and adjusts the behavior of the virtual space and the generative AI based on the user's emotional state. The user accesses and operates the virtual space through a terminal. A personalized learning experience is provided by combining the emotion engine and the generative AI model.

[1306] server

[1307] The server runs using cloud services such as Amazon Web Services (AWS®) and Google Cloud Platform (GCP®). The server retrieves information from the destination database and the era database, and generates the virtual space based on that information. The server uses emotion engines such as Affectiva®'s SDK and IBM Watson® to recognize the user's emotional state. This makes it possible to adjust the behavior of the virtual space and the generation AI.

[1308] Terminal

[1309] The terminals include devices such as iPhone (registered trademark), Android (registered trademark) smartphones, Windows (registered trademark) PCs, and Mac (registered trademark). The terminals display the virtual space generated from the server and provide an interface for the user to operate within the virtual space. The terminals use, for example, Apple (registered trademark)'s ARKit (registered trademark) or Google (registered trademark)'s ARCore (registered trademark) to display the virtual space superimposed on the real space. The user operates objects in this environment using a touch screen or mouse.

[1310] User

[1311] Users access and interact with the virtual space through their devices. The emotion engine recognizes the user's emotional state in real time. For example, if a user is losing focus while learning about ancient Egypt, the system will use generative AI models to adjust their interactions and activities.

[1312] Generation AI

[1313] Generative AI (e.g., OpenAI® ChatGPT® or Google® BERT®) provides explanations on academic subjects in a virtual space. The user's emotional state is recognized, and appropriate explanations and dialogue are provided based on that information. Generative AI uses prompts to generate dialogue for the user.

[1314] Usage example

[1315] For example, if a user is learning about ancient Egypt, the server will retrieve information about "ancient Egypt" from the destination database and information about "3000 BC" from the period database. Based on this, the server will generate a virtual space and send it to the terminal.

[1316] The device displays this virtual space and the user begins to operate it. If the user's emotional state is determined to be "low concentration" by the emotion engine, the generative AI provides dialogue using prompt sentences such as the following:

[1317] “What interactions or activities should we suggest if a user is having trouble concentrating while learning about Ancient Egypt?”

[1318] Based on this prompt, the generative AI can provide users with interesting quizzes and stories, which can recapture the user's attention and improve their learning outcomes.

[1319] As described above, the present invention provides a personalized learning experience by having the various elements of the server, terminal, user, emotion engine, and generative AI model work in coordination.

[1320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1321] Step 1:

[1322] The server retrieves information from a destination database and a period database.

[1323] Specifically, the server executes the following queries: "SELECT FROM destination database WHERE destination='Ancient Egypt'" and "SELECT FROM period database WHERE era='3000 BC'".

[1324] Input: Destination database, era database

[1325] Output: Data set for generating virtual space (destination information, time information)

[1326] Step 2:

[1327] A virtual space is generated based on the data acquired by the server.

[1328] Specifically, the server uses the generative AI model to generate a 3D model, which is designed taking into account the user's past behavioral and emotional data.

[1329] Input: Data set (destination information, time information), user's past behavior data, emotion data

[1330] Output: 3D model of the virtual space

[1331] Step 3:

[1332] The server transmits the virtual space to the terminal.

[1333] As a specific operation, the server transmits the generated 3D model data to the terminal using the HTTP protocol.

[1334] Input: 3D model of the virtual space

[1335] Output: 3D model data sent to the device

[1336] Step 4:

[1337] The device displays the virtual space.

[1338] Specifically, the device uses Apple's (registered trademark) ARKit (registered trademark) or Google's (registered trademark) ARCore (registered trademark) to overlay the received 3D model data onto real space.

[1339] Input: 3D model data

[1340] Output: Virtual space displayed on the device

[1341] Step 5:

[1342] The user performs operations within the virtual space.

[1343] Specifically, the user operates objects and views information panels using a touch screen or a mouse.

[1344] Input: Terminal user interface

[1345] Output: User operation data

[1346] Step 6:

[1347] The server uses an emotion engine to recognize the user's emotional state.

[1348] Specifically, the server analyzes the user's facial expressions and voice through the device's camera and microphone to measure their level of concentration and excitement.

[1349] Input: User's facial expression data, voice data

[1350] Output: The user's emotional state.

[1351] Step 7:

[1352] The server uses the generative AI model to generate interactions and activities based on the user's emotional state.

[1353] Specifically, the server inputs the prompt sentence into the generative AI model to get an appropriate response. For example, prompt sentence: "If the user is losing concentration while learning about ancient Egypt, what dialogue or activity should be suggested?"

[1354] Input: prompt text, user's emotional state

[1355] Output: The generated interactions and activities

[1356] Step 8:

[1357] The terminal provides the generated interactions and activities to the user.

[1358] As a specific operation, the terminal displays the generated dialogue content on the screen and outputs it as voice.

[1359] Input: Generated interactions and activities

[1360] Output: The interactions and activities presented to the user

[1361] By following these processing steps, the system of the present invention is able to provide the user with a personalized learning experience. (Application example 2)

[1362] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1363] In conventional systems using virtual space, it was difficult to dynamically provide content according to the user's emotional state, and only uniform information could be provided. As a result, it was difficult to provide learning or purchasing experiences according to the user's interests, and it was not possible to provide personalized services. In addition, there were limited methods for effectively providing detailed information about products and promoting them in virtual space. This made it a challenge to improve user satisfaction.

[1364] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1365] In this invention, the server includes a means for generating a virtual space using virtual reality technology, a means for acquiring information from a destination database and an era database and constructing the virtual space, a terminal for a user to operate in the virtual space, a generation AI for explaining a subject in the virtual space, an emotion engine for recognizing the emotional state of the user and adjusting the behavior of the virtual space and the generation AI according to the emotional state, and a means for acquiring information from a product database and providing details about products in the virtual space. This makes it possible to provide a dynamic and personalized learning experience or purchasing experience according to the emotional state of the user.

[1366] "Virtual reality technology" is a technology that uses computer technology to create a virtual space that is different from the real world, allowing users to immerse themselves in the experience.

[1367] "Virtual space" refers to a digital three-dimensional space created using virtual reality technology.

[1368] A "server" is an information processing device that generates a virtual space using virtual reality technology and handles user operations and data processing.

[1369] A "destination database" is a database that stores information about destinations for constructing a virtual space.

[1370] An "era database" is a database that stores information about a specific era, and is used when constructing a virtual space.

[1371] A "terminal" is an information device that a user uses to access and operate a virtual space.

[1372] "Generative AI" is a program that uses artificial intelligence technology to provide explanations and dialogue on subjects in a virtual space.

[1373] The "emotion engine" is a component that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI according to that emotional state.

[1374] A "product database" is a database that stores detailed information about products displayed in a virtual space.

[1375] "Means for providing product details" is a function for obtaining information from a product database and providing detailed information about the product to the user within the virtual space.

[1376] In order to carry out the invention, a system including the following elements is configured. First, the server generates a virtual space using virtual reality technology and obtains information from a destination database and an era database to construct the virtual space. This virtual space is displayed on a user terminal and can be operated by the user.

[1377] The server is equipped with an emotion engine that can recognize the user's emotional state. The emotion engine uses facial expression and voice analysis technology to grasp the user's real-time emotions and dynamically adjusts the behavior of the virtual space and the generative AI accordingly. It also has a means of retrieving information from a product database and providing details about products in the virtual space.

[1378] Generative AI can provide personalized product explanations and promotions based on the user's emotional state, providing real-time information tailored to the user's interests and preferences, and improving satisfaction within virtual stores.

[1379] The terminal is an information device that allows users to access and operate the virtual space, and is equipped with 3D graphics and audio playback functions. This allows users to experience the virtual space using both their eyes and ears. It also has a dialogue function based on the user's emotional state using an emotion engine. Specifically, when a user selects a product, the system reads the user's emotions from their facial expressions and voice, and generates an optimal product description.

[1380] As a concrete example of this system, consider a user selecting clothes in a virtual store: if the user is interested in a particular dress, the emotion engine will recognize the user's surprised expression, and the generative AI will provide a personalized description such as "This dress is from our latest collection. It's perfect for a special event!", providing a more personalized experience.

[1381] Examples of prompts for generative AI models include the following:

[1382] "Users are surprised to see a dress from your new collection. Can you help me explain it to them in a way that will keep them interested?"

[1383] In terms of hardware, the server needs to be an information processing device equipped with a high-performance processor and sufficient memory, and as user terminals, smartphones, smart glasses, or head-mounted displays that support virtual reality technology are suitable. For software, it is desirable to use libraries and SDKs (software development kits) for generating virtual spaces, and AI frameworks such as TensorFlow (registered trademark) and PyTorch (registered trademark) for implementing artificial intelligence. For emotion recognition, it is recommended to use OpenCV (registered trademark) or the Emotion Recognition library.

[1384] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1385] Step 1:

[1386] The server generates a virtual space using virtual reality technology. It uses information obtained from the destination database and the era database as input, and generates the shape of the virtual space, the placement of objects, and background sounds based on that information. The output is the data of the generated virtual space.

[1387] Step 2:

[1388] The server transmits the generated virtual space data to the user's terminal. The terminal receives it and displays the virtual space to the user using 3D graphics and audio. The input is the virtual space data transmitted from the server, and the output is to allow the user to experience the virtual space visually and aurally.

[1389] Step 3:

[1390] The user performs operations in the virtual space. Specifically, when the user shows interest in a particular object (e.g., a product), the user makes a selection. The input is the user's operation data, and the output is the object to be selected (e.g., the product ID).

[1391] Step 4:

[1392] The server retrieves detailed information about the selected object from the product database. The input is the ID of the object selected by the user, and the output is detailed information about that object.

[1393] Step 5:

[1394] The server uses an emotion engine to recognize the user's emotional state. To do so, the server uses the user's facial expression data and voice data as input. The emotion engine analyzes these data and outputs the user's emotional state (e.g., surprise, joy, etc.).

[1395] Step 6:

[1396] The server sends a prompt to the generation AI based on the emotional state. Specifically, it creates a prompt such as, "The user is surprised to see a dress from the new collection. Please advise how to explain it so that they will remain interested." The input is the user's emotional state and the selected product information, and the output is the prompt sent to the generation AI.

[1397] Step 7:

[1398] The generation AI generates personalized product descriptions and promotional content based on the prompt text. The input is the prompt text sent from the server, and the output is the product description or promotional content provided to the user.

[1399] Step 8:

[1400] The server transmits the product description and promotional content generated by the AI ​​to the user's device. The device receives it and displays it to the user in the virtual space. The input is the product description generated by the AI, and the output is to provide the user with the product description or promotional content visually and audibly.

[1401] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1402] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1403] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1404] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1405] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1406] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1407] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[1408] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1409] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1410] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1411] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[1412] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[1413] In the above embodiment, the specific processing program 56 is stored in the storage 32, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable memory such as a Universal Serial Bus (USB) memory.

[1414] The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1415] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1416] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1417] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1418] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1419] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1420] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1421] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[1422] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[1423] The following is further disclosed regarding the above embodiment.

[1424] (Claim 1) A server for generating a virtual space using virtual reality technology; A means for acquiring information from a destination database and an era database and constructing the virtual space; A terminal for a user to operate within the virtual space; A generation AI that provides explanations about subjects in the virtual space; A system including:

[1425] (Claim 2) As the virtual space generating means, a shape of the space, an arrangement of objects, and background sounds are generated based on the information acquired from the destination database and the era database. The system of claim 1.

[1426] (Claim 3) The terminal includes a means for reproducing 3D graphics and sound to provide display of the virtual space and operation of the user; The system of claim 1.

[1427] (Claim 4) The system of claim 1 , wherein an emotion engine adjusts the behavior of the virtual space and the generating AI according to the user's emotions. "Example 1"

[1428] (Claim 1) A server for generating a virtual space; A means for acquiring information from a destination database and an era database and constructing the virtual space; A terminal for a user to operate within the virtual space; A generative AI that provides explanations about subjects in the virtual space; means for accepting user operations via an interface displayed on the terminal; A means for generative AI to provide content in real time within a virtual space based on user input; and A system including:

[1429] (Claim 2) As the virtual space generating means, a shape of the space, an arrangement of objects, and background sounds are generated based on the information acquired from the destination database and the era database. 2. The system of claim 1.

[1430] (Claim 3) The terminal includes a means for reproducing 3D graphics and sound to provide display of the virtual space and operation of the user; 2. The system of claim 1. "Application example 1"

[1431] (Claim 1) A server for generating a virtual space using virtual reality technology; A means for acquiring information from a destination database and an era database and constructing the virtual space; A terminal for a user to operate within the virtual space; A generation AI that provides explanations about subjects in the virtual space; A means for acquiring information from an exhibition information database and reflecting the exhibition information in a virtual space; A generation AI that provides commentary on the exhibits in the virtual space; A system including:

[1432] (Claim 2) 2. The system according to claim 1, wherein the virtual space generating means generates the shape of the space, the arrangement of objects, and background sounds based on information acquired from the destination database and the era database.

[1433] (Claim 3) The system of claim 1 , wherein the terminal includes means for playing 3D graphics and audio to provide display of the virtual space and manipulation by the user. "Example 2 of combining emotion engines"

[1434] (Claim 1) A server for generating a virtual space using virtual reality technology; A means for acquiring information from a destination database and an era database and constructing the virtual space; A means for combining an emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generating AI; A terminal for a user to operate within the virtual space; A generation AI that provides explanations about subjects in the virtual space; A system including:

[1435] (Claim 2) As the virtual space generating means, a shape of the space, an arrangement of objects, and background sounds are generated based on the information acquired from the destination database and the era database. 2. The system of claim 1.

[1436] (Claim 3) The terminal includes a means for reproducing 3D graphics and sound to provide display of the virtual space and operation of the user; 2. The system of claim 1.

[1437] (Claim 4) means for operating the generative AI model with dialogue and informational prompts based on the user's emotional state; 2. The system of claim 1. "Application example 2 when combining emotion engines"

[1438] (Claim 1) A server for generating a virtual space using virtual reality technology; A means for acquiring information from a destination database and an era database and constructing the virtual space; A terminal for a user to operate within the virtual space; A generation AI that provides explanations about subjects in the virtual space; An emotion engine that recognizes the user's emotional state and adjusts the behavior of the virtual space and the generative AI according to that emotional state; a means for retrieving information from a product database and providing details about the product in a virtual space; A system including:

[1439] (Claim 2) As the means for generating the virtual space, a shape of the space, an arrangement of objects, and background sounds are generated based on information acquired from the destination database and the era database, and further, product explanations and promotions are dynamically provided according to the emotional state of the user. 2. The system of claim 1.

[1440] (Claim 3) The terminal includes a means for playing 3D graphics and audio to display the virtual space and to provide the user with an operation, and further includes an interaction function based on the user's emotional state by the emotion engine. 2. The system of claim 1. [Explanation of symbols]

[1441] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A server for generating a virtual space using virtual reality technology; A means for acquiring information from a destination database and an era database and constructing the virtual space; A terminal for a user to operate within the virtual space; A generative AI that generates an explanation to be provided to the user regarding a subject that the user is studying in the virtual space; means for accepting an operation by the user via a user interface displayed on the terminal; means for providing content including the commentary to the user in real time within the virtual space based on the received operation of the user; A system including:

2. As the virtual space generating means, at least one of a shape of a space, an arrangement of objects, and a background sound is generated based on information acquired from the destination database and the era database. The system of claim 1 .

3. The terminal further includes a means for executing the virtual space by displaying 3D graphics and playing audio based on the data of the virtual space generated by the server. The system of claim 1 .

4. the server comprises an emotion engine that recognizes an emotional state of the user; The system of claim 1 , wherein the emotion engine adjusts the expression of the virtual space and the output content of the generative AI in response to the user's emotion.

Images