system

Abstract

Provide a system. A means for a user to select a virtual history experience scenario; means for receiving a 3D model and audio data from a server based on a scenario selected by the terminal; means for recognizing a user's voice in the virtual space and transmitting the voice to a server; A means for the server to analyze the received user voice data, generate an answer using the generation AI, and transmit the voice data to the terminal; means for playing back the audio data received by the terminal and transmitting it to the user; a means of recording progress and learning in a database upon completion of the virtual history experience; a means by which the server terminates the session; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional history education has mainly been a passive learning method using textbooks and reference books, and text and images alone have not been enough to attract students' interest or make them difficult to understand. Furthermore, because it is difficult to visit actual historical places and periods, students have limited opportunities to directly experience actual history and culture. For this reason, there has been a need for a more interactive way to provide opportunities for students to learn about and experience history and culture in a way that is more suitable for today's students. [Means for solving the problem]

[0005] This invention enables users to learn about history and culture interactively while interacting with historical figures in a virtual space through a system that includes: means for a user to select a virtual historical experience scenario; means for the terminal to receive 3D models and audio data from a server based on the selected scenario; means for recognizing the user's voice in the virtual space and sending it to the server; means for the server to analyze the user's received audio data, generate an answer using a generation AI, and send the audio data to the terminal; means for the terminal to play back the received audio data and communicate it to the user; means for recording the progress and learning content in a database when the virtual historical experience ends; and means for the server to end the session.

[0006] A "user" is a person who uses the virtual history experience system to learn interactively.

[0007] A "terminal" is a device that a user operates and uses to experience a virtual space through communication with a server.

[0008] The "server" is a computer system that manages the entire virtual history experience system, loads the database and voice generation engine, and provides the necessary information to the terminal.

[0009] A "scenario" refers to the content of a historical setting, period, or event that a user selects in a virtual history experience.

[0010] A "3D model" is a computer graphic that recreates historical places and people in a virtual history experience scenario in three dimensions.

[0011] "Audio data" refers to audio information used as dialogue or narration by historical figures.

[0012] "Generative AI" is an artificial intelligence technology that generates appropriate answers to questions and instructions from users.

[0013] A "virtual space" is a digital environment in which users can virtually experience history and culture through 3D models and audio data.

[0014] "Progress" is information that records what the user has learned and the level of achievement in the virtual history experience system.

[0015] A "session" refers to a specific period of time during which a user uses the system, during which all actions and experiences are recorded.

[0016] "Database" refers to a system for storing and managing all data related to the Virtual History Experience System.

[0017] "Speech recognition" is a technology that analyzes a user's voice and converts it into text data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that allows users to interactively learn about history and culture through a virtual historical experience. Specific processing of each step in the embodiment of the invention will be described below.

[0040] User Authentication and Login

[0041] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[0042] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[0043] Select and start your virtual history experience

[0044] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[0045] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[0046] Interactive History Experience

[0047] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, converts it into text, and sends it to the server. The server uses generative AI to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device plays the audio and communicates it to the user.

[0048] For example, if a user asks, "Who built this pyramid?", the device converts the question into text and sends it to the server. The server uses generative AI to generate the answer, "This pyramid was built by Pharaoh Khufu," and sends it along with the audio data to the device. The device then plays back the audio and relays it to the user.

[0049] Ending the experience and recording the results

[0050] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experience details. The server then ends the current session and invalidates the session ID.

[0051] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress in a database and ends the session. This records the user's learning history and can be used for the next experience.

[0052] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture through an interactive virtual historical experience.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The server starts the system, loads the necessary modules and databases, and loads the 3D model data, speech generation engine, and historical data into memory.

[0056] Step 2:

[0057] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[0058] Step 3:

[0059] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[0060] Step 4:

[0061] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[0062] Step 5:

[0063] The terminal saves the session ID and notifies the user that the login was successful. The terminal remembers the session ID received from the server and uses it for subsequent requests.

[0064] Step 6:

[0065] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[0066] Step 7:

[0067] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[0068] Step 8:

[0069] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[0070] Step 9:

[0071] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[0072] Step 10:

[0073] Users interact with historical figures in a virtual space, asking questions and giving instructions through an interface.

[0074] Step 11:

[0075] The device recognizes the user's voice and sends it to the server, which converts it into text, encrypts it, and sends it to the server.

[0076] Step 12:

[0077] The server uses the generation AI to analyze the user's question and generate an appropriate answer, which is then passed to a speech synthesis engine and converted into voice data.

[0078] Step 13:

[0079] The server transmits the generated voice data to the terminal.

[0080] Step 14:

[0081] The terminal plays back the audio and conveys it to the user. The terminal plays back the received audio data and conveys it to the user as if a historical figure were speaking in the virtual space.

[0082] Step 15:

[0083] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[0084] Step 16:

[0085] The device sends a termination notification to the server. The device notifies the server that the user has terminated the experience.

[0086] Step 17:

[0087] The server compiles and stores user session data in a database, recording learning progress, questions, and experience details.

[0088] Step 18:

[0089] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[0090] Through these steps, users can engage in an interactive virtual historical experience and learn more about history and culture.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Conventional educational systems have struggled to provide users with a conversational and interactive learning experience when learning about history and culture. They also lacked sophisticated systems that could respond appropriately to users' questions and provide information in real time. Furthermore, they lacked the ability to record learning progress and experiences and use them for future learning, making it difficult to improve users' learning efficiency.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the received user voice data, generate an answer using a generative artificial intelligence, and transmit the voice data to the terminal; means for the terminal to play back the received voice data and communicate it to the user; means for recording the learning progress and learning content in a database when the virtual historical experience ends; and means for the server to end the session. This allows users to have a high-quality learning experience in real time through a conversational and interactive virtual historical experience, thereby improving learning efficiency.

[0096] "User" refers to an individual who uses this system to experience virtual history.

[0097] "Virtual history experience scenario" refers to content that allows people to learn about specific historical events or cultures that are recreated in a virtual space.

[0098] "Terminal" refers to the device used by a user to access this system, including computers, tablets, smartphones, VR headsets, etc.

[0099] A "three-dimensional model" refers to a digital representation of three-dimensional objects and environments that a user can visually experience in a virtual space.

[0100] "Audio data" refers to data that stores audio information related to a virtual history experience scenario in digital format.

[0101] "Server" refers to a central component of a computer network that processes requests from user terminals and provides the necessary data.

[0102] "Generative AI" refers to AI technology that analyzes users' questions and generates appropriate answers.

[0103] "Answer" refers to the response content generated in response to a user's question analyzed by the generating artificial intelligence.

[0104] "Speech recognition" refers to the technology of converting a user's speech into digital data and understanding its content as text.

[0105] "Database" refers to an information system for systematically storing and managing information such as learning progress and experience.

[0106] A "session" refers to a period of time during which a user performs a series of operations from the time the user starts using the system until the time the user finishes using the system.

[0107] MODE FOR CARRYING OUT THE INVENTION

[0108] This invention is a system that allows users to learn about history and culture in an interactive manner through a virtual historical experience. The program of this system consists of a series of processes, such as user authentication and login, selection and start of the virtual historical experience, interactive historical experience, end of the experience, and recording of the results.

[0109] User Authentication and Login

[0110] The user enters a username and password from the terminal. The terminal encrypts this information using the AES encryption method and sends it to the server. The server checks the received information against a database (e.g., MySQL (registered trademark)) to confirm whether the user exists. If authentication is successful, the server generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[0111] Examples:

[0112] The user enters the username "example_user" and password "example_pass" and presses the "Login" button on the terminal. The terminal encrypts this with AES and sends the "encrypted data" to the server. The server decrypts the received data and checks it against the MySQL database. If authentication is successful, the server generates a session ID "example_session_id" and sends it to the terminal. The terminal records the session ID and displays "Login successful" to the user.

[0113] Select and start your virtual history experience

[0114] The user checks a list of virtual history experience scenarios provided on the device. When the user selects a scenario of interest, the device sends the selection information to the server. The server collects a 3D model (e.g., Unity engine), audio data (e.g., MP3 file), and related text information corresponding to the selected scenario and sends them to the device. The device uses the received data to render the virtual space.

[0115] Examples:

[0116] The user opens the "Scenario Selection" screen on their device and selects "Ancient Egyptian Pyramids." When they press the "Select" button, the device sends "Scenario ID: egypt_pyramid" to the server. The server uses the Unity 3D engine to collect a three-dimensional model of the pyramid, related MP3 audio data, and other necessary information, and sends it to the device. The device receives this and renders it through a VR headset, allowing the user to explore the virtual pyramid.

[0117] Interactive History Experience

[0118] In the virtual space, the user interacts with historical figures. The user uses a microphone to ask questions or give instructions. The device converts the audio input from the microphone into text using speech recognition software (e.g., Google (registered trademark) Cloud Speech-to-Text) and sends it to the server. The server uses generative AI (e.g., OpenAI (registered trademark) GPT-3 (registered trademark)) to analyze the user's question and generate an appropriate answer. The answer is converted into audio data using speech synthesis software (e.g., Amazon Polly) and sent to the device. The device plays the audio data and communicates it to the user.

[0119] Examples:

[0120] The user asks in the virtual space, "Who built this pyramid?" The device converts the speech to text using Google Cloud Speech-to-Text and sends the text "Who built this pyramid?" to the server. The server analyzes the question using OpenAI GPT-3 and generates the answer "This pyramid was built by Pharaoh Khufu." The answer is then converted into audio data using Amazon Polly and sent to the device. The device plays the audio data and communicates it to the user.

[0121] Ending the experience and recording the results

[0122] When the user finishes the experience, the device notifies the server. The server saves the user's session data (learning progress, questions, experience details, etc.) in a database. After saving, the server ends the current session and invalidates the session ID.

[0123] Examples:

[0124] When the user presses the "End" button on the device, the device sends "User ID: example_user, Session ID: example_session_id, Status: Ended" to the server. The server compiles the session data (learning progress, question content, experience details) and saves it in a MySQL database. Once saved, the server ends the session and invalidates the session ID. The user is then told "Experience has ended."

[0125] Prompt Sentence Examples

[0126] Below is a concrete example of a prompt sentence to be input to the generative AI model. The purpose is to generate answers to questions posed by users during a virtual history experience.

[0127] Prompt Sentence Examples

[0128] User Question: "Who built this pyramid?"

[0129] Context: "A user asked this question while exploring an ancient Egyptian pyramid. This pyramid was built by Pharaoh Khufu."

[0130] Expected answer: "This pyramid was built by Khufu."

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] User Authentication and Login

[0133] Step 1:

[0134] Specific behavior:

[0135] The user enters the user name and password from the terminal.

[0136] input:

[0137] Username and password.

[0138] process:

[0139] The terminal encrypts the input information using the AES encryption method.

[0140] output:

[0141] Encrypted authentication data.

[0142] Step 2:

[0143] Specific behavior:

[0144] The terminal transmits the encrypted authentication data to the server.

[0145] input:

[0146] Encrypted authentication data.

[0147] process:

[0148] Send data from the terminal to the server.

[0149] output:

[0150] The authentication data received by the server.

[0151] Step 3:

[0152] Specific behavior:

[0153] The server checks the received authentication data against a database.

[0154] input:

[0155] Authentication data.

[0156] process:

[0157] The server checks if the user exists by checking against a database (e.g. MySQL).

[0158] output:

[0159] The authentication result (success or failure).

[0160] Step 4:

[0161] Specific behavior:

[0162] If authentication is successful, the server generates a new session ID.

[0163] input:

[0164] Authentication result (success).

[0165] process:

[0166] The server generates a session ID.

[0167] output:

[0168] The generated session ID.

[0169] Step 5:

[0170] Specific behavior:

[0171] The server sends the generated session ID to the terminal.

[0172] input:

[0173] Session ID.

[0174] process:

[0175] The server sends a session ID to the terminal.

[0176] output:

[0177] The session ID received by the device.

[0178] Step 6:

[0179] Specific behavior:

[0180] The terminal saves the session ID and notifies the user that the login was successful.

[0181] input:

[0182] Session ID.

[0183] process:

[0184] The terminal will save the session ID and display a successful login message on the screen.

[0185] output:

[0186] Stored session ID and successful login notification.

[0187] Select and start your virtual history experience

[0188] Step 1:

[0189] Specific behavior:

[0190] The user checks the list of virtual history experience scenarios provided on the terminal.

[0191] input:

[0192] View the user's scenario list.

[0193] process:

[0194] The device displays a list of scenarios.

[0195] output:

[0196] Scenario list.

[0197] Step 2:

[0198] Specific behavior:

[0199] The user selects the scenario that interests them.

[0200] input:

[0201] Select a scenario.

[0202] process:

[0203] The terminal stores the selection information.

[0204] output:

[0205] Selected scenario information.

[0206] Step 3:

[0207] Specific behavior:

[0208] The terminal transmits the selection information to the server.

[0209] input:

[0210] Selected scenario information.

[0211] process:

[0212] Send data from the terminal to the server.

[0213] output:

[0214] Scenario information received on the server.

[0215] Step 4:

[0216] Specific behavior:

[0217] The server collects the three-dimensional model, audio data, and associated text information corresponding to the selected scenario.

[0218] input:

[0219] Selected scenario information.

[0220] process:

[0221] The server collects 3D models (e.g., Unity engine), audio data (e.g., MP3 files), and text information.

[0222] output:

[0223] Data collected.

[0224] Step 5:

[0225] Specific behavior:

[0226] The server transmits the collected data to the terminal.

[0227] input:

[0228] Data collected.

[0229] process:

[0230] Data is sent from the server to the terminal.

[0231] output:

[0232] Three-dimensional models, audio data, and text information arrive at the terminal.

[0233] Step 6:

[0234] Specific behavior:

[0235] The device uses the received data to render the virtual space.

[0236] input:

[0237] Three-dimensional models, audio data, and text information.

[0238] process:

[0239] The device renders the virtual space based on the received data.

[0240] output:

[0241] Rendered virtual space.

[0242] Interactive History Experience

[0243] Step 1:

[0244] Specific behavior:

[0245] Users can ask questions and give instructions through a microphone in the virtual space.

[0246] input:

[0247] User voice.

[0248] process:

[0249] The device records the audio.

[0250] output:

[0251] Recorded audio data.

[0252] Step 2:

[0253] Specific behavior:

[0254] The device converts the voice into text using speech recognition software.

[0255] input:

[0256] Audio data.

[0257] process:

[0258] The device converts speech to text using Google Cloud Speech-to-Text.

[0259] output:

[0260] Text data.

[0261] Step 3:

[0262] Specific behavior:

[0263] The terminal transmits the converted text data to the server.

[0264] input:

[0265] Text data.

[0266] process:

[0267] Send data from the terminal to the server.

[0268] output:

[0269] Text data received by the server.

[0270] Step 4:

[0271] Specific behavior:

[0272] The server uses generative AI to analyze the text data and generate appropriate answers.

[0273] input:

[0274] Text data.

[0275] process:

[0276] The server uses generative AI (e.g., OpenAI GPT-3) to analyze the question and generate an answer.

[0277] output:

[0278] The generated answer text.

[0279] Step 5:

[0280] Specific behavior:

[0281] The server converts the generated response into voice data using voice synthesis software.

[0282] input:

[0283] Answer text.

[0284] process:

[0285] The server uses Amazon Polly to convert the response into audio data.

[0286] output:

[0287] Audio data.

[0288] Step 6:

[0289] Specific behavior:

[0290] The server transmits the generated voice data to the terminal.

[0291] input:

[0292] Audio data.

[0293] process:

[0294] Data is sent from the server to the terminal.

[0295] output:

[0296] Audio data received at the device.

[0297] Step 7:

[0298] Specific behavior:

[0299] The terminal reproduces the audio data and conveys it to the user.

[0300] input:

[0301] Audio data.

[0302] process:

[0303] The device plays the audio data.

[0304] output:

[0305] A spoken response is conveyed to the user.

[0306] Ending the experience and recording the results

[0307] Step 1:

[0308] Specific behavior:

[0309] The user selects to end the experience.

[0310] input:

[0311] Exit selection.

[0312] process:

[0313] The terminal generates an exit command.

[0314] output:

[0315] Exit command.

[0316] Step 2:

[0317] Specific behavior:

[0318] The device sends a notification of the end of the experience to the server.

[0319] input:

[0320] Exit command.

[0321] process:

[0322] Send commands from the terminal to the server.

[0323] output:

[0324] The termination command received by the server.

[0325] Step 3:

[0326] Specific behavior:

[0327] The server collects user session data (learning progress, questions, experience details, etc.) and stores them in a database.

[0328] input:

[0329] Exit command.

[0330] process:

[0331] The server stores the session data in a database (e.g. MySQL).

[0332] output:

[0333] Saved session data.

[0334] Step 4:

[0335] Specific behavior:

[0336] The server ends the current session and invalidates the session ID.

[0337] input:

[0338] Termination notice.

[0339] process:

[0340] The server ends the session and invalidates the session ID.

[0341] output:

[0342] Invalidated session ID and session end notification.

[0343] (Application example 1)

[0344] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0345] Conventional virtual history experience systems are often experienced by individuals in limited spaces such as their own rooms, and are insufficient as a place to share experiences with other visitors. Furthermore, it is difficult to provide an interactive experience that requires users to physically move around, limiting the effectiveness of learning about history and culture. This has reduced opportunities for visitors to learn while enjoying deeper experiences in brick-and-mortar locations such as museums and theme parks.

[0346] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0347] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and audio data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the user's received audio data, generate an answer using a generation AI, and transmit the audio data to the terminal; means for the terminal to play back the received audio data and communicate it to the user; means for recording the progress and learning content in a database when the virtual historical experience ends; means for the server to end the session; means for providing the physical store with a video display device and audio playback device for the user to experience visually and audibly; and means for the user to experience the virtual historical experience while physically moving around the store. This enables visitors to participate in a virtual historical experience while interactively moving around the physical store.

[0348] "User authentication" is the process by which a user provides individual identifying information, that information is verified, and access rights are granted.

[0349] A "virtual history experience scenario" is a scenario that recreates a specific historical period or scene, allowing users to experience and interact with that period within a virtual environment.

[0350] A "terminal" is a device that a user uses to experience virtual history, and includes smartphones, smart glasses, head-mounted displays, etc.

[0351] A "3D model" is a digital model of an object represented in three-dimensional space using computer graphics.

[0352] "Audio data" refers to data in which audio is recorded in digital format, and is used for audio communication with a user and for reproducing audio information.

[0353] A "server" is a remote computer system that processes requests from user terminals and provides the required data or services.

[0354] "Generative AI" is an artificial intelligence system that performs natural language processing in response to user questions and instructions, generating appropriate answers and information.

[0355] "Progress" is data that records what the user has learned and the goals they have achieved through the virtual history experience.

[0356] A "database" is a system that can efficiently and safely manage large amounts of data and quickly search, update, and delete it.

[0357] A "session" refers to a series of interactions between a user's login and logout, and is a process in which the operations and data exchanges that occur during that time are tracked.

[0358] An "image display device" is a device that allows a user to visually check information or images, and includes smart glasses and head-mounted displays.

[0359] An "audio playback device" is a device for playing back audio data as sound, and includes speakers and headsets.

[0360] A "physical store" is a store or facility where commerce or services are conducted at a physical location.

[0361] "Physically moving around the store" means that the user actually uses their own body to walk around the store.

[0362] MODE FOR CARRYING OUT THE INVENTION

[0363] To implement this invention, it is necessary to build a system that combines specific hardware and software to provide a virtual history experience system in a physical store. The main components of this invention are a server, a terminal, and a user. How each component operates and how the overall system functions are described below.

[0364] Hardware and Software Configuration

[0365] Hardware:

[0366] 1. Devices (head-mounted displays, smart glasses, etc.)

[0367] This is a display device that allows users to visually experience virtual history.

[0368] 2. Audio playback devices (speakers, headsets)

[0369] This is a device that plays back audio data sent from the server and transmits it to the user.

[0370] 3. Microphone

[0371] This is a device that recognizes the user's voice and transmits it to the server.

[0372] 4. Network Connectivity Devices

[0373] It is used to connect terminals and servers over a network.

[0374] software:

[0375] 1. Server

[0376] Data management software: Records user progress and data.

[0377] Generative AI model: Generative AI is used to generate appropriate answers to user questions, for example, OpenAI's GPT-3.

[0378] Database system: Manages users' learning progress and questions.

[0379] 2. Terminal

[0380] Speech recognition software: Converts user speech into text using software such as Google Speech-to-Text API or AWS® Transcribe.

[0381] Rendering engine: Rendering 3D models and virtual spaces using Unity, Unreal Engine, etc.

[0382] Interface software: manages the user-system interface.

[0383] Specific example of system operation

[0384] 1. User authentication and login

[0385] The user enters login information (e.g., username and password) from the terminal. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, a session ID is generated and sent to the terminal.

[0386] 2. Select and start your virtual history experience

[0387] The user selects a virtual historical experience scenario (e.g., "Japan in the Edo Period") from their device. The server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space, allowing the user to experience the historical setting as if they were actually there.

[0388] 3. Interactive History Experience

[0389] Within the virtual space, users can interact with historical figures. The user's voice is recognized through a microphone, converted into text, and sent to the server. The server uses generative AI to generate appropriate answers to the user's questions, converts the answers into audio data, and sends it to the device. The audio data is played back on the device and conveyed to the user.

[0390] 4. Completing the experience and recording the results

[0391] When the user finishes the experience, the device sends a completion notification to the server. The server records the user's progress and learning content in a database and ends the session. This saves the user's learning history and can be used for the next experience.

[0392] Specific prompt examples

[0393] In response to a user question, "What products do you have available today?", the generative AI model is given the following prompt:

[0394] Suppose a user asks a merchant in the Edo period about today's products. User: "What products do you have available today?" The merchant will respond based on the Edo period setting. Merchant: "Today's recommendations are fresh fish and freshly baked dumplings."

[0395] In this way, the brick-and-mortar virtual history experience system can provide users with an interactive and immersive learning experience.

[0396] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0397] Step 1:

[0398] The user enters login information (username and password) using the terminal. The terminal encrypts this information and sends it to the server. The server compares the entered authentication information with the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the received session ID, and the user is notified that login was successful.

[0399] Input: Username, Password

[0400] Data processing: Username and password encryption

[0401] Data calculation: Authentication verification

[0402] Output: Session ID

[0403] Specific actions: Entering data into the login form, encrypting data, generating a token (session ID)

[0404] Step 2:

[0405] The user displays a list of virtual historical experience scenarios provided by the terminal and selects one that interests them. The terminal then sends the selected scenario information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the terminal. The terminal then receives this data and renders the virtual space.

[0406] Input: User selects a scenario

[0407] Data processing: Collecting data corresponding to the scenario

[0408] Data calculation: Sending 3D models and audio data

[0409] Output: Virtual space rendering

[0410] Specific operations: scenario selection, data collection and transmission, construction of a virtual environment using a rendering engine

[0411] Step 3:

[0412] Within the virtual space, users interact with historical figures through voice. The user's voice is picked up by the device's microphone and converted into text using speech recognition software (e.g., Google Speech-to-Text API). The converted text is sent to a server. The server uses generative AI (e.g., OpenAI's GPT-3) to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device then plays the audio data to the user through a playback device.

[0413] Input: User's voice

[0414] Data processing: Speech-to-text (voice recognition)

[0415] Data calculation: Question analysis and answer generation (generative AI)

[0416] Output: Audio data

[0417] Specific operations: Acquiring voice using a microphone, converting it to text, generating answers using AI, and playing back voice data

[0418] Step 4:

[0419] When the user finishes the virtual history experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experiences. The server then ends the current session and invalidates the session ID.

[0420] Input: User's termination notice

[0421] Data processing: Session data aggregation

[0422] Data calculation: saving to database and ending session

[0423] Output: Saved learning progress data

[0424] Specific actions: Sending a completion notice, recording progress in the database, and terminating the session

[0425] In this way, the system of the present invention can provide users with an interactive and immersive virtual history experience.

[0426] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0427] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[0428] User Authentication and Login

[0429] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[0430] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[0431] Select and start your virtual history experience

[0432] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[0433] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[0434] Interactive History Experience and Emotion Recognition

[0435] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[0436] As a concrete example, when a user asks, "Who built this pyramid?", the device converts the question into text and analyzes the emotion using an emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Amazing, isn't it?" The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[0437] Ending the experience and recording the results

[0438] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[0439] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server then saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[0440] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture while taking their emotions into consideration through an interactive virtual historical experience.

[0441] The processing flow will be explained below.

[0442] Step 1:

[0443] The server starts the system. The server loads the necessary modules and databases, and loads the 3D model data, speech generation engine, emotion engine, and historical data into memory.

[0444] Step 2:

[0445] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[0446] Step 3:

[0447] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[0448] Step 4:

[0449] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[0450] Step 5:

[0451] The terminal saves the session ID and notifies the user that they have successfully logged in. The terminal records the session ID received from the server and uses it for subsequent requests.

[0452] Step 6:

[0453] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[0454] Step 7:

[0455] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[0456] Step 8:

[0457] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[0458] Step 9:

[0459] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[0460] Step 10:

[0461] Users interact with historical figures in a virtual space, asking questions and giving instructions by voice.

[0462] Step 11:

[0463] The device recognizes the user's voice and analyzes their emotions using an emotion engine. The device then converts the voice data into text and sends it to the server along with the emotion data.

[0464] Step 12:

[0465] The server uses generation AI to analyze the user's question and generates the optimal answer based on the user's question content and emotional data.

[0466] Step 13:

[0467] The server converts the generated response into voice data using a speech synthesis engine and sends it to the device. The server then reflects emotional data in the response data before sending it.

[0468] Step 14:

[0469] The device then plays back the received voice data and conveys it to the user. The device then plays back the voice, conveying it as if the historical figure were responding in a virtual space according to the user's emotions.

[0470] Step 15:

[0471] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[0472] Step 16:

[0473] The device sends a completion notification to the server. The device notifies the server that the user has completed the experience.

[0474] Step 17:

[0475] The server compiles and stores the user's session data in a database, including learning progress, questions, answers, and emotional data.

[0476] Step 18:

[0477] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[0478] Through these steps, users can engage in an interactive virtual history experience, learning about history and culture while receiving emotionally sensitive responses.

[0479] Example 2

[0480] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0481] Conventional virtual history experience systems often lack appropriate responses and interactions that take into account the user's emotional changes when learning about history and culture. This can reduce the user's learning effectiveness. Furthermore, when the user finishes the experience, the records are incomplete, which means that the experience cannot be fully utilized for the next experience.

[0482] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0483] In this invention, the server includes: means for the user to select a virtual historical experience scenario; means for the terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space, analyzing their emotions, and transmitting the results to the server; means for the server to analyze the received user's voice data and emotion data, generate answers using a generation AI, and transmit the voice data to the terminal; means for recording the progress, learning content, and emotion data in a database when the virtual historical experience ends; and means for the server to end the session and invalidate the session ID. This enables an interactive historical experience that takes the user's emotions into consideration.

[0484] "User" refers to an individual who utilizes the virtual history experience provided by the invention.

[0485] "Virtual history experience scenario" refers to a series of scenarios that allow users to learn about history and culture in a virtual space.

[0486] "Terminal" refers to electronic devices such as computers, tablets, and smartphones that are operated by users.

[0487] "Server" refers to the central computer system that receives, transmits, analyzes, and stores data.

[0488] "Three-dimensional model" refers to a three-dimensional shape or object displayed in a virtual space.

[0489] "Voice data" refers to data used to provide interaction and explanations to the user by voice.

[0490] "Virtual space" refers to a three-dimensional interactive space generated by a computer.

[0491] "Emotion" refers to the psychological state or reaction a user exhibits during a virtual experience.

[0492] "Analyzing emotions" refers to analyzing the user's psychological state and reactions from voice data, etc.

[0493] A "database" refers to a digital information storage system that systematically stores collected information and makes it available for retrieval as needed.

[0494] A "session ID" refers to a unique identifier used to identify a series of activities from when a user logs in to when they log out of a system.

[0495] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers to users' questions.

[0496] "Generating an answer" refers to the generative AI creating an answer based on the user's question.

[0497] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[0498] User Authentication and Login

[0499] First, the user enters login information (username and password) from the terminal. The terminal encrypts this information and sends it to the server. The server compares the authentication information with a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID, and the user is notified that login was successful. As a concrete example, when a user enters a "username" and "password," the terminal sends this to the server, and the server compares it with the database for authentication. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[0500] Select and start your virtual history experience

[0501] The user reviews a list of virtual history experience scenarios provided by the device and selects one that interests them. The device sends the selection information to the server. The server collects 3D models, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space. As a specific example, when a user selects "Ancient Egyptian Pyramids," the server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space in which the user can explore the inside of a pyramid through a virtual reality headset or tablet.

[0502] Interactive History Experience and Emotion Recognition

[0503] In a virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then analyzes the user's question using a generative AI model and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user. For example, if a user asks, "Who built this pyramid?" the device converts the question into text and analyzes the emotion using the emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Isn't that amazing?" The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user.

[0504] Ending the experience and recording the results

[0505] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, details of the experience, and emotional data. The server ends the current session and invalidates the session ID. For example, when the user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[0506] Examples of prompt statements

[0507] Example prompt for generative AI model:

[0508] Prompt: "User asks: Who built this pyramid? Emotion: Surprise"

[0509] This system allows users to learn more about history and culture through an interactive virtual historical experience, taking their emotions into account.

[0510] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0511] Step 1:

[0512] The user enters login information (user name and password) from the terminal.

[0513] Input: Username, Password

[0514] Output: Encrypted login information

[0515] Specific behavior:

[0516] The user enters their username and password into the terminal's login screen, which encrypts the information and sends it to the next processing step.

[0517] Step 2:

[0518] The device sends the encrypted login information to the server.

[0519] Input: Encrypted login information

[0520] Output: Sending login information to the server

[0521] Specific behavior:

[0522] The device sends encrypted login information to the server using the HTTPS protocol.

[0523] Step 3:

[0524] The server decrypts the received login information and checks it against the credentials in its database.

[0525] Input: Encrypted login information

[0526] Output: Authentication success or failure

[0527] Specific behavior:

[0528] The server decrypts the encrypted login information, checks it against the credentials stored in the database, and if successful, generates a new session ID.

[0529] Step 4:

[0530] The server sends the generated session ID to the terminal.

[0531] Input: Authentication success result, session ID

[0532] Output: Sending the session ID to the terminal

[0533] Specific behavior:

[0534] The server sends the successful authentication result and the newly generated session ID to the terminal.

[0535] Step 5:

[0536] The terminal saves the received session ID and notifies the user that login was successful.

[0537] Input: Session ID

[0538] Output: Successful login notification

[0539] Specific behavior:

[0540] The terminal stores the received session ID and displays a login success message to the user.

[0541] Step 6:

[0542] The user selects a virtual history experience scenario.

[0543] Input: Select a virtual history experience scenario

[0544] Output: Information about the selected scenario

[0545] Specific behavior:

[0546] The user selects a scenario of interest from a list of multiple virtual history experience scenarios.

[0547] Step 7:

[0548] The terminal transmits the selected scenario information to the server.

[0549] Input: Scenario selection information

[0550] Output: Sending the selected information to the server

[0551] Specific behavior:

[0552] The terminal transmits the scenario information selected by the user to the server.

[0553] Step 8:

[0554] The server collects the three-dimensional model, audio data, and related information corresponding to the selected scenario from a database.

[0555] Input: Scenario selection information

[0556] Output: 3D model, audio data, related information

[0557] Specific behavior:

[0558] The server accesses a database to collect three-dimensional models, audio data, and related information corresponding to the selected scenario.

[0559] Step 9:

[0560] The server sends the collected data to the terminal.

[0561] Input: 3D model, audio data, related information

[0562] Output: Sending data to a terminal

[0563] Specific behavior:

[0564] The server transmits the collected data to the terminal.

[0565] Step 10:

[0566] The terminal uses the received data to render the virtual space and displays it to the user.

[0567] Input: 3D model, audio data, related information

[0568] Output: Rendering of the virtual space

[0569] Specific behavior:

[0570] The device uses the Unity engine to render the received 3D models and audio data, displaying the virtual space to the user.

[0571] Step 11:

[0572] Users interact with historical figures in a virtual space.

[0573] Input: Voice input (questions and instructions)

[0574] Output: Audio data

[0575] Specific behavior:

[0576] Users can ask questions and give instructions to historical figures by voice using a microphone in the virtual space.

[0577] Step 12:

[0578] The device recognizes the voice data, analyzes the emotion using an emotion engine, and sends it to the server.

[0579] Input: Audio data

[0580] Output: Text data, emotion data

[0581] Specific behavior:

[0582] The device converts the voice data into text using a speech recognition engine (e.g., Google Speech-to-Text API), analyzes the emotion using an emotion engine, and then sends the text data and emotion data to the server.

[0583] Step 13:

[0584] The server uses the received text data and emotion data to generate an answer using a generative AI model.

[0585] Input: Text data, emotion data

[0586] Output: Answer text

[0587] Specific behavior:

[0588] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate answer based on the received text data and emotion data.

[0589] Step 14:

[0590] The server converts the generated response into voice data and transmits it to the terminal.

[0591] Input: Answer text

[0592] Output: Audio data

[0593] Specific behavior:

[0594] The server uses a service (e.g., Amazon Polly) to convert the response text into voice data and sends the generated voice data to the device.

[0595] Step 15:

[0596] The terminal plays back the received audio data and conveys it to the user.

[0597] Input: Audio data

[0598] Output: Audio playback

[0599] Specific behavior:

[0600] The terminal plays back the received voice data and conveys the answer to the user.

[0601] Step 16:

[0602] When the user finishes the experience, the device sends a completion notification to the server.

[0603] Input: Termination notification

[0604] Output: Sending notification of completion to the server

[0605] Specific behavior:

[0606] The user performs an end operation (e.g., clicking the "End" button), and the terminal sends an end notification to the server.

[0607] Step 17:

[0608] The server saves the user's session data to a database and ends the session.

[0609] Input: Termination notification, session data

[0610] Output: Save data, end session

[0611] Specific behavior:

[0612] The server stores the user's learning progress, questions, experience details, and emotional data in a database, invalidates the session ID, and ends the session.

[0613] (Application example 2)

[0614] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0615] Conventional virtual history experience systems have struggled to provide an interactive experience that reflects the user's emotions. Furthermore, due to a lack of efficient means for recording and utilizing the user's learning progress and emotions, it has been difficult to provide an optimal learning environment for each individual user. This has led to the issue of users being unable to learn about history and culture in depth.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0617] In this invention, the server includes: a means for the user to select a virtual history experience scenario; a means for the terminal to receive 3D models and voice data from the server based on the selected scenario; a means for recognizing the user's voice in the virtual space and transmitting it to the server; a means for using a generation AI to generate answers to the user's questions and creating the voice data; a means for recognizing the user's emotions in real time and generating dialogue content using the emotion data; a means for recording the progress and learning content in a database when the virtual history experience ends; and a means for the server to end the session. This enables a more natural interactive virtual history experience that reflects the user's emotions.

[0618] A "user" is an entity that uses the system to experience virtual history.

[0619] A "virtual history experience scenario" is a historical or cultural scenario that a user can select and experience.

[0620] A "terminal" is a device used by a user for inputting and outputting information.

[0621] "3D model" refers to a three-dimensional object or environment displayed in a virtual space.

[0622] "Audio data" is digitized audio data used for recording, playing, transmitting, etc.

[0623] A "server" is a computer system that provides services to terminals over a network.

[0624] "Virtual space" refers to a three-dimensional digital space generated by a computer.

[0625] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate output data or content based on user input data.

[0626] A "session" refers to the period of time during which a user performs a series of operations or activities within a system.

[0627] "Emotion data" refers to data that analyzes and indicates the user's emotional state.

[0628] "Progress" is information that indicates how much learning and experience the user has achieved during the virtual history experience.

[0629] A "database" is a structured collection of data that a system manages and makes available.

[0630] "Dialogue content" refers to the content of questions and responses exchanged between the user and the system.

[0631] "Interactive" refers to the characteristic that operations and experiences progress as the user and the system influence each other.

[0632] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[0633] Hardware and Software Configuration

[0634] Hardware

[0635] Device: Includes the smartphone, head-mounted display (HMD), microphone, and camera used by the user.

[0636] Server: A computer system that provides services to terminals over a network.

[0637] software

[0638] Django: a web application framework

[0639] OpenCV: A library for image processing

[0640] Google Cloud Speech-to-Text API: Speech Recognition

[0641] NVIDIA TENSORFLOW®: A Deep Learning Model for Emotion Recognition

[0642] Unity: 3D content rendering engine

[0643] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0644] User Authentication and Login

[0645] The user enters login information on the terminal. The terminal encrypts this information and sends it to the server. The server checks the authentication information against a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the session ID, and the user is notified that login was successful.

[0646] Select and start your virtual history experience

[0647] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[0648] Interactive History Experience and Emotion Recognition

[0649] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[0650] Ending the experience and recording the results

[0651] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[0652] (Example of a specific prompt)

[0653] As an example of a prompt sentence, the following can be entered:

[0654] Emotional state: Surprise, User: Who built this pyramid?

[0655] This allows the system to generate an appropriate response that takes into account the user's emotion when the user expresses surprise.

[0656] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0657] Step 1:

[0658] The user enters login information from the terminal. The entered username and password are encrypted and sent to the server. The server checks the authentication information against the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[0659] Input: Username, Password

[0660] Output: Session ID, login success notification

[0661] Step 2:

[0662] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The selection information is sent to the server. The server collects 3D models and audio data corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space.

[0663] Input: Scenario selection information

[0664] Output: 3D models, audio data, rendered virtual worlds

[0665] Step 3:

[0666] Users interact with historical figures in a virtual space. The user's voice is captured by the device's microphone, converted into text, and the emotion engine analyzes the emotions. The analysis results are sent to the server.

[0667] Input: User voice

[0668] Output: Text data, emotion data

[0669] Step 4:

[0670] The server analyzes the received text data and emotional data and generates an appropriate response using a generative AI model. It generates a prompt sentence that takes the emotional data into account and creates a response based on that. The generated response is converted into voice data and sent to the device.

[0671] Input: Text data, emotion data

[0672] Output: Audio data, prompt

[0673] Step 5:

[0674] The device then plays back the received audio data and transmits it to the user, who continues the virtual history experience through voice responses.

[0675] Input: Audio data

[0676] Output: Audio playback

[0677] Step 6:

[0678] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores the learning progress, questions, experience details, and emotional data in a database. The server then ends the current session and invalidates the session ID.

[0679] Input: Termination notification

[0680] Output: Database record, session termination confirmation

[0681] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0682] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0683] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0684] [Second embodiment]

[0685] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0686] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0687] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0688] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0689] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0690] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0691] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0692] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0693] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0694] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0695] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0696] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0697] This invention is a system that allows users to interactively learn about history and culture through a virtual historical experience. Specific processing of each step in the embodiment of the invention will be described below.

[0698] User Authentication and Login

[0699] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[0700] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[0701] Select and start your virtual history experience

[0702] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[0703] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[0704] Interactive History Experience

[0705] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, converts it into text, and sends it to the server. The server uses generative AI to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device plays the audio and communicates it to the user.

[0706] For example, if a user asks, "Who built this pyramid?", the device converts the question into text and sends it to the server. The server uses generative AI to generate the answer, "This pyramid was built by Pharaoh Khufu," and sends it along with the audio data to the device. The device then plays back the audio and relays it to the user.

[0707] Ending the experience and recording the results

[0708] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experience details. The server then ends the current session and invalidates the session ID.

[0709] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress in a database and ends the session. This records the user's learning history and can be used for the next experience.

[0710] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture through an interactive virtual historical experience.

[0711] The processing flow will be explained below.

[0712] Step 1:

[0713] The server starts the system, loads the necessary modules and databases, and loads the 3D model data, speech generation engine, and historical data into memory.

[0714] Step 2:

[0715] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[0716] Step 3:

[0717] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[0718] Step 4:

[0719] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[0720] Step 5:

[0721] The terminal saves the session ID and notifies the user that the login was successful. The terminal remembers the session ID received from the server and uses it for subsequent requests.

[0722] Step 6:

[0723] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[0724] Step 7:

[0725] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[0726] Step 8:

[0727] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[0728] Step 9:

[0729] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[0730] Step 10:

[0731] Users interact with historical figures in a virtual space, asking questions and giving instructions through an interface.

[0732] Step 11:

[0733] The device recognizes the user's voice and sends it to the server, which converts it into text, encrypts it, and sends it to the server.

[0734] Step 12:

[0735] The server uses the generation AI to analyze the user's question and generate an appropriate answer, which is then passed to a speech synthesis engine and converted into voice data.

[0736] Step 13:

[0737] The server transmits the generated voice data to the terminal.

[0738] Step 14:

[0739] The terminal plays back the audio and conveys it to the user. The terminal plays back the received audio data and conveys it to the user as if a historical figure were speaking in the virtual space.

[0740] Step 15:

[0741] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[0742] Step 16:

[0743] The device sends a termination notification to the server. The device notifies the server that the user has terminated the experience.

[0744] Step 17:

[0745] The server compiles and stores user session data in a database, recording learning progress, questions, and experience details.

[0746] Step 18:

[0747] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[0748] Through these steps, users can engage in an interactive virtual historical experience and learn more about history and culture.

[0749] Example 1

[0750] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0751] Conventional educational systems have struggled to provide users with a conversational and interactive learning experience when learning about history and culture. They also lacked sophisticated systems that could respond appropriately to users' questions and provide information in real time. Furthermore, they lacked the ability to record learning progress and experiences and use them for future learning, making it difficult to improve users' learning efficiency.

[0752] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0753] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the received user voice data, generate an answer using a generative artificial intelligence, and transmit the voice data to the terminal; means for the terminal to play back the received voice data and communicate it to the user; means for recording the learning progress and learning content in a database when the virtual historical experience ends; and means for the server to end the session. This allows users to have a high-quality learning experience in real time through a conversational and interactive virtual historical experience, thereby improving learning efficiency.

[0754] "User" refers to an individual who uses this system to experience virtual history.

[0755] "Virtual history experience scenario" refers to content that allows people to learn about specific historical events or cultures that are recreated in a virtual space.

[0756] "Terminal" refers to the device used by a user to access this system, including computers, tablets, smartphones, VR headsets, etc.

[0757] A "three-dimensional model" refers to a digital representation of three-dimensional objects and environments that a user can visually experience in a virtual space.

[0758] "Audio data" refers to data that stores audio information related to a virtual history experience scenario in digital format.

[0759] "Server" refers to a central component of a computer network that processes requests from user terminals and provides the necessary data.

[0760] "Generative AI" refers to AI technology that analyzes users' questions and generates appropriate answers.

[0761] "Answer" refers to the response content generated in response to a user's question analyzed by the generating artificial intelligence.

[0762] "Speech recognition" refers to the technology of converting a user's speech into digital data and understanding its content as text.

[0763] "Database" refers to an information system for systematically storing and managing information such as learning progress and experience.

[0764] A "session" refers to a period of time during which a user performs a series of operations from the time the user starts using the system until the time the user finishes using the system.

[0765] MODE FOR CARRYING OUT THE INVENTION

[0766] This invention is a system that allows users to learn about history and culture in an interactive manner through a virtual historical experience. The program of this system consists of a series of processes, such as user authentication and login, selection and start of the virtual historical experience, interactive historical experience, end of the experience, and recording of the results.

[0767] User Authentication and Login

[0768] The user enters a username and password from the terminal. The terminal encrypts this information using the AES encryption method and sends it to the server. The server checks the received information against a database (e.g., MySQL) to see if the user exists. If authentication is successful, the server generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[0769] Examples:

[0770] The user enters the username "example_user" and password "example_pass" and presses the "Login" button on the terminal. The terminal encrypts this with AES and sends the "encrypted data" to the server. The server decrypts the received data and checks it against the MySQL database. If authentication is successful, the server generates a session ID "example_session_id" and sends it to the terminal. The terminal records the session ID and displays "Login successful" to the user.

[0771] Select and start your virtual history experience

[0772] The user checks a list of virtual history experience scenarios provided on the device. When the user selects a scenario of interest, the device sends the selection information to the server. The server collects a 3D model (e.g., Unity engine), audio data (e.g., MP3 file), and related text information corresponding to the selected scenario and sends them to the device. The device uses the received data to render the virtual space.

[0773] Examples:

[0774] The user opens the "Scenario Selection" screen on their device and selects "Ancient Egyptian Pyramids." When they press the "Select" button, the device sends "Scenario ID: egypt_pyramid" to the server. The server uses the Unity 3D engine to collect a three-dimensional model of the pyramid, related MP3 audio data, and other necessary information, and sends it to the device. The device receives this and renders it through a VR headset, allowing the user to explore the virtual pyramid.

[0775] Interactive History Experience

[0776] In a virtual space, users interact with historical figures. They use a microphone to ask questions or give instructions. The device converts the audio input from the microphone into text using speech recognition software (e.g., Google Cloud Speech-to-Text) and sends it to a server. The server uses generative AI (e.g., OpenAI GPT-3) to analyze the user's question and generate an appropriate answer. The answer is converted into audio data using speech synthesis software (e.g., Amazon Polly) and sent to the device. The device plays the audio data and communicates it to the user.

[0777] Examples:

[0778] The user asks in the virtual space, "Who built this pyramid?" The device converts the speech to text using Google Cloud Speech-to-Text and sends the text "Who built this pyramid?" to the server. The server analyzes the question using OpenAI GPT-3 and generates the answer "This pyramid was built by Pharaoh Khufu." The answer is then converted into audio data using Amazon Polly and sent to the device. The device plays the audio data and communicates it to the user.

[0779] Ending the experience and recording the results

[0780] When the user finishes the experience, the device notifies the server. The server saves the user's session data (learning progress, questions, experience details, etc.) in a database. After saving, the server ends the current session and invalidates the session ID.

[0781] Examples:

[0782] When the user presses the "End" button on the device, the device sends "User ID: example_user, Session ID: example_session_id, Status: Ended" to the server. The server compiles the session data (learning progress, question content, experience details) and saves it in a MySQL database. Once saved, the server ends the session and invalidates the session ID. The user is then told "Experience has ended."

[0783] Prompt Sentence Examples

[0784] Below is a concrete example of a prompt sentence to be input to the generative AI model. The purpose is to generate answers to questions posed by users during a virtual history experience.

[0785] Prompt Sentence Examples

[0786] User Question: "Who built this pyramid?"

[0787] Context: "A user asked this question while exploring an ancient Egyptian pyramid. This pyramid was built by Pharaoh Khufu."

[0788] Expected answer: "This pyramid was built by Khufu."

[0789] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0790] User Authentication and Login

[0791] Step 1:

[0792] Specific behavior:

[0793] The user enters the user name and password from the terminal.

[0794] input:

[0795] Username and password.

[0796] process:

[0797] The terminal encrypts the input information using the AES encryption method.

[0798] output:

[0799] Encrypted authentication data.

[0800] Step 2:

[0801] Specific behavior:

[0802] The terminal transmits the encrypted authentication data to the server.

[0803] input:

[0804] Encrypted authentication data.

[0805] process:

[0806] Send data from the terminal to the server.

[0807] output:

[0808] The authentication data received by the server.

[0809] Step 3:

[0810] Specific behavior:

[0811] The server checks the received authentication data against a database.

[0812] input:

[0813] Authentication data.

[0814] process:

[0815] The server checks if the user exists by checking against a database (e.g. MySQL).

[0816] output:

[0817] The authentication result (success or failure).

[0818] Step 4:

[0819] Specific behavior:

[0820] If authentication is successful, the server generates a new session ID.

[0821] input:

[0822] Authentication result (success).

[0823] process:

[0824] The server generates a session ID.

[0825] output:

[0826] The generated session ID.

[0827] Step 5:

[0828] Specific behavior:

[0829] The server sends the generated session ID to the terminal.

[0830] input:

[0831] Session ID.

[0832] process:

[0833] The server sends a session ID to the terminal.

[0834] output:

[0835] The session ID received by the device.

[0836] Step 6:

[0837] Specific behavior:

[0838] The terminal saves the session ID and notifies the user that the login was successful.

[0839] input:

[0840] Session ID.

[0841] process:

[0842] The terminal will save the session ID and display a successful login message on the screen.

[0843] output:

[0844] Stored session ID and successful login notification.

[0845] Select and start your virtual history experience

[0846] Step 1:

[0847] Specific behavior:

[0848] The user checks the list of virtual history experience scenarios provided on the terminal.

[0849] input:

[0850] View the user's scenario list.

[0851] process:

[0852] The device displays a list of scenarios.

[0853] output:

[0854] Scenario list.

[0855] Step 2:

[0856] Specific behavior:

[0857] The user selects the scenario that interests them.

[0858] input:

[0859] Select a scenario.

[0860] process:

[0861] The terminal stores the selection information.

[0862] output:

[0863] Selected scenario information.

[0864] Step 3:

[0865] Specific behavior:

[0866] The terminal transmits the selection information to the server.

[0867] input:

[0868] Selected scenario information.

[0869] process:

[0870] Send data from the terminal to the server.

[0871] output:

[0872] Scenario information received on the server.

[0873] Step 4:

[0874] Specific behavior:

[0875] The server collects the three-dimensional model, audio data, and associated text information corresponding to the selected scenario.

[0876] input:

[0877] Selected scenario information.

[0878] process:

[0879] The server collects 3D models (e.g., Unity engine), audio data (e.g., MP3 files), and text information.

[0880] output:

[0881] Data collected.

[0882] Step 5:

[0883] Specific behavior:

[0884] The server transmits the collected data to the terminal.

[0885] input:

[0886] Data collected.

[0887] process:

[0888] Data is sent from the server to the terminal.

[0889] output:

[0890] Three-dimensional models, audio data, and text information arrive at the terminal.

[0891] Step 6:

[0892] Specific behavior:

[0893] The device uses the received data to render the virtual space.

[0894] input:

[0895] Three-dimensional models, audio data, and text information.

[0896] process:

[0897] The device renders the virtual space based on the received data.

[0898] output:

[0899] Rendered virtual space.

[0900] Interactive History Experience

[0901] Step 1:

[0902] Specific behavior:

[0903] Users can ask questions and give instructions through a microphone in the virtual space.

[0904] input:

[0905] User voice.

[0906] process:

[0907] The device records the audio.

[0908] output:

[0909] Recorded audio data.

[0910] Step 2:

[0911] Specific behavior:

[0912] The device converts the voice into text using speech recognition software.

[0913] input:

[0914] Audio data.

[0915] process:

[0916] The device converts speech to text using Google Cloud Speech-to-Text.

[0917] output:

[0918] Text data.

[0919] Step 3:

[0920] Specific behavior:

[0921] The terminal transmits the converted text data to the server.

[0922] input:

[0923] Text data.

[0924] process:

[0925] Send data from the terminal to the server.

[0926] output:

[0927] Text data received by the server.

[0928] Step 4:

[0929] Specific behavior:

[0930] The server uses generative AI to analyze the text data and generate appropriate answers.

[0931] input:

[0932] Text data.

[0933] process:

[0934] The server uses generative AI (e.g., OpenAI GPT-3) to analyze the question and generate an answer.

[0935] output:

[0936] The generated answer text.

[0937] Step 5:

[0938] Specific behavior:

[0939] The server converts the generated response into voice data using voice synthesis software.

[0940] input:

[0941] Answer text.

[0942] process:

[0943] The server uses Amazon Polly to convert the response into audio data.

[0944] output:

[0945] Audio data.

[0946] Step 6:

[0947] Specific behavior:

[0948] The server transmits the generated voice data to the terminal.

[0949] input:

[0950] Audio data.

[0951] process:

[0952] Data is sent from the server to the terminal.

[0953] output:

[0954] Audio data received at the device.

[0955] Step 7:

[0956] Specific behavior:

[0957] The terminal reproduces the audio data and conveys it to the user.

[0958] input:

[0959] Audio data.

[0960] process:

[0961] The device plays the audio data.

[0962] output:

[0963] A spoken response is conveyed to the user.

[0964] Ending the experience and recording the results

[0965] Step 1:

[0966] Specific behavior:

[0967] The user selects to end the experience.

[0968] input:

[0969] Exit selection.

[0970] process:

[0971] The terminal generates an exit command.

[0972] output:

[0973] Exit command.

[0974] Step 2:

[0975] Specific behavior:

[0976] The device sends a notification of the end of the experience to the server.

[0977] input:

[0978] Exit command.

[0979] process:

[0980] Send commands from the terminal to the server.

[0981] output:

[0982] The termination command received by the server.

[0983] Step 3:

[0984] Specific behavior:

[0985] The server collects user session data (learning progress, questions, experience details, etc.) and stores them in a database.

[0986] input:

[0987] Exit command.

[0988] process:

[0989] The server stores the session data in a database (e.g. MySQL).

[0990] output:

[0991] Saved session data.

[0992] Step 4:

[0993] Specific behavior:

[0994] The server ends the current session and invalidates the session ID.

[0995] input:

[0996] Termination notice.

[0997] process:

[0998] The server ends the session and invalidates the session ID.

[0999] output:

[1000] Invalidated session ID and session end notification.

[1001] (Application example 1)

[1002] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1003] Conventional virtual history experience systems are often experienced by individuals in limited spaces such as their own rooms, and are insufficient as a place to share experiences with other visitors. Furthermore, it is difficult to provide an interactive experience that requires users to physically move around, limiting the effectiveness of learning about history and culture. This has reduced opportunities for visitors to learn while enjoying deeper experiences in brick-and-mortar locations such as museums and theme parks.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1005] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and audio data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the user's received audio data, generate an answer using a generation AI, and transmit the audio data to the terminal; means for the terminal to play back the received audio data and communicate it to the user; means for recording the progress and learning content in a database when the virtual historical experience ends; means for the server to end the session; means for providing the physical store with a video display device and audio playback device for the user to experience visually and audibly; and means for the user to experience the virtual historical experience while physically moving around the store. This enables visitors to participate in a virtual historical experience while interactively moving around the physical store.

[1006] "User authentication" is the process by which a user provides individual identifying information, that information is verified, and access rights are granted.

[1007] A "virtual history experience scenario" is a scenario that recreates a specific historical period or scene, allowing users to experience and interact with that period within a virtual environment.

[1008] A "terminal" is a device that a user uses to experience virtual history, and includes smartphones, smart glasses, head-mounted displays, etc.

[1009] A "3D model" is a digital model of an object represented in three-dimensional space using computer graphics.

[1010] "Audio data" refers to data in which audio is recorded in digital format, and is used for audio communication with a user and for reproducing audio information.

[1011] A "server" is a remote computer system that processes requests from user terminals and provides the required data or services.

[1012] "Generative AI" is an artificial intelligence system that performs natural language processing in response to user questions and instructions, generating appropriate answers and information.

[1013] "Progress" is data that records what the user has learned and the goals they have achieved through the virtual history experience.

[1014] A "database" is a system that can efficiently and safely manage large amounts of data and quickly search, update, and delete it.

[1015] A "session" refers to a series of interactions between a user's login and logout, and is a process in which the operations and data exchanges that occur during that time are tracked.

[1016] An "image display device" is a device that allows a user to visually check information or images, and includes smart glasses and head-mounted displays.

[1017] An "audio playback device" is a device for playing back audio data as sound, and includes speakers and headsets.

[1018] A "physical store" is a store or facility where commerce or services are conducted at a physical location.

[1019] "Physically moving around the store" means that the user actually uses their own body to walk around the store.

[1020] MODE FOR CARRYING OUT THE INVENTION

[1021] To implement this invention, it is necessary to build a system that combines specific hardware and software to provide a virtual history experience system in a physical store. The main components of this invention are a server, a terminal, and a user. How each component operates and how the overall system functions are described below.

[1022] Hardware and Software Configuration

[1023] Hardware:

[1024] 1. Devices (head-mounted displays, smart glasses, etc.)

[1025] This is a display device that allows users to visually experience virtual history.

[1026] 2. Audio playback devices (speakers, headsets)

[1027] This is a device that plays back audio data sent from the server and transmits it to the user.

[1028] 3. Microphone

[1029] This is a device that recognizes the user's voice and transmits it to the server.

[1030] 4. Network Connectivity Devices

[1031] It is used to connect terminals and servers over a network.

[1032] software:

[1033] 1. Server

[1034] Data management software: Records user progress and data.

[1035] Generative AI model: Generative AI is used to generate appropriate answers to user questions, for example, OpenAI's GPT-3.

[1036] Database system: Manages users' learning progress and questions.

[1037] 2. Terminal

[1038] Speech recognition software: Converts user speech into text using software such as the Google Speech-to-Text API or AWS Transcribe.

[1039] Rendering engine: Rendering 3D models and virtual spaces using Unity, Unreal Engine, etc.

[1040] Interface software: manages the user-system interface.

[1041] Specific example of system operation

[1042] 1. User authentication and login

[1043] The user enters login information (e.g., username and password) from the terminal. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, a session ID is generated and sent to the terminal.

[1044] 2. Select and start your virtual history experience

[1045] The user selects a virtual historical experience scenario (e.g., "Japan in the Edo Period") from their device. The server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space, allowing the user to experience the historical setting as if they were actually there.

[1046] 3. Interactive History Experience

[1047] Within the virtual space, users can interact with historical figures. The user's voice is recognized through a microphone, converted into text, and sent to the server. The server uses generative AI to generate appropriate answers to the user's questions, converts the answers into audio data, and sends it to the device. The audio data is played back on the device and conveyed to the user.

[1048] 4. Completing the experience and recording the results

[1049] When the user finishes the experience, the device sends a completion notification to the server. The server records the user's progress and learning content in a database and ends the session. This saves the user's learning history and can be used for the next experience.

[1050] Specific prompt examples

[1051] In response to a user question, "What products do you have available today?", the generative AI model is given the following prompt:

[1052] Suppose a user asks a merchant in the Edo period about today's products. User: "What products do you have available today?" The merchant will respond based on the Edo period setting. Merchant: "Today's recommendations are fresh fish and freshly baked dumplings."

[1053] In this way, the brick-and-mortar virtual history experience system can provide users with an interactive and immersive learning experience.

[1054] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1055] Step 1:

[1056] The user enters login information (username and password) using the terminal. The terminal encrypts this information and sends it to the server. The server compares the entered authentication information with the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the received session ID, and the user is notified that login was successful.

[1057] Input: Username, Password

[1058] Data processing: Username and password encryption

[1059] Data calculation: Authentication verification

[1060] Output: Session ID

[1061] Specific actions: Entering data into the login form, encrypting data, generating a token (session ID)

[1062] Step 2:

[1063] The user displays a list of virtual historical experience scenarios provided by the terminal and selects one that interests them. The terminal then sends the selected scenario information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the terminal. The terminal then receives this data and renders the virtual space.

[1064] Input: User selects a scenario

[1065] Data processing: Collecting data corresponding to the scenario

[1066] Data calculation: Sending 3D models and audio data

[1067] Output: Virtual space rendering

[1068] Specific operations: scenario selection, data collection and transmission, construction of a virtual environment using a rendering engine

[1069] Step 3:

[1070] Within the virtual space, users interact with historical figures through voice. The user's voice is picked up by the device's microphone and converted into text using speech recognition software (e.g., Google Speech-to-Text API). The converted text is sent to a server. The server uses generative AI (e.g., OpenAI's GPT-3) to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device then plays the audio data to the user through a playback device.

[1071] Input: User's voice

[1072] Data processing: Speech-to-text (voice recognition)

[1073] Data calculation: Question analysis and answer generation (generative AI)

[1074] Output: Audio data

[1075] Specific operations: Acquiring voice using a microphone, converting it to text, generating answers using AI, and playing back voice data

[1076] Step 4:

[1077] When the user finishes the virtual history experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experiences. The server then ends the current session and invalidates the session ID.

[1078] Input: User's termination notice

[1079] Data processing: Session data aggregation

[1080] Data calculation: saving to database and ending session

[1081] Output: Saved learning progress data

[1082] Specific actions: Sending a completion notice, recording progress in the database, and terminating the session

[1083] In this way, the system of the present invention can provide users with an interactive and immersive virtual history experience.

[1084] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1085] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1086] User Authentication and Login

[1087] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[1088] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[1089] Select and start your virtual history experience

[1090] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[1091] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[1092] Interactive History Experience and Emotion Recognition

[1093] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1094] As a concrete example, when a user asks, "Who built this pyramid?", the device converts the question into text and analyzes the emotion using an emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Amazing, isn't it?" The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1095] Ending the experience and recording the results

[1096] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[1097] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server then saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[1098] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture while taking their emotions into consideration through an interactive virtual historical experience.

[1099] The processing flow will be explained below.

[1100] Step 1:

[1101] The server starts the system. The server loads the necessary modules and databases, and loads the 3D model data, speech generation engine, emotion engine, and historical data into memory.

[1102] Step 2:

[1103] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[1104] Step 3:

[1105] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[1106] Step 4:

[1107] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[1108] Step 5:

[1109] The terminal saves the session ID and notifies the user that they have successfully logged in. The terminal records the session ID received from the server and uses it for subsequent requests.

[1110] Step 6:

[1111] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[1112] Step 7:

[1113] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[1114] Step 8:

[1115] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[1116] Step 9:

[1117] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[1118] Step 10:

[1119] Users interact with historical figures in a virtual space, asking questions and giving instructions by voice.

[1120] Step 11:

[1121] The device recognizes the user's voice and analyzes their emotions using an emotion engine. The device then converts the voice data into text and sends it to the server along with the emotion data.

[1122] Step 12:

[1123] The server uses generation AI to analyze the user's question and generates the optimal answer based on the user's question content and emotional data.

[1124] Step 13:

[1125] The server converts the generated response into voice data using a speech synthesis engine and sends it to the device. The server then reflects emotional data in the response data before sending it.

[1126] Step 14:

[1127] The device then plays back the received voice data and conveys it to the user. The device then plays back the voice, conveying it as if the historical figure were responding in a virtual space according to the user's emotions.

[1128] Step 15:

[1129] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[1130] Step 16:

[1131] The device sends a completion notification to the server. The device notifies the server that the user has completed the experience.

[1132] Step 17:

[1133] The server compiles and stores the user's session data in a database, including learning progress, questions, answers, and emotional data.

[1134] Step 18:

[1135] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[1136] Through these steps, users can engage in an interactive virtual history experience, learning about history and culture while receiving emotionally sensitive responses.

[1137] Example 2

[1138] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1139] Conventional virtual history experience systems often lack appropriate responses and interactions that take into account the user's emotional changes when learning about history and culture. This can reduce the user's learning effectiveness. Furthermore, when the user finishes the experience, the records are incomplete, which means that the experience cannot be fully utilized for the next experience.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1141] In this invention, the server includes: means for the user to select a virtual historical experience scenario; means for the terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space, analyzing their emotions, and transmitting the results to the server; means for the server to analyze the received user's voice data and emotion data, generate answers using a generation AI, and transmit the voice data to the terminal; means for recording the progress, learning content, and emotion data in a database when the virtual historical experience ends; and means for the server to end the session and invalidate the session ID. This enables an interactive historical experience that takes the user's emotions into consideration.

[1142] "User" refers to an individual who utilizes the virtual history experience provided by the invention.

[1143] "Virtual history experience scenario" refers to a series of scenarios that allow users to learn about history and culture in a virtual space.

[1144] "Terminal" refers to electronic devices such as computers, tablets, and smartphones that are operated by users.

[1145] "Server" refers to the central computer system that receives, transmits, analyzes, and stores data.

[1146] "Three-dimensional model" refers to a three-dimensional shape or object displayed in a virtual space.

[1147] "Voice data" refers to data used to provide interaction and explanations to the user by voice.

[1148] "Virtual space" refers to a three-dimensional interactive space generated by a computer.

[1149] "Emotion" refers to the psychological state or reaction a user exhibits during a virtual experience.

[1150] "Analyzing emotions" refers to analyzing the user's psychological state and reactions from voice data, etc.

[1151] A "database" refers to a digital information storage system that systematically stores collected information and makes it available for retrieval as needed.

[1152] A "session ID" refers to a unique identifier used to identify a series of activities from when a user logs in to when they log out of a system.

[1153] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers to users' questions.

[1154] "Generating an answer" refers to the generative AI creating an answer based on the user's question.

[1155] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1156] User Authentication and Login

[1157] First, the user enters login information (username and password) from the terminal. The terminal encrypts this information and sends it to the server. The server compares the authentication information with a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID, and the user is notified that login was successful. As a concrete example, when a user enters a "username" and "password," the terminal sends this to the server, and the server compares it with the database for authentication. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[1158] Select and start your virtual history experience

[1159] The user reviews a list of virtual history experience scenarios provided by the device and selects one that interests them. The device sends the selection information to the server. The server collects 3D models, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space. As a specific example, when a user selects "Ancient Egyptian Pyramids," the server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space in which the user can explore the inside of a pyramid through a virtual reality headset or tablet.

[1160] Interactive History Experience and Emotion Recognition

[1161] In a virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then analyzes the user's question using a generative AI model and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user. For example, if a user asks, "Who built this pyramid?" the device converts the question into text and analyzes the emotion using the emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Isn't that amazing?" The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user.

[1162] Ending the experience and recording the results

[1163] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, details of the experience, and emotional data. The server ends the current session and invalidates the session ID. For example, when the user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[1164] Examples of prompt statements

[1165] Example prompt for generative AI model:

[1166] Prompt: "User asks: Who built this pyramid? Emotion: Surprise"

[1167] This system allows users to learn more about history and culture through an interactive virtual historical experience, taking their emotions into account.

[1168] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1169] Step 1:

[1170] The user enters login information (user name and password) from the terminal.

[1171] Input: Username, Password

[1172] Output: Encrypted login information

[1173] Specific behavior:

[1174] The user enters their username and password into the terminal's login screen, which encrypts the information and sends it to the next processing step.

[1175] Step 2:

[1176] The device sends the encrypted login information to the server.

[1177] Input: Encrypted login information

[1178] Output: Sending login information to the server

[1179] Specific behavior:

[1180] The device sends encrypted login information to the server using the HTTPS protocol.

[1181] Step 3:

[1182] The server decrypts the received login information and checks it against the credentials in its database.

[1183] Input: Encrypted login information

[1184] Output: Authentication success or failure

[1185] Specific behavior:

[1186] The server decrypts the encrypted login information, checks it against the credentials stored in the database, and if successful, generates a new session ID.

[1187] Step 4:

[1188] The server sends the generated session ID to the terminal.

[1189] Input: Authentication success result, session ID

[1190] Output: Sending the session ID to the terminal

[1191] Specific behavior:

[1192] The server sends the successful authentication result and the newly generated session ID to the terminal.

[1193] Step 5:

[1194] The terminal saves the received session ID and notifies the user that login was successful.

[1195] Input: Session ID

[1196] Output: Successful login notification

[1197] Specific behavior:

[1198] The terminal stores the received session ID and displays a login success message to the user.

[1199] Step 6:

[1200] The user selects a virtual history experience scenario.

[1201] Input: Select a virtual history experience scenario

[1202] Output: Information about the selected scenario

[1203] Specific behavior:

[1204] The user selects a scenario of interest from a list of multiple virtual history experience scenarios.

[1205] Step 7:

[1206] The terminal transmits the selected scenario information to the server.

[1207] Input: Scenario selection information

[1208] Output: Sending the selected information to the server

[1209] Specific behavior:

[1210] The terminal transmits the scenario information selected by the user to the server.

[1211] Step 8:

[1212] The server collects the three-dimensional model, audio data, and related information corresponding to the selected scenario from a database.

[1213] Input: Scenario selection information

[1214] Output: 3D model, audio data, related information

[1215] Specific behavior:

[1216] The server accesses a database to collect three-dimensional models, audio data, and related information corresponding to the selected scenario.

[1217] Step 9:

[1218] The server sends the collected data to the terminal.

[1219] Input: 3D model, audio data, related information

[1220] Output: Sending data to a terminal

[1221] Specific behavior:

[1222] The server transmits the collected data to the terminal.

[1223] Step 10:

[1224] The terminal uses the received data to render the virtual space and displays it to the user.

[1225] Input: 3D model, audio data, related information

[1226] Output: Rendering of the virtual space

[1227] Specific behavior:

[1228] The device uses the Unity engine to render the received 3D models and audio data, displaying the virtual space to the user.

[1229] Step 11:

[1230] Users interact with historical figures in a virtual space.

[1231] Input: Voice input (questions and instructions)

[1232] Output: Audio data

[1233] Specific behavior:

[1234] Users can ask questions and give instructions to historical figures by voice using a microphone in the virtual space.

[1235] Step 12:

[1236] The device recognizes the voice data, analyzes the emotion using an emotion engine, and sends it to the server.

[1237] Input: Audio data

[1238] Output: Text data, emotion data

[1239] Specific behavior:

[1240] The device converts the voice data into text using a speech recognition engine (e.g., Google Speech-to-Text API), analyzes the emotion using an emotion engine, and then sends the text data and emotion data to the server.

[1241] Step 13:

[1242] The server uses the received text data and emotion data to generate an answer using a generative AI model.

[1243] Input: Text data, emotion data

[1244] Output: Answer text

[1245] Specific behavior:

[1246] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate answer based on the received text data and emotion data.

[1247] Step 14:

[1248] The server converts the generated response into voice data and transmits it to the terminal.

[1249] Input: Answer text

[1250] Output: Audio data

[1251] Specific behavior:

[1252] The server uses a service (e.g., Amazon Polly) to convert the response text into voice data and sends the generated voice data to the device.

[1253] Step 15:

[1254] The terminal plays back the received audio data and conveys it to the user.

[1255] Input: Audio data

[1256] Output: Audio playback

[1257] Specific behavior:

[1258] The terminal plays back the received voice data and conveys the answer to the user.

[1259] Step 16:

[1260] When the user finishes the experience, the device sends a completion notification to the server.

[1261] Input: Termination notification

[1262] Output: Sending notification of completion to the server

[1263] Specific behavior:

[1264] The user performs an end operation (e.g., clicking the "End" button), and the terminal sends an end notification to the server.

[1265] Step 17:

[1266] The server saves the user's session data to a database and ends the session.

[1267] Input: Termination notification, session data

[1268] Output: Save data, end session

[1269] Specific behavior:

[1270] The server stores the user's learning progress, questions, experience details, and emotional data in a database, invalidates the session ID, and ends the session.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1273] Conventional virtual history experience systems have struggled to provide an interactive experience that reflects the user's emotions. Furthermore, due to a lack of efficient means for recording and utilizing the user's learning progress and emotions, it has been difficult to provide an optimal learning environment for each individual user. This has led to the issue of users being unable to learn about history and culture in depth.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1275] In this invention, the server includes: a means for the user to select a virtual history experience scenario; a means for the terminal to receive 3D models and voice data from the server based on the selected scenario; a means for recognizing the user's voice in the virtual space and transmitting it to the server; a means for using a generation AI to generate answers to the user's questions and creating the voice data; a means for recognizing the user's emotions in real time and generating dialogue content using the emotion data; a means for recording the progress and learning content in a database when the virtual history experience ends; and a means for the server to end the session. This enables a more natural interactive virtual history experience that reflects the user's emotions.

[1276] A "user" is an entity that uses the system to experience virtual history.

[1277] A "virtual history experience scenario" is a historical or cultural scenario that a user can select and experience.

[1278] A "terminal" is a device used by a user for inputting and outputting information.

[1279] "3D model" refers to a three-dimensional object or environment displayed in a virtual space.

[1280] "Audio data" is digitized audio data used for recording, playing, transmitting, etc.

[1281] A "server" is a computer system that provides services to terminals over a network.

[1282] "Virtual space" refers to a three-dimensional digital space generated by a computer.

[1283] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate output data or content based on user input data.

[1284] A "session" refers to the period of time during which a user performs a series of operations or activities within a system.

[1285] "Emotion data" refers to data that analyzes and indicates the user's emotional state.

[1286] "Progress" is information that indicates how much learning and experience the user has achieved during the virtual history experience.

[1287] A "database" is a structured collection of data that a system manages and makes available.

[1288] "Dialogue content" refers to the content of questions and responses exchanged between the user and the system.

[1289] "Interactive" refers to the characteristic that operations and experiences progress as the user and the system influence each other.

[1290] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1291] Hardware and Software Configuration

[1292] Hardware

[1293] Device: Includes the smartphone, head-mounted display (HMD), microphone, and camera used by the user.

[1294] Server: A computer system that provides services to terminals over a network.

[1295] software

[1296] Django: a web application framework

[1297] OpenCV: A library for image processing

[1298] Google Cloud Speech-to-Text API: Speech Recognition

[1299] NVIDIA TensorFlow: A deep learning model for emotion recognition

[1300] Unity: 3D content rendering engine

[1301] DETAILED DESCRIPTION OF THE EMBODIMENTS

[1302] User Authentication and Login

[1303] The user enters login information on the terminal. The terminal encrypts this information and sends it to the server. The server checks the authentication information against a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the session ID, and the user is notified that login was successful.

[1304] Select and start your virtual history experience

[1305] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[1306] Interactive History Experience and Emotion Recognition

[1307] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1308] Ending the experience and recording the results

[1309] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[1310] (Example of a specific prompt)

[1311] As an example of a prompt sentence, the following can be entered:

[1312] Emotional state: Surprise, User: Who built this pyramid?

[1313] This allows the system to generate an appropriate response that takes into account the user's emotion when the user expresses surprise.

[1314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1315] Step 1:

[1316] The user enters login information from the terminal. The entered username and password are encrypted and sent to the server. The server checks the authentication information against the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[1317] Input: Username, Password

[1318] Output: Session ID, login success notification

[1319] Step 2:

[1320] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The selection information is sent to the server. The server collects 3D models and audio data corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space.

[1321] Input: Scenario selection information

[1322] Output: 3D models, audio data, rendered virtual worlds

[1323] Step 3:

[1324] Users interact with historical figures in a virtual space. The user's voice is captured by the device's microphone, converted into text, and the emotion engine analyzes the emotions. The analysis results are sent to the server.

[1325] Input: User voice

[1326] Output: Text data, emotion data

[1327] Step 4:

[1328] The server analyzes the received text data and emotional data and generates an appropriate response using a generative AI model. It generates a prompt sentence that takes the emotional data into account and creates a response based on that. The generated response is converted into voice data and sent to the device.

[1329] Input: Text data, emotion data

[1330] Output: Audio data, prompt

[1331] Step 5:

[1332] The device then plays back the received audio data and transmits it to the user, who continues the virtual history experience through voice responses.

[1333] Input: Audio data

[1334] Output: Audio playback

[1335] Step 6:

[1336] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores the learning progress, questions, experience details, and emotional data in a database. The server then ends the current session and invalidates the session ID.

[1337] Input: Termination notification

[1338] Output: Database record, session termination confirmation

[1339] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1341] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1342] [Third embodiment]

[1343] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1344] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1345] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1346] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1347] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1348] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1349] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1350] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1351] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1352] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1353] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1354] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1355] This invention is a system that allows users to interactively learn about history and culture through a virtual historical experience. Specific processing of each step in the embodiment of the invention will be described below.

[1356] User Authentication and Login

[1357] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[1358] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[1359] Select and start your virtual history experience

[1360] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[1361] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[1362] Interactive History Experience

[1363] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, converts it into text, and sends it to the server. The server uses generative AI to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device plays the audio and communicates it to the user.

[1364] For example, if a user asks, "Who built this pyramid?", the device converts the question into text and sends it to the server. The server uses generative AI to generate the answer, "This pyramid was built by Pharaoh Khufu," and sends it along with the audio data to the device. The device then plays back the audio and relays it to the user.

[1365] Ending the experience and recording the results

[1366] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experience details. The server then ends the current session and invalidates the session ID.

[1367] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress in a database and ends the session. This records the user's learning history and can be used for the next experience.

[1368] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture through an interactive virtual historical experience.

[1369] The processing flow will be explained below.

[1370] Step 1:

[1371] The server starts the system, loads the necessary modules and databases, and loads the 3D model data, speech generation engine, and historical data into memory.

[1372] Step 2:

[1373] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[1374] Step 3:

[1375] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[1376] Step 4:

[1377] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[1378] Step 5:

[1379] The terminal saves the session ID and notifies the user that the login was successful. The terminal remembers the session ID received from the server and uses it for subsequent requests.

[1380] Step 6:

[1381] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[1382] Step 7:

[1383] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[1384] Step 8:

[1385] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[1386] Step 9:

[1387] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[1388] Step 10:

[1389] Users interact with historical figures in a virtual space, asking questions and giving instructions through an interface.

[1390] Step 11:

[1391] The device recognizes the user's voice and sends it to the server, which converts it into text, encrypts it, and sends it to the server.

[1392] Step 12:

[1393] The server uses the generation AI to analyze the user's question and generate an appropriate answer, which is then passed to a speech synthesis engine and converted into voice data.

[1394] Step 13:

[1395] The server transmits the generated voice data to the terminal.

[1396] Step 14:

[1397] The terminal plays back the audio and conveys it to the user. The terminal plays back the received audio data and conveys it to the user as if a historical figure were speaking in the virtual space.

[1398] Step 15:

[1399] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[1400] Step 16:

[1401] The device sends a termination notification to the server. The device notifies the server that the user has terminated the experience.

[1402] Step 17:

[1403] The server compiles and stores user session data in a database, recording learning progress, questions, and experience details.

[1404] Step 18:

[1405] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[1406] Through these steps, users can engage in an interactive virtual historical experience and learn more about history and culture.

[1407] Example 1

[1408] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] Conventional educational systems have struggled to provide users with a conversational and interactive learning experience when learning about history and culture. They also lacked sophisticated systems that could respond appropriately to users' questions and provide information in real time. Furthermore, they lacked the ability to record learning progress and experiences and use them for future learning, making it difficult to improve users' learning efficiency.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1411] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the received user voice data, generate an answer using a generative artificial intelligence, and transmit the voice data to the terminal; means for the terminal to play back the received voice data and communicate it to the user; means for recording the learning progress and learning content in a database when the virtual historical experience ends; and means for the server to end the session. This allows users to have a high-quality learning experience in real time through a conversational and interactive virtual historical experience, thereby improving learning efficiency.

[1412] "User" refers to an individual who uses this system to experience virtual history.

[1413] "Virtual history experience scenario" refers to content that allows people to learn about specific historical events or cultures that are recreated in a virtual space.

[1414] "Terminal" refers to the device used by a user to access this system, including computers, tablets, smartphones, VR headsets, etc.

[1415] A "three-dimensional model" refers to a digital representation of three-dimensional objects and environments that a user can visually experience in a virtual space.

[1416] "Audio data" refers to data that stores audio information related to a virtual history experience scenario in digital format.

[1417] "Server" refers to a central component of a computer network that processes requests from user terminals and provides the necessary data.

[1418] "Generative AI" refers to AI technology that analyzes users' questions and generates appropriate answers.

[1419] "Answer" refers to the response content generated in response to a user's question analyzed by the generating artificial intelligence.

[1420] "Speech recognition" refers to the technology of converting a user's speech into digital data and understanding its content as text.

[1421] "Database" refers to an information system for systematically storing and managing information such as learning progress and experience.

[1422] A "session" refers to a period of time during which a user performs a series of operations from the time the user starts using the system until the time the user finishes using the system.

[1423] MODE FOR CARRYING OUT THE INVENTION

[1424] This invention is a system that allows users to learn about history and culture in an interactive manner through a virtual historical experience. The program of this system consists of a series of processes, such as user authentication and login, selection and start of the virtual historical experience, interactive historical experience, end of the experience, and recording of the results.

[1425] User Authentication and Login

[1426] The user enters a username and password from the terminal. The terminal encrypts this information using the AES encryption method and sends it to the server. The server checks the received information against a database (e.g., MySQL) to see if the user exists. If authentication is successful, the server generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[1427] Examples:

[1428] The user enters the username "example_user" and password "example_pass" and presses the "Login" button on the terminal. The terminal encrypts this with AES and sends the "encrypted data" to the server. The server decrypts the received data and checks it against the MySQL database. If authentication is successful, the server generates a session ID "example_session_id" and sends it to the terminal. The terminal records the session ID and displays "Login successful" to the user.

[1429] Select and start your virtual history experience

[1430] The user checks a list of virtual history experience scenarios provided on the device. When the user selects a scenario of interest, the device sends the selection information to the server. The server collects a 3D model (e.g., Unity engine), audio data (e.g., MP3 file), and related text information corresponding to the selected scenario and sends them to the device. The device uses the received data to render the virtual space.

[1431] Examples:

[1432] The user opens the "Scenario Selection" screen on their device and selects "Ancient Egyptian Pyramids." When they press the "Select" button, the device sends "Scenario ID: egypt_pyramid" to the server. The server uses the Unity 3D engine to collect a three-dimensional model of the pyramid, related MP3 audio data, and other necessary information, and sends it to the device. The device receives this and renders it through a VR headset, allowing the user to explore the virtual pyramid.

[1433] Interactive History Experience

[1434] In a virtual space, users interact with historical figures. They use a microphone to ask questions or give instructions. The device converts the audio input from the microphone into text using speech recognition software (e.g., Google Cloud Speech-to-Text) and sends it to a server. The server uses generative AI (e.g., OpenAI GPT-3) to analyze the user's question and generate an appropriate answer. The answer is converted into audio data using speech synthesis software (e.g., Amazon Polly) and sent to the device. The device plays the audio data and communicates it to the user.

[1435] Examples:

[1436] The user asks in the virtual space, "Who built this pyramid?" The device converts the speech to text using Google Cloud Speech-to-Text and sends the text "Who built this pyramid?" to the server. The server analyzes the question using OpenAI GPT-3 and generates the answer "This pyramid was built by Pharaoh Khufu." The answer is then converted into audio data using Amazon Polly and sent to the device. The device plays the audio data and communicates it to the user.

[1437] Ending the experience and recording the results

[1438] When the user finishes the experience, the device notifies the server. The server saves the user's session data (learning progress, questions, experience details, etc.) in a database. After saving, the server ends the current session and invalidates the session ID.

[1439] Examples:

[1440] When the user presses the "End" button on the device, the device sends "User ID: example_user, Session ID: example_session_id, Status: Ended" to the server. The server compiles the session data (learning progress, question content, experience details) and saves it in a MySQL database. Once saved, the server ends the session and invalidates the session ID. The user is then told "Experience has ended."

[1441] Prompt Sentence Examples

[1442] Below is a concrete example of a prompt sentence to be input to the generative AI model. The purpose is to generate answers to questions posed by users during a virtual history experience.

[1443] Prompt Sentence Examples

[1444] User Question: "Who built this pyramid?"

[1445] Context: "A user asked this question while exploring an ancient Egyptian pyramid. This pyramid was built by Pharaoh Khufu."

[1446] Expected answer: "This pyramid was built by Khufu."

[1447] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1448] User Authentication and Login

[1449] Step 1:

[1450] Specific behavior:

[1451] The user enters the user name and password from the terminal.

[1452] input:

[1453] Username and password.

[1454] process:

[1455] The terminal encrypts the input information using the AES encryption method.

[1456] output:

[1457] Encrypted authentication data.

[1458] Step 2:

[1459] Specific behavior:

[1460] The terminal transmits the encrypted authentication data to the server.

[1461] input:

[1462] Encrypted authentication data.

[1463] process:

[1464] Send data from the terminal to the server.

[1465] output:

[1466] The authentication data received by the server.

[1467] Step 3:

[1468] Specific behavior:

[1469] The server checks the received authentication data against a database.

[1470] input:

[1471] Authentication data.

[1472] process:

[1473] The server checks if the user exists by checking against a database (e.g. MySQL).

[1474] output:

[1475] The authentication result (success or failure).

[1476] Step 4:

[1477] Specific behavior:

[1478] If authentication is successful, the server generates a new session ID.

[1479] input:

[1480] Authentication result (success).

[1481] process:

[1482] The server generates a session ID.

[1483] output:

[1484] The generated session ID.

[1485] Step 5:

[1486] Specific behavior:

[1487] The server sends the generated session ID to the terminal.

[1488] input:

[1489] Session ID.

[1490] process:

[1491] The server sends a session ID to the terminal.

[1492] output:

[1493] The session ID received by the device.

[1494] Step 6:

[1495] Specific behavior:

[1496] The terminal saves the session ID and notifies the user that the login was successful.

[1497] input:

[1498] Session ID.

[1499] process:

[1500] The terminal will save the session ID and display a successful login message on the screen.

[1501] output:

[1502] Stored session ID and successful login notification.

[1503] Select and start your virtual history experience

[1504] Step 1:

[1505] Specific behavior:

[1506] The user checks the list of virtual history experience scenarios provided on the terminal.

[1507] input:

[1508] View the user's scenario list.

[1509] process:

[1510] The device displays a list of scenarios.

[1511] output:

[1512] Scenario list.

[1513] Step 2:

[1514] Specific behavior:

[1515] The user selects the scenario that interests them.

[1516] input:

[1517] Select a scenario.

[1518] process:

[1519] The terminal stores the selection information.

[1520] output:

[1521] Selected scenario information.

[1522] Step 3:

[1523] Specific behavior:

[1524] The terminal transmits the selection information to the server.

[1525] input:

[1526] Selected scenario information.

[1527] process:

[1528] Send data from the terminal to the server.

[1529] output:

[1530] Scenario information received on the server.

[1531] Step 4:

[1532] Specific behavior:

[1533] The server collects the three-dimensional model, audio data, and associated text information corresponding to the selected scenario.

[1534] input:

[1535] Selected scenario information.

[1536] process:

[1537] The server collects 3D models (e.g., Unity engine), audio data (e.g., MP3 files), and text information.

[1538] output:

[1539] Data collected.

[1540] Step 5:

[1541] Specific behavior:

[1542] The server transmits the collected data to the terminal.

[1543] input:

[1544] Data collected.

[1545] process:

[1546] Data is sent from the server to the terminal.

[1547] output:

[1548] Three-dimensional models, audio data, and text information arrive at the terminal.

[1549] Step 6:

[1550] Specific behavior:

[1551] The device uses the received data to render the virtual space.

[1552] input:

[1553] Three-dimensional models, audio data, and text information.

[1554] process:

[1555] The device renders the virtual space based on the received data.

[1556] output:

[1557] Rendered virtual space.

[1558] Interactive History Experience

[1559] Step 1:

[1560] Specific behavior:

[1561] Users can ask questions and give instructions through a microphone in the virtual space.

[1562] input:

[1563] User voice.

[1564] process:

[1565] The device records the audio.

[1566] output:

[1567] Recorded audio data.

[1568] Step 2:

[1569] Specific behavior:

[1570] The device converts the voice into text using speech recognition software.

[1571] input:

[1572] Audio data.

[1573] process:

[1574] The device converts speech to text using Google Cloud Speech-to-Text.

[1575] output:

[1576] Text data.

[1577] Step 3:

[1578] Specific behavior:

[1579] The terminal transmits the converted text data to the server.

[1580] input:

[1581] Text data.

[1582] process:

[1583] Send data from the terminal to the server.

[1584] output:

[1585] Text data received by the server.

[1586] Step 4:

[1587] Specific behavior:

[1588] The server uses generative AI to analyze the text data and generate appropriate answers.

[1589] input:

[1590] Text data.

[1591] process:

[1592] The server uses generative AI (e.g., OpenAI GPT-3) to analyze the question and generate an answer.

[1593] output:

[1594] The generated answer text.

[1595] Step 5:

[1596] Specific behavior:

[1597] The server converts the generated response into voice data using voice synthesis software.

[1598] input:

[1599] Answer text.

[1600] process:

[1601] The server uses Amazon Polly to convert the response into audio data.

[1602] output:

[1603] Audio data.

[1604] Step 6:

[1605] Specific behavior:

[1606] The server transmits the generated voice data to the terminal.

[1607] input:

[1608] Audio data.

[1609] process:

[1610] Data is sent from the server to the terminal.

[1611] output:

[1612] Audio data received at the device.

[1613] Step 7:

[1614] Specific behavior:

[1615] The terminal reproduces the audio data and conveys it to the user.

[1616] input:

[1617] Audio data.

[1618] process:

[1619] The device plays the audio data.

[1620] output:

[1621] A spoken response is conveyed to the user.

[1622] Ending the experience and recording the results

[1623] Step 1:

[1624] Specific behavior:

[1625] The user selects to end the experience.

[1626] input:

[1627] Exit selection.

[1628] process:

[1629] The terminal generates an exit command.

[1630] output:

[1631] Exit command.

[1632] Step 2:

[1633] Specific behavior:

[1634] The device sends a notification of the end of the experience to the server.

[1635] input:

[1636] Exit command.

[1637] process:

[1638] Send commands from the terminal to the server.

[1639] output:

[1640] The termination command received by the server.

[1641] Step 3:

[1642] Specific behavior:

[1643] The server collects user session data (learning progress, questions, experience details, etc.) and stores them in a database.

[1644] input:

[1645] Exit command.

[1646] process:

[1647] The server stores the session data in a database (e.g. MySQL).

[1648] output:

[1649] Saved session data.

[1650] Step 4:

[1651] Specific behavior:

[1652] The server ends the current session and invalidates the session ID.

[1653] input:

[1654] Termination notice.

[1655] process:

[1656] The server ends the session and invalidates the session ID.

[1657] output:

[1658] Invalidated session ID and session end notification.

[1659] (Application example 1)

[1660] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1661] Conventional virtual history experience systems are often experienced by individuals in limited spaces such as their own rooms, and are insufficient as a place to share experiences with other visitors. Furthermore, it is difficult to provide an interactive experience that requires users to physically move around, limiting the effectiveness of learning about history and culture. This has reduced opportunities for visitors to learn while enjoying deeper experiences in brick-and-mortar locations such as museums and theme parks.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1663] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and audio data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the user's received audio data, generate an answer using a generation AI, and transmit the audio data to the terminal; means for the terminal to play back the received audio data and communicate it to the user; means for recording the progress and learning content in a database when the virtual historical experience ends; means for the server to end the session; means for providing the physical store with a video display device and audio playback device for the user to experience visually and audibly; and means for the user to experience the virtual historical experience while physically moving around the store. This enables visitors to participate in a virtual historical experience while interactively moving around the physical store.

[1664] "User authentication" is the process by which a user provides individual identifying information, that information is verified, and access rights are granted.

[1665] A "virtual history experience scenario" is a scenario that recreates a specific historical period or scene, allowing users to experience and interact with that period within a virtual environment.

[1666] A "terminal" is a device that a user uses to experience virtual history, and includes smartphones, smart glasses, head-mounted displays, etc.

[1667] A "3D model" is a digital model of an object represented in three-dimensional space using computer graphics.

[1668] "Audio data" refers to data in which audio is recorded in digital format, and is used for audio communication with a user and for reproducing audio information.

[1669] A "server" is a remote computer system that processes requests from user terminals and provides the required data or services.

[1670] "Generative AI" is an artificial intelligence system that performs natural language processing in response to user questions and instructions, generating appropriate answers and information.

[1671] "Progress" is data that records what the user has learned and the goals they have achieved through the virtual history experience.

[1672] A "database" is a system that can efficiently and safely manage large amounts of data and quickly search, update, and delete it.

[1673] A "session" refers to a series of interactions between a user's login and logout, and is a process in which the operations and data exchanges that occur during that time are tracked.

[1674] An "image display device" is a device that allows a user to visually check information or images, and includes smart glasses and head-mounted displays.

[1675] An "audio playback device" is a device for playing back audio data as sound, and includes speakers and headsets.

[1676] A "physical store" is a store or facility where commerce or services are conducted at a physical location.

[1677] "Physically moving around the store" means that the user actually uses their own body to walk around the store.

[1678] MODE FOR CARRYING OUT THE INVENTION

[1679] To implement this invention, it is necessary to build a system that combines specific hardware and software to provide a virtual history experience system in a physical store. The main components of this invention are a server, a terminal, and a user. How each component operates and how the overall system functions are described below.

[1680] Hardware and Software Configuration

[1681] Hardware:

[1682] 1. Devices (head-mounted displays, smart glasses, etc.)

[1683] This is a display device that allows users to visually experience virtual history.

[1684] 2. Audio playback devices (speakers, headsets)

[1685] This is a device that plays back audio data sent from the server and transmits it to the user.

[1686] 3. Microphone

[1687] This is a device that recognizes the user's voice and transmits it to the server.

[1688] 4. Network Connectivity Devices

[1689] It is used to connect terminals and servers over a network.

[1690] software:

[1691] 1. Server

[1692] Data management software: Records user progress and data.

[1693] Generative AI model: Generative AI is used to generate appropriate answers to user questions, for example, OpenAI's GPT-3.

[1694] Database system: Manages users' learning progress and questions.

[1695] 2. Terminal

[1696] Speech recognition software: Converts user speech into text using software such as the Google Speech-to-Text API or AWS Transcribe.

[1697] Rendering engine: Rendering 3D models and virtual spaces using Unity, Unreal Engine, etc.

[1698] Interface software: manages the user-system interface.

[1699] Specific example of system operation

[1700] 1. User authentication and login

[1701] The user enters login information (e.g., username and password) from the terminal. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, a session ID is generated and sent to the terminal.

[1702] 2. Select and start your virtual history experience

[1703] The user selects a virtual historical experience scenario (e.g., "Japan in the Edo Period") from their device. The server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space, allowing the user to experience the historical setting as if they were actually there.

[1704] 3. Interactive History Experience

[1705] Within the virtual space, users can interact with historical figures. The user's voice is recognized through a microphone, converted into text, and sent to the server. The server uses generative AI to generate appropriate answers to the user's questions, converts the answers into audio data, and sends it to the device. The audio data is played back on the device and conveyed to the user.

[1706] 4. Completing the experience and recording the results

[1707] When the user finishes the experience, the device sends a completion notification to the server. The server records the user's progress and learning content in a database and ends the session. This saves the user's learning history and can be used for the next experience.

[1708] Specific prompt examples

[1709] In response to a user question, "What products do you have available today?", the generative AI model is given the following prompt:

[1710] Suppose a user asks a merchant in the Edo period about today's products. User: "What products do you have available today?" The merchant will respond based on the Edo period setting. Merchant: "Today's recommendations are fresh fish and freshly baked dumplings."

[1711] In this way, the brick-and-mortar virtual history experience system can provide users with an interactive and immersive learning experience.

[1712] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1713] Step 1:

[1714] The user enters login information (username and password) using the terminal. The terminal encrypts this information and sends it to the server. The server compares the entered authentication information with the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the received session ID, and the user is notified that login was successful.

[1715] Input: Username, Password

[1716] Data processing: Username and password encryption

[1717] Data calculation: Authentication verification

[1718] Output: Session ID

[1719] Specific actions: Entering data into the login form, encrypting data, generating a token (session ID)

[1720] Step 2:

[1721] The user displays a list of virtual historical experience scenarios provided by the terminal and selects one that interests them. The terminal then sends the selected scenario information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the terminal. The terminal then receives this data and renders the virtual space.

[1722] Input: User selects a scenario

[1723] Data processing: Collecting data corresponding to the scenario

[1724] Data calculation: Sending 3D models and audio data

[1725] Output: Virtual space rendering

[1726] Specific operations: scenario selection, data collection and transmission, construction of a virtual environment using a rendering engine

[1727] Step 3:

[1728] Within the virtual space, users interact with historical figures through voice. The user's voice is picked up by the device's microphone and converted into text using speech recognition software (e.g., Google Speech-to-Text API). The converted text is sent to a server. The server uses generative AI (e.g., OpenAI's GPT-3) to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device then plays the audio data to the user through a playback device.

[1729] Input: User's voice

[1730] Data processing: Speech-to-text (voice recognition)

[1731] Data calculation: Question analysis and answer generation (generative AI)

[1732] Output: Audio data

[1733] Specific operations: Acquiring voice using a microphone, converting it to text, generating answers using AI, and playing back voice data

[1734] Step 4:

[1735] When the user finishes the virtual history experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experiences. The server then ends the current session and invalidates the session ID.

[1736] Input: User's termination notice

[1737] Data processing: Session data aggregation

[1738] Data calculation: saving to database and ending session

[1739] Output: Saved learning progress data

[1740] Specific actions: Sending a completion notice, recording progress in the database, and terminating the session

[1741] In this way, the system of the present invention can provide users with an interactive and immersive virtual history experience.

[1742] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1743] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1744] User Authentication and Login

[1745] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[1746] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[1747] Select and start your virtual history experience

[1748] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[1749] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[1750] Interactive History Experience and Emotion Recognition

[1751] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1752] As a concrete example, when a user asks, "Who built this pyramid?", the device converts the question into text and analyzes the emotion using an emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Amazing, isn't it?" The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1753] Ending the experience and recording the results

[1754] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[1755] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server then saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[1756] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture while taking their emotions into consideration through an interactive virtual historical experience.

[1757] The processing flow will be explained below.

[1758] Step 1:

[1759] The server starts the system. The server loads the necessary modules and databases, and loads the 3D model data, speech generation engine, emotion engine, and historical data into memory.

[1760] Step 2:

[1761] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[1762] Step 3:

[1763] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[1764] Step 4:

[1765] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[1766] Step 5:

[1767] The terminal saves the session ID and notifies the user that they have successfully logged in. The terminal records the session ID received from the server and uses it for subsequent requests.

[1768] Step 6:

[1769] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[1770] Step 7:

[1771] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[1772] Step 8:

[1773] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[1774] Step 9:

[1775] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[1776] Step 10:

[1777] Users interact with historical figures in a virtual space, asking questions and giving instructions by voice.

[1778] Step 11:

[1779] The device recognizes the user's voice and analyzes their emotions using an emotion engine. The device then converts the voice data into text and sends it to the server along with the emotion data.

[1780] Step 12:

[1781] The server uses generation AI to analyze the user's question and generates the optimal answer based on the user's question content and emotional data.

[1782] Step 13:

[1783] The server converts the generated response into voice data using a speech synthesis engine and sends it to the device. The server then reflects emotional data in the response data before sending it.

[1784] Step 14:

[1785] The device then plays back the received voice data and conveys it to the user. The device then plays back the voice, conveying it as if the historical figure were responding in a virtual space according to the user's emotions.

[1786] Step 15:

[1787] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[1788] Step 16:

[1789] The device sends a completion notification to the server. The device notifies the server that the user has completed the experience.

[1790] Step 17:

[1791] The server compiles and stores the user's session data in a database, including learning progress, questions, answers, and emotional data.

[1792] Step 18:

[1793] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[1794] Through these steps, users can engage in an interactive virtual history experience, learning about history and culture while receiving emotionally sensitive responses.

[1795] Example 2

[1796] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1797] Conventional virtual history experience systems often lack appropriate responses and interactions that take into account the user's emotional changes when learning about history and culture. This can reduce the user's learning effectiveness. Furthermore, when the user finishes the experience, the records are incomplete, which means that the experience cannot be fully utilized for the next experience.

[1798] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1799] In this invention, the server includes: means for the user to select a virtual historical experience scenario; means for the terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space, analyzing their emotions, and transmitting the results to the server; means for the server to analyze the received user's voice data and emotion data, generate answers using a generation AI, and transmit the voice data to the terminal; means for recording the progress, learning content, and emotion data in a database when the virtual historical experience ends; and means for the server to end the session and invalidate the session ID. This enables an interactive historical experience that takes the user's emotions into consideration.

[1800] "User" refers to an individual who utilizes the virtual history experience provided by the invention.

[1801] "Virtual history experience scenario" refers to a series of scenarios that allow users to learn about history and culture in a virtual space.

[1802] "Terminal" refers to electronic devices such as computers, tablets, and smartphones that are operated by users.

[1803] "Server" refers to the central computer system that receives, transmits, analyzes, and stores data.

[1804] "Three-dimensional model" refers to a three-dimensional shape or object displayed in a virtual space.

[1805] "Voice data" refers to data used to provide interaction and explanations to the user by voice.

[1806] "Virtual space" refers to a three-dimensional interactive space generated by a computer.

[1807] "Emotion" refers to the psychological state or reaction a user exhibits during a virtual experience.

[1808] "Analyzing emotions" refers to analyzing the user's psychological state and reactions from voice data, etc.

[1809] A "database" refers to a digital information storage system that systematically stores collected information and makes it available for retrieval as needed.

[1810] A "session ID" refers to a unique identifier used to identify a series of activities from when a user logs in to when they log out of a system.

[1811] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers to users' questions.

[1812] "Generating an answer" refers to the generative AI creating an answer based on the user's question.

[1813] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1814] User Authentication and Login

[1815] First, the user enters login information (username and password) from the terminal. The terminal encrypts this information and sends it to the server. The server compares the authentication information with a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID, and the user is notified that login was successful. As a concrete example, when a user enters a "username" and "password," the terminal sends this to the server, and the server compares it with the database for authentication. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[1816] Select and start your virtual history experience

[1817] The user reviews a list of virtual history experience scenarios provided by the device and selects one that interests them. The device sends the selection information to the server. The server collects 3D models, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space. As a specific example, when a user selects "Ancient Egyptian Pyramids," the server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space in which the user can explore the inside of a pyramid through a virtual reality headset or tablet.

[1818] Interactive History Experience and Emotion Recognition

[1819] In a virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then analyzes the user's question using a generative AI model and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user. For example, if a user asks, "Who built this pyramid?" the device converts the question into text and analyzes the emotion using the emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Isn't that amazing?" The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user.

[1820] Ending the experience and recording the results

[1821] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, details of the experience, and emotional data. The server ends the current session and invalidates the session ID. For example, when the user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[1822] Examples of prompt statements

[1823] Example prompt for generative AI model:

[1824] Prompt: "User asks: Who built this pyramid? Emotion: Surprise"

[1825] This system allows users to learn more about history and culture through an interactive virtual historical experience, taking their emotions into account.

[1826] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1827] Step 1:

[1828] The user enters login information (user name and password) from the terminal.

[1829] Input: Username, Password

[1830] Output: Encrypted login information

[1831] Specific behavior:

[1832] The user enters their username and password into the terminal's login screen, which encrypts the information and sends it to the next processing step.

[1833] Step 2:

[1834] The device sends the encrypted login information to the server.

[1835] Input: Encrypted login information

[1836] Output: Sending login information to the server

[1837] Specific behavior:

[1838] The device sends encrypted login information to the server using the HTTPS protocol.

[1839] Step 3:

[1840] The server decrypts the received login information and checks it against the credentials in its database.

[1841] Input: Encrypted login information

[1842] Output: Authentication success or failure

[1843] Specific behavior:

[1844] The server decrypts the encrypted login information, checks it against the credentials stored in the database, and if successful, generates a new session ID.

[1845] Step 4:

[1846] The server sends the generated session ID to the terminal.

[1847] Input: Authentication success result, session ID

[1848] Output: Sending the session ID to the terminal

[1849] Specific behavior:

[1850] The server sends the successful authentication result and the newly generated session ID to the terminal.

[1851] Step 5:

[1852] The terminal saves the received session ID and notifies the user that login was successful.

[1853] Input: Session ID

[1854] Output: Successful login notification

[1855] Specific behavior:

[1856] The terminal stores the received session ID and displays a login success message to the user.

[1857] Step 6:

[1858] The user selects a virtual history experience scenario.

[1859] Input: Select a virtual history experience scenario

[1860] Output: Information about the selected scenario

[1861] Specific behavior:

[1862] The user selects a scenario of interest from a list of multiple virtual history experience scenarios.

[1863] Step 7:

[1864] The terminal transmits the selected scenario information to the server.

[1865] Input: Scenario selection information

[1866] Output: Sending the selected information to the server

[1867] Specific behavior:

[1868] The terminal transmits the scenario information selected by the user to the server.

[1869] Step 8:

[1870] The server collects the three-dimensional model, audio data, and related information corresponding to the selected scenario from a database.

[1871] Input: Scenario selection information

[1872] Output: 3D model, audio data, related information

[1873] Specific behavior:

[1874] The server accesses a database to collect three-dimensional models, audio data, and related information corresponding to the selected scenario.

[1875] Step 9:

[1876] The server sends the collected data to the terminal.

[1877] Input: 3D model, audio data, related information

[1878] Output: Sending data to a terminal

[1879] Specific behavior:

[1880] The server transmits the collected data to the terminal.

[1881] Step 10:

[1882] The terminal uses the received data to render the virtual space and displays it to the user.

[1883] Input: 3D model, audio data, related information

[1884] Output: Rendering of the virtual space

[1885] Specific behavior:

[1886] The device uses the Unity engine to render the received 3D models and audio data, displaying the virtual space to the user.

[1887] Step 11:

[1888] Users interact with historical figures in a virtual space.

[1889] Input: Voice input (questions and instructions)

[1890] Output: Audio data

[1891] Specific behavior:

[1892] Users can ask questions and give instructions to historical figures by voice using a microphone in the virtual space.

[1893] Step 12:

[1894] The device recognizes the voice data, analyzes the emotion using an emotion engine, and sends it to the server.

[1895] Input: Audio data

[1896] Output: Text data, emotion data

[1897] Specific behavior:

[1898] The device converts the voice data into text using a speech recognition engine (e.g., Google Speech-to-Text API), analyzes the emotion using an emotion engine, and then sends the text data and emotion data to the server.

[1899] Step 13:

[1900] The server uses the received text data and emotion data to generate an answer using a generative AI model.

[1901] Input: Text data, emotion data

[1902] Output: Answer text

[1903] Specific behavior:

[1904] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate answer based on the received text data and emotion data.

[1905] Step 14:

[1906] The server converts the generated response into voice data and transmits it to the terminal.

[1907] Input: Answer text

[1908] Output: Audio data

[1909] Specific behavior:

[1910] The server uses a service (e.g., Amazon Polly) to convert the response text into voice data and sends the generated voice data to the device.

[1911] Step 15:

[1912] The terminal plays back the received audio data and conveys it to the user.

[1913] Input: Audio data

[1914] Output: Audio playback

[1915] Specific behavior:

[1916] The terminal plays back the received voice data and conveys the answer to the user.

[1917] Step 16:

[1918] When the user finishes the experience, the device sends a completion notification to the server.

[1919] Input: Termination notification

[1920] Output: Sending notification of completion to the server

[1921] Specific behavior:

[1922] The user performs an end operation (e.g., clicking the "End" button), and the terminal sends an end notification to the server.

[1923] Step 17:

[1924] The server saves the user's session data to a database and ends the session.

[1925] Input: Termination notification, session data

[1926] Output: Save data, end session

[1927] Specific behavior:

[1928] The server stores the user's learning progress, questions, experience details, and emotional data in a database, invalidates the session ID, and ends the session.

[1929] (Application example 2)

[1930] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1931] Conventional virtual history experience systems have struggled to provide an interactive experience that reflects the user's emotions. Furthermore, due to a lack of efficient means for recording and utilizing the user's learning progress and emotions, it has been difficult to provide an optimal learning environment for each individual user. This has led to the issue of users being unable to learn about history and culture in depth.

[1932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1933] In this invention, the server includes: a means for the user to select a virtual history experience scenario; a means for the terminal to receive 3D models and voice data from the server based on the selected scenario; a means for recognizing the user's voice in the virtual space and transmitting it to the server; a means for using a generation AI to generate answers to the user's questions and creating the voice data; a means for recognizing the user's emotions in real time and generating dialogue content using the emotion data; a means for recording the progress and learning content in a database when the virtual history experience ends; and a means for the server to end the session. This enables a more natural interactive virtual history experience that reflects the user's emotions.

[1934] A "user" is an entity that uses the system to experience virtual history.

[1935] A "virtual history experience scenario" is a historical or cultural scenario that a user can select and experience.

[1936] A "terminal" is a device used by a user for inputting and outputting information.

[1937] "3D model" refers to a three-dimensional object or environment displayed in a virtual space.

[1938] "Audio data" is digitized audio data used for recording, playing, transmitting, etc.

[1939] A "server" is a computer system that provides services to terminals over a network.

[1940] "Virtual space" refers to a three-dimensional digital space generated by a computer.

[1941] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate output data or content based on user input data.

[1942] A "session" refers to the period of time during which a user performs a series of operations or activities within a system.

[1943] "Emotion data" refers to data that analyzes and indicates the user's emotional state.

[1944] "Progress" is information that indicates how much learning and experience the user has achieved during the virtual history experience.

[1945] A "database" is a structured collection of data that a system manages and makes available.

[1946] "Dialogue content" refers to the content of questions and responses exchanged between the user and the system.

[1947] "Interactive" refers to the characteristic that operations and experiences progress as the user and the system influence each other.

[1948] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[1949] Hardware and Software Configuration

[1950] Hardware

[1951] Device: Includes the smartphone, head-mounted display (HMD), microphone, and camera used by the user.

[1952] Server: A computer system that provides services to terminals over a network.

[1953] software

[1954] Django: a web application framework

[1955] OpenCV: A library for image processing

[1956] Google Cloud Speech-to-Text API: Speech Recognition

[1957] NVIDIA TensorFlow: A deep learning model for emotion recognition

[1958] Unity: 3D content rendering engine

[1959] DETAILED DESCRIPTION OF THE EMBODIMENTS

[1960] User Authentication and Login

[1961] The user enters login information on the terminal. The terminal encrypts this information and sends it to the server. The server checks the authentication information against a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the session ID, and the user is notified that login was successful.

[1962] Select and start your virtual history experience

[1963] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[1964] Interactive History Experience and Emotion Recognition

[1965] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[1966] Ending the experience and recording the results

[1967] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[1968] (Example of a specific prompt)

[1969] As an example of a prompt sentence, the following can be entered:

[1970] Emotional state: Surprise, User: Who built this pyramid?

[1971] This allows the system to generate an appropriate response that takes into account the user's emotion when the user expresses surprise.

[1972] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1973] Step 1:

[1974] The user enters login information from the terminal. The entered username and password are encrypted and sent to the server. The server checks the authentication information against the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[1975] Input: Username, Password

[1976] Output: Session ID, login success notification

[1977] Step 2:

[1978] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The selection information is sent to the server. The server collects 3D models and audio data corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space.

[1979] Input: Scenario selection information

[1980] Output: 3D models, audio data, rendered virtual worlds

[1981] Step 3:

[1982] Users interact with historical figures in a virtual space. The user's voice is captured by the device's microphone, converted into text, and the emotion engine analyzes the emotions. The analysis results are sent to the server.

[1983] Input: User voice

[1984] Output: Text data, emotion data

[1985] Step 4:

[1986] The server analyzes the received text data and emotional data and generates an appropriate response using a generative AI model. It generates a prompt sentence that takes the emotional data into account and creates a response based on that. The generated response is converted into voice data and sent to the device.

[1987] Input: Text data, emotion data

[1988] Output: Audio data, prompt

[1989] Step 5:

[1990] The device then plays back the received audio data and transmits it to the user, who continues the virtual history experience through voice responses.

[1991] Input: Audio data

[1992] Output: Audio playback

[1993] Step 6:

[1994] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores the learning progress, questions, experience details, and emotional data in a database. The server then ends the current session and invalidates the session ID.

[1995] Input: Termination notification

[1996] Output: Database record, session termination confirmation

[1997] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1998] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1999] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2000] [Fourth embodiment]

[2001] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2002] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2003] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2004] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2005] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2006] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2007] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2008] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2009] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2010] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2011] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2012] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2013] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2014] This invention is a system that allows users to interactively learn about history and culture through a virtual historical experience. Specific processing of each step in the embodiment of the invention will be described below.

[2015] User Authentication and Login

[2016] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[2017] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[2018] Select and start your virtual history experience

[2019] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[2020] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[2021] Interactive History Experience

[2022] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, converts it into text, and sends it to the server. The server uses generative AI to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device plays the audio and communicates it to the user.

[2023] For example, if a user asks, "Who built this pyramid?", the device converts the question into text and sends it to the server. The server uses generative AI to generate the answer, "This pyramid was built by Pharaoh Khufu," and sends it along with the audio data to the device. The device then plays back the audio and relays it to the user.

[2024] Ending the experience and recording the results

[2025] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experience details. The server then ends the current session and invalidates the session ID.

[2026] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress in a database and ends the session. This records the user's learning history and can be used for the next experience.

[2027] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture through an interactive virtual historical experience.

[2028] The processing flow will be explained below.

[2029] Step 1:

[2030] The server starts the system, loads the necessary modules and databases, and loads the 3D model data, speech generation engine, and historical data into memory.

[2031] Step 2:

[2032] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[2033] Step 3:

[2034] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[2035] Step 4:

[2036] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[2037] Step 5:

[2038] The terminal saves the session ID and notifies the user that the login was successful. The terminal remembers the session ID received from the server and uses it for subsequent requests.

[2039] Step 6:

[2040] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[2041] Step 7:

[2042] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[2043] Step 8:

[2044] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[2045] Step 9:

[2046] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[2047] Step 10:

[2048] Users interact with historical figures in a virtual space, asking questions and giving instructions through an interface.

[2049] Step 11:

[2050] The device recognizes the user's voice and sends it to the server, which converts it into text, encrypts it, and sends it to the server.

[2051] Step 12:

[2052] The server uses the generation AI to analyze the user's question and generate an appropriate answer, which is then passed to a speech synthesis engine and converted into voice data.

[2053] Step 13:

[2054] The server transmits the generated voice data to the terminal.

[2055] Step 14:

[2056] The terminal plays back the audio and conveys it to the user. The terminal plays back the received audio data and conveys it to the user as if a historical figure were speaking in the virtual space.

[2057] Step 15:

[2058] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[2059] Step 16:

[2060] The device sends a termination notification to the server. The device notifies the server that the user has terminated the experience.

[2061] Step 17:

[2062] The server compiles and stores user session data in a database, recording learning progress, questions, and experience details.

[2063] Step 18:

[2064] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[2065] Through these steps, users can engage in an interactive virtual historical experience and learn more about history and culture.

[2066] Example 1

[2067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2068] Conventional educational systems have struggled to provide users with a conversational and interactive learning experience when learning about history and culture. They also lacked sophisticated systems that could respond appropriately to users' questions and provide information in real time. Furthermore, they lacked the ability to record learning progress and experiences and use them for future learning, making it difficult to improve users' learning efficiency.

[2069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2070] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the received user voice data, generate an answer using a generative artificial intelligence, and transmit the voice data to the terminal; means for the terminal to play back the received voice data and communicate it to the user; means for recording the learning progress and learning content in a database when the virtual historical experience ends; and means for the server to end the session. This allows users to have a high-quality learning experience in real time through a conversational and interactive virtual historical experience, thereby improving learning efficiency.

[2071] "User" refers to an individual who uses this system to experience virtual history.

[2072] "Virtual history experience scenario" refers to content that allows people to learn about specific historical events or cultures that are recreated in a virtual space.

[2073] "Terminal" refers to the device used by a user to access this system, including computers, tablets, smartphones, VR headsets, etc.

[2074] A "three-dimensional model" refers to a digital representation of three-dimensional objects and environments that a user can visually experience in a virtual space.

[2075] "Audio data" refers to data that stores audio information related to a virtual history experience scenario in digital format.

[2076] "Server" refers to a central component of a computer network that processes requests from user terminals and provides the necessary data.

[2077] "Generative AI" refers to AI technology that analyzes users' questions and generates appropriate answers.

[2078] "Answer" refers to the response content generated in response to a user's question analyzed by the generating artificial intelligence.

[2079] "Speech recognition" refers to the technology of converting a user's speech into digital data and understanding its content as text.

[2080] "Database" refers to an information system for systematically storing and managing information such as learning progress and experience.

[2081] A "session" refers to a period of time during which a user performs a series of operations from the time the user starts using the system until the time the user finishes using the system.

[2082] MODE FOR CARRYING OUT THE INVENTION

[2083] This invention is a system that allows users to learn about history and culture in an interactive manner through a virtual historical experience. The program of this system consists of a series of processes, such as user authentication and login, selection and start of the virtual historical experience, interactive historical experience, end of the experience, and recording of the results.

[2084] User Authentication and Login

[2085] The user enters a username and password from the terminal. The terminal encrypts this information using the AES encryption method and sends it to the server. The server checks the received information against a database (e.g., MySQL) to see if the user exists. If authentication is successful, the server generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[2086] Examples:

[2087] The user enters the username "example_user" and password "example_pass" and presses the "Login" button on the terminal. The terminal encrypts this with AES and sends the "encrypted data" to the server. The server decrypts the received data and checks it against the MySQL database. If authentication is successful, the server generates a session ID "example_session_id" and sends it to the terminal. The terminal records the session ID and displays "Login successful" to the user.

[2088] Select and start your virtual history experience

[2089] The user checks a list of virtual history experience scenarios provided on the device. When the user selects a scenario of interest, the device sends the selection information to the server. The server collects a 3D model (e.g., Unity engine), audio data (e.g., MP3 file), and related text information corresponding to the selected scenario and sends them to the device. The device uses the received data to render the virtual space.

[2090] Examples:

[2091] The user opens the "Scenario Selection" screen on their device and selects "Ancient Egyptian Pyramids." When they press the "Select" button, the device sends "Scenario ID: egypt_pyramid" to the server. The server uses the Unity 3D engine to collect a three-dimensional model of the pyramid, related MP3 audio data, and other necessary information, and sends it to the device. The device receives this and renders it through a VR headset, allowing the user to explore the virtual pyramid.

[2092] Interactive History Experience

[2093] In a virtual space, users interact with historical figures. They use a microphone to ask questions or give instructions. The device converts the audio input from the microphone into text using speech recognition software (e.g., Google Cloud Speech-to-Text) and sends it to a server. The server uses generative AI (e.g., OpenAI GPT-3) to analyze the user's question and generate an appropriate answer. The answer is converted into audio data using speech synthesis software (e.g., Amazon Polly) and sent to the device. The device plays the audio data and communicates it to the user.

[2094] Examples:

[2095] The user asks in the virtual space, "Who built this pyramid?" The device converts the speech to text using Google Cloud Speech-to-Text and sends the text "Who built this pyramid?" to the server. The server analyzes the question using OpenAI GPT-3 and generates the answer "This pyramid was built by Pharaoh Khufu." The answer is then converted into audio data using Amazon Polly and sent to the device. The device plays the audio data and communicates it to the user.

[2096] Ending the experience and recording the results

[2097] When the user finishes the experience, the device notifies the server. The server saves the user's session data (learning progress, questions, experience details, etc.) in a database. After saving, the server ends the current session and invalidates the session ID.

[2098] Examples:

[2099] When the user presses the "End" button on the device, the device sends "User ID: example_user, Session ID: example_session_id, Status: Ended" to the server. The server compiles the session data (learning progress, question content, experience details) and saves it in a MySQL database. Once saved, the server ends the session and invalidates the session ID. The user is then told "Experience has ended."

[2100] Prompt Sentence Examples

[2101] Below is a concrete example of a prompt sentence to be input to the generative AI model. The purpose is to generate answers to questions posed by users during a virtual history experience.

[2102] Prompt Sentence Examples

[2103] User Question: "Who built this pyramid?"

[2104] Context: "A user asked this question while exploring an ancient Egyptian pyramid. This pyramid was built by Pharaoh Khufu."

[2105] Expected answer: "This pyramid was built by Khufu."

[2106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2107] User Authentication and Login

[2108] Step 1:

[2109] Specific behavior:

[2110] The user enters the user name and password from the terminal.

[2111] input:

[2112] Username and password.

[2113] process:

[2114] The terminal encrypts the input information using the AES encryption method.

[2115] output:

[2116] Encrypted authentication data.

[2117] Step 2:

[2118] Specific behavior:

[2119] The terminal transmits the encrypted authentication data to the server.

[2120] input:

[2121] Encrypted authentication data.

[2122] process:

[2123] Send data from the terminal to the server.

[2124] output:

[2125] The authentication data received by the server.

[2126] Step 3:

[2127] Specific behavior:

[2128] The server checks the received authentication data against a database.

[2129] input:

[2130] Authentication data.

[2131] process:

[2132] The server checks if the user exists by checking against a database (e.g. MySQL).

[2133] output:

[2134] The authentication result (success or failure).

[2135] Step 4:

[2136] Specific behavior:

[2137] If authentication is successful, the server generates a new session ID.

[2138] input:

[2139] Authentication result (success).

[2140] process:

[2141] The server generates a session ID.

[2142] output:

[2143] The generated session ID.

[2144] Step 5:

[2145] Specific behavior:

[2146] The server sends the generated session ID to the terminal.

[2147] input:

[2148] Session ID.

[2149] process:

[2150] The server sends a session ID to the terminal.

[2151] output:

[2152] The session ID received by the device.

[2153] Step 6:

[2154] Specific behavior:

[2155] The terminal saves the session ID and notifies the user that the login was successful.

[2156] input:

[2157] Session ID.

[2158] process:

[2159] The terminal will save the session ID and display a successful login message on the screen.

[2160] output:

[2161] Stored session ID and successful login notification.

[2162] Select and start your virtual history experience

[2163] Step 1:

[2164] Specific behavior:

[2165] The user checks the list of virtual history experience scenarios provided on the terminal.

[2166] input:

[2167] View the user's scenario list.

[2168] process:

[2169] The device displays a list of scenarios.

[2170] output:

[2171] Scenario list.

[2172] Step 2:

[2173] Specific behavior:

[2174] The user selects the scenario that interests them.

[2175] input:

[2176] Select a scenario.

[2177] process:

[2178] The terminal stores the selection information.

[2179] output:

[2180] Selected scenario information.

[2181] Step 3:

[2182] Specific behavior:

[2183] The terminal transmits the selection information to the server.

[2184] input:

[2185] Selected scenario information.

[2186] process:

[2187] Send data from the terminal to the server.

[2188] output:

[2189] Scenario information received on the server.

[2190] Step 4:

[2191] Specific behavior:

[2192] The server collects the three-dimensional model, audio data, and associated text information corresponding to the selected scenario.

[2193] input:

[2194] Selected scenario information.

[2195] process:

[2196] The server collects 3D models (e.g., Unity engine), audio data (e.g., MP3 files), and text information.

[2197] output:

[2198] Data collected.

[2199] Step 5:

[2200] Specific behavior:

[2201] The server transmits the collected data to the terminal.

[2202] input:

[2203] Data collected.

[2204] process:

[2205] Data is sent from the server to the terminal.

[2206] output:

[2207] Three-dimensional models, audio data, and text information arrive at the terminal.

[2208] Step 6:

[2209] Specific behavior:

[2210] The device uses the received data to render the virtual space.

[2211] input:

[2212] Three-dimensional models, audio data, and text information.

[2213] process:

[2214] The device renders the virtual space based on the received data.

[2215] output:

[2216] Rendered virtual space.

[2217] Interactive History Experience

[2218] Step 1:

[2219] Specific behavior:

[2220] Users can ask questions and give instructions through a microphone in the virtual space.

[2221] input:

[2222] User voice.

[2223] process:

[2224] The device records the audio.

[2225] output:

[2226] Recorded audio data.

[2227] Step 2:

[2228] Specific behavior:

[2229] The device converts the voice into text using speech recognition software.

[2230] input:

[2231] Audio data.

[2232] process:

[2233] The device converts speech to text using Google Cloud Speech-to-Text.

[2234] output:

[2235] Text data.

[2236] Step 3:

[2237] Specific behavior:

[2238] The terminal transmits the converted text data to the server.

[2239] input:

[2240] Text data.

[2241] process:

[2242] Send data from the terminal to the server.

[2243] output:

[2244] Text data received by the server.

[2245] Step 4:

[2246] Specific behavior:

[2247] The server uses generative AI to analyze the text data and generate appropriate answers.

[2248] input:

[2249] Text data.

[2250] process:

[2251] The server uses generative AI (e.g., OpenAI GPT-3) to analyze the question and generate an answer.

[2252] output:

[2253] The generated answer text.

[2254] Step 5:

[2255] Specific behavior:

[2256] The server converts the generated response into voice data using voice synthesis software.

[2257] input:

[2258] Answer text.

[2259] process:

[2260] The server uses Amazon Polly to convert the response into audio data.

[2261] output:

[2262] Audio data.

[2263] Step 6:

[2264] Specific behavior:

[2265] The server transmits the generated voice data to the terminal.

[2266] input:

[2267] Audio data.

[2268] process:

[2269] Data is sent from the server to the terminal.

[2270] output:

[2271] Audio data received at the device.

[2272] Step 7:

[2273] Specific behavior:

[2274] The terminal reproduces the audio data and conveys it to the user.

[2275] input:

[2276] Audio data.

[2277] process:

[2278] The device plays the audio data.

[2279] output:

[2280] A spoken response is conveyed to the user.

[2281] Ending the experience and recording the results

[2282] Step 1:

[2283] Specific behavior:

[2284] The user selects to end the experience.

[2285] input:

[2286] Exit selection.

[2287] process:

[2288] The terminal generates an exit command.

[2289] output:

[2290] Exit command.

[2291] Step 2:

[2292] Specific behavior:

[2293] The device sends a notification of the end of the experience to the server.

[2294] input:

[2295] Exit command.

[2296] process:

[2297] Send commands from the terminal to the server.

[2298] output:

[2299] The termination command received by the server.

[2300] Step 3:

[2301] Specific behavior:

[2302] The server collects user session data (learning progress, questions, experience details, etc.) and stores them in a database.

[2303] input:

[2304] Exit command.

[2305] process:

[2306] The server stores the session data in a database (e.g. MySQL).

[2307] output:

[2308] Saved session data.

[2309] Step 4:

[2310] Specific behavior:

[2311] The server ends the current session and invalidates the session ID.

[2312] input:

[2313] Termination notice.

[2314] process:

[2315] The server ends the session and invalidates the session ID.

[2316] output:

[2317] Invalidated session ID and session end notification.

[2318] (Application example 1)

[2319] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2320] Conventional virtual history experience systems are often experienced by individuals in limited spaces such as their own rooms, and are insufficient as a place to share experiences with other visitors. Furthermore, it is difficult to provide an interactive experience that requires users to physically move around, limiting the effectiveness of learning about history and culture. This has reduced opportunities for visitors to learn while enjoying deeper experiences in brick-and-mortar locations such as museums and theme parks.

[2321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2322] In this invention, the server includes: means for a user to select a virtual historical experience scenario; means for a terminal to receive from the server a three-dimensional model and audio data based on the selected scenario; means for recognizing the user's voice in the virtual space and transmitting it to the server; means for the server to analyze the user's received audio data, generate an answer using a generation AI, and transmit the audio data to the terminal; means for the terminal to play back the received audio data and communicate it to the user; means for recording the progress and learning content in a database when the virtual historical experience ends; means for the server to end the session; means for providing the physical store with a video display device and audio playback device for the user to experience visually and audibly; and means for the user to experience the virtual historical experience while physically moving around the store. This enables visitors to participate in a virtual historical experience while interactively moving around the physical store.

[2323] "User authentication" is the process by which a user provides individual identifying information, that information is verified, and access rights are granted.

[2324] A "virtual history experience scenario" is a scenario that recreates a specific historical period or scene, allowing users to experience and interact with that period within a virtual environment.

[2325] A "terminal" is a device that a user uses to experience virtual history, and includes smartphones, smart glasses, head-mounted displays, etc.

[2326] A "3D model" is a digital model of an object represented in three-dimensional space using computer graphics.

[2327] "Audio data" refers to data in which audio is recorded in digital format, and is used for audio communication with a user and for reproducing audio information.

[2328] A "server" is a remote computer system that processes requests from user terminals and provides the required data or services.

[2329] "Generative AI" is an artificial intelligence system that performs natural language processing in response to user questions and instructions, generating appropriate answers and information.

[2330] "Progress" is data that records what the user has learned and the goals they have achieved through the virtual history experience.

[2331] A "database" is a system that can efficiently and safely manage large amounts of data and quickly search, update, and delete it.

[2332] A "session" refers to a series of interactions between a user's login and logout, and is a process in which the operations and data exchanges that occur during that time are tracked.

[2333] An "image display device" is a device that allows a user to visually check information or images, and includes smart glasses and head-mounted displays.

[2334] An "audio playback device" is a device for playing back audio data as sound, and includes speakers and headsets.

[2335] A "physical store" is a store or facility where commerce or services are conducted at a physical location.

[2336] "Physically moving around the store" means that the user actually uses their own body to walk around the store.

[2337] MODE FOR CARRYING OUT THE INVENTION

[2338] To implement this invention, it is necessary to build a system that combines specific hardware and software to provide a virtual history experience system in a physical store. The main components of this invention are a server, a terminal, and a user. How each component operates and how the overall system functions are described below.

[2339] Hardware and Software Configuration

[2340] Hardware:

[2341] 1. Devices (head-mounted displays, smart glasses, etc.)

[2342] This is a display device that allows users to visually experience virtual history.

[2343] 2. Audio playback devices (speakers, headsets)

[2344] This is a device that plays back audio data sent from the server and transmits it to the user.

[2345] 3. Microphone

[2346] This is a device that recognizes the user's voice and transmits it to the server.

[2347] 4. Network Connectivity Devices

[2348] It is used to connect terminals and servers over a network.

[2349] software:

[2350] 1. Server

[2351] Data management software: Records user progress and data.

[2352] Generative AI model: Generative AI is used to generate appropriate answers to user questions, for example, OpenAI's GPT-3.

[2353] Database system: Manages users' learning progress and questions.

[2354] 2. Terminal

[2355] Speech recognition software: Converts user speech into text using software such as the Google Speech-to-Text API or AWS Transcribe.

[2356] Rendering engine: Rendering 3D models and virtual spaces using Unity, Unreal Engine, etc.

[2357] Interface software: manages the user-system interface.

[2358] Specific example of system operation

[2359] 1. User authentication and login

[2360] The user enters login information (e.g., username and password) from the terminal. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, a session ID is generated and sent to the terminal.

[2361] 2. Select and start your virtual history experience

[2362] The user selects a virtual historical experience scenario (e.g., "Japan in the Edo Period") from their device. The server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space, allowing the user to experience the historical setting as if they were actually there.

[2363] 3. Interactive History Experience

[2364] Within the virtual space, users can interact with historical figures. The user's voice is recognized through a microphone, converted into text, and sent to the server. The server uses generative AI to generate appropriate answers to the user's questions, converts the answers into audio data, and sends it to the device. The audio data is played back on the device and conveyed to the user.

[2365] 4. Completing the experience and recording the results

[2366] When the user finishes the experience, the device sends a completion notification to the server. The server records the user's progress and learning content in a database and ends the session. This saves the user's learning history and can be used for the next experience.

[2367] Specific prompt examples

[2368] In response to a user question, "What products do you have available today?", the generative AI model is given the following prompt:

[2369] Suppose a user asks a merchant in the Edo period about today's products. User: "What products do you have available today?" The merchant will respond based on the Edo period setting. Merchant: "Today's recommendations are fresh fish and freshly baked dumplings."

[2370] In this way, the brick-and-mortar virtual history experience system can provide users with an interactive and immersive learning experience.

[2371] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2372] Step 1:

[2373] The user enters login information (username and password) using the terminal. The terminal encrypts this information and sends it to the server. The server compares the entered authentication information with the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the received session ID, and the user is notified that login was successful.

[2374] Input: Username, Password

[2375] Data processing: Username and password encryption

[2376] Data calculation: Authentication verification

[2377] Output: Session ID

[2378] Specific actions: Entering data into the login form, encrypting data, generating a token (session ID)

[2379] Step 2:

[2380] The user displays a list of virtual historical experience scenarios provided by the terminal and selects one that interests them. The terminal then sends the selected scenario information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the terminal. The terminal then receives this data and renders the virtual space.

[2381] Input: User selects a scenario

[2382] Data processing: Collecting data corresponding to the scenario

[2383] Data calculation: Sending 3D models and audio data

[2384] Output: Virtual space rendering

[2385] Specific operations: scenario selection, data collection and transmission, construction of a virtual environment using a rendering engine

[2386] Step 3:

[2387] Within the virtual space, users interact with historical figures through voice. The user's voice is picked up by the device's microphone and converted into text using speech recognition software (e.g., Google Speech-to-Text API). The converted text is sent to a server. The server uses generative AI (e.g., OpenAI's GPT-3) to analyze the user's question and generate an appropriate answer. The generated answer is converted into audio data and sent to the device. The device then plays the audio data to the user through a playback device.

[2388] Input: User's voice

[2389] Data processing: Speech-to-text (voice recognition)

[2390] Data calculation: Question analysis and answer generation (generative AI)

[2391] Output: Audio data

[2392] Specific operations: Acquiring voice using a microphone, converting it to text, generating answers using AI, and playing back voice data

[2393] Step 4:

[2394] When the user finishes the virtual history experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, and experiences. The server then ends the current session and invalidates the session ID.

[2395] Input: User's termination notice

[2396] Data processing: Session data aggregation

[2397] Data calculation: saving to database and ending session

[2398] Output: Saved learning progress data

[2399] Specific actions: Sending a completion notice, recording progress in the database, and terminating the session

[2400] In this way, the system of the present invention can provide users with an interactive and immersive virtual history experience.

[2401] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2402] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[2403] User Authentication and Login

[2404] First, the user enters their login information on the device. The device encrypts this information and sends it to the server. The server compares the authentication information with the database, and if authentication is successful, it generates a new session ID and sends it to the device. The device saves the session ID, and the user is notified that login was successful.

[2405] For example, when a user enters a "user name" and "password," the terminal sends this to the server, which then verifies it against a database. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[2406] Select and start your virtual history experience

[2407] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[2408] For example, if a user selects "Ancient Egyptian Pyramids," the server collects and transmits related 3D models and audio data to the device, which uses this data to render a virtual space that allows the user to explore the inside of a pyramid through a VR headset or tablet.

[2409] Interactive History Experience and Emotion Recognition

[2410] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[2411] As a concrete example, when a user asks, "Who built this pyramid?", the device converts the question into text and analyzes the emotion using an emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Amazing, isn't it?" The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[2412] Ending the experience and recording the results

[2413] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[2414] For example, when a user presses the "Exit" button, the device sends an end notification to the server. The server then saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[2415] As described above, the present invention can be implemented as a system that allows users to learn more deeply about history and culture while taking their emotions into consideration through an interactive virtual historical experience.

[2416] The processing flow will be explained below.

[2417] Step 1:

[2418] The server starts the system. The server loads the necessary modules and databases, and loads the 3D model data, speech generation engine, emotion engine, and historical data into memory.

[2419] Step 2:

[2420] The user enters login information on the terminal. The user enters the username and password and presses the send button.

[2421] Step 3:

[2422] The terminal sends the authentication information to the server. The terminal encrypts the entered authentication information and sends it to the server.

[2423] Step 4:

[2424] The server checks the authentication information against its database. If the authentication is successful, it generates a new session ID and sends it to the device.

[2425] Step 5:

[2426] The terminal saves the session ID and notifies the user that they have successfully logged in. The terminal records the session ID received from the server and uses it for subsequent requests.

[2427] Step 6:

[2428] The user selects a virtual history experience scenario. The user checks the list of scenarios provided on the terminal interface and selects the desired scenario.

[2429] Step 7:

[2430] The terminal transmits the selection information to the server. The terminal encrypts the user's selection information and transmits it to the server.

[2431] Step 8:

[2432] The server prepares the 3D model and audio data corresponding to the scenario and sends them to the terminal. The server collects the necessary data and sends it in packet format to the terminal.

[2433] Step 9:

[2434] The device uses the received data to render the virtual space. The device uses the 3D model and audio data to recreate the virtual space for the user.

[2435] Step 10:

[2436] Users interact with historical figures in a virtual space, asking questions and giving instructions by voice.

[2437] Step 11:

[2438] The device recognizes the user's voice and analyzes their emotions using an emotion engine. The device then converts the voice data into text and sends it to the server along with the emotion data.

[2439] Step 12:

[2440] The server uses generation AI to analyze the user's question and generates the optimal answer based on the user's question content and emotional data.

[2441] Step 13:

[2442] The server converts the generated response into voice data using a speech synthesis engine and sends it to the device. The server then reflects emotional data in the response data before sending it.

[2443] Step 14:

[2444] The device then plays back the received voice data and conveys it to the user. The device then plays back the voice, conveying it as if the historical figure were responding in a virtual space according to the user's emotions.

[2445] Step 15:

[2446] The user ends the experience. The user presses the "Exit" button in the virtual space or on the device interface.

[2447] Step 16:

[2448] The device sends a completion notification to the server. The device notifies the server that the user has completed the experience.

[2449] Step 17:

[2450] The server compiles and stores the user's session data in a database, including learning progress, questions, answers, and emotional data.

[2451] Step 18:

[2452] The server ends the current session and invalidates the session ID. The server saves the session data and then ends the session.

[2453] Through these steps, users can engage in an interactive virtual history experience, learning about history and culture while receiving emotionally sensitive responses.

[2454] Example 2

[2455] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2456] Conventional virtual history experience systems often lack appropriate responses and interactions that take into account the user's emotional changes when learning about history and culture. This can reduce the user's learning effectiveness. Furthermore, when the user finishes the experience, the records are incomplete, which means that the experience cannot be fully utilized for the next experience.

[2457] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2458] In this invention, the server includes: means for the user to select a virtual historical experience scenario; means for the terminal to receive from the server a three-dimensional model and voice data based on the selected scenario; means for recognizing the user's voice in the virtual space, analyzing their emotions, and transmitting the results to the server; means for the server to analyze the received user's voice data and emotion data, generate answers using a generation AI, and transmit the voice data to the terminal; means for recording the progress, learning content, and emotion data in a database when the virtual historical experience ends; and means for the server to end the session and invalidate the session ID. This enables an interactive historical experience that takes the user's emotions into consideration.

[2459] "User" refers to an individual who utilizes the virtual history experience provided by the invention.

[2460] "Virtual history experience scenario" refers to a series of scenarios that allow users to learn about history and culture in a virtual space.

[2461] "Terminal" refers to electronic devices such as computers, tablets, and smartphones that are operated by users.

[2462] "Server" refers to the central computer system that receives, transmits, analyzes, and stores data.

[2463] "Three-dimensional model" refers to a three-dimensional shape or object displayed in a virtual space.

[2464] "Voice data" refers to data used to provide interaction and explanations to the user by voice.

[2465] "Virtual space" refers to a three-dimensional interactive space generated by a computer.

[2466] "Emotion" refers to the psychological state or reaction a user exhibits during a virtual experience.

[2467] "Analyzing emotions" refers to analyzing the user's psychological state and reactions from voice data, etc.

[2468] A "database" refers to a digital information storage system that systematically stores collected information and makes it available for retrieval as needed.

[2469] A "session ID" refers to a unique identifier used to identify a series of activities from when a user logs in to when they log out of a system.

[2470] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers to users' questions.

[2471] "Generating an answer" refers to the generative AI creating an answer based on the user's question.

[2472] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[2473] User Authentication and Login

[2474] First, the user enters login information (username and password) from the terminal. The terminal encrypts this information and sends it to the server. The server compares the authentication information with a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID, and the user is notified that login was successful. As a concrete example, when a user enters a "username" and "password," the terminal sends this to the server, and the server compares it with the database for authentication. If authentication is successful, a session ID is generated and sent to the terminal. The user maintains their logged-in state using this session ID.

[2475] Select and start your virtual history experience

[2476] The user reviews a list of virtual history experience scenarios provided by the device and selects one that interests them. The device sends the selection information to the server. The server collects 3D models, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device receives this data and renders the virtual space. As a specific example, when a user selects "Ancient Egyptian Pyramids," the server collects related 3D models and audio data and sends them to the device. The device uses this data to render a virtual space in which the user can explore the inside of a pyramid through a virtual reality headset or tablet.

[2477] Interactive History Experience and Emotion Recognition

[2478] In a virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then analyzes the user's question using a generative AI model and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user. For example, if a user asks, "Who built this pyramid?" the device converts the question into text and analyzes the emotion using the emotion engine. For example, if the user expresses surprise, the server generates an answer such as, "This pyramid was built by Pharaoh Khufu. Isn't that amazing?" The generated answer is converted into audio data and sent to the device. The device plays the audio and conveys it to the user.

[2479] Ending the experience and recording the results

[2480] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, details of the experience, and emotional data. The server ends the current session and invalidates the session ID. For example, when the user presses the "Exit" button, the device sends an end notification to the server. The server saves the user's learning progress and emotional data in a database and ends the session. This records the user's learning history and emotional fluctuations, which can be used for the next experience.

[2481] Examples of prompt statements

[2482] Example prompt for generative AI model:

[2483] Prompt: "User asks: Who built this pyramid? Emotion: Surprise"

[2484] This system allows users to learn more about history and culture through an interactive virtual historical experience, taking their emotions into account.

[2485] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2486] Step 1:

[2487] The user enters login information (user name and password) from the terminal.

[2488] Input: Username, Password

[2489] Output: Encrypted login information

[2490] Specific behavior:

[2491] The user enters their username and password into the terminal's login screen, which encrypts the information and sends it to the next processing step.

[2492] Step 2:

[2493] The device sends the encrypted login information to the server.

[2494] Input: Encrypted login information

[2495] Output: Sending login information to the server

[2496] Specific behavior:

[2497] The device sends encrypted login information to the server using the HTTPS protocol.

[2498] Step 3:

[2499] The server decrypts the received login information and checks it against the credentials in its database.

[2500] Input: Encrypted login information

[2501] Output: Authentication success or failure

[2502] Specific behavior:

[2503] The server decrypts the encrypted login information, checks it against the credentials stored in the database, and if successful, generates a new session ID.

[2504] Step 4:

[2505] The server sends the generated session ID to the terminal.

[2506] Input: Authentication success result, session ID

[2507] Output: Sending the session ID to the terminal

[2508] Specific behavior:

[2509] The server sends the successful authentication result and the newly generated session ID to the terminal.

[2510] Step 5:

[2511] The terminal saves the received session ID and notifies the user that login was successful.

[2512] Input: Session ID

[2513] Output: Successful login notification

[2514] Specific behavior:

[2515] The terminal stores the received session ID and displays a login success message to the user.

[2516] Step 6:

[2517] The user selects a virtual history experience scenario.

[2518] Input: Select a virtual history experience scenario

[2519] Output: Information about the selected scenario

[2520] Specific behavior:

[2521] The user selects a scenario of interest from a list of multiple virtual history experience scenarios.

[2522] Step 7:

[2523] The terminal transmits the selected scenario information to the server.

[2524] Input: Scenario selection information

[2525] Output: Sending the selected information to the server

[2526] Specific behavior:

[2527] The terminal transmits the scenario information selected by the user to the server.

[2528] Step 8:

[2529] The server collects the three-dimensional model, audio data, and related information corresponding to the selected scenario from a database.

[2530] Input: Scenario selection information

[2531] Output: 3D model, audio data, related information

[2532] Specific behavior:

[2533] The server accesses a database to collect three-dimensional models, audio data, and related information corresponding to the selected scenario.

[2534] Step 9:

[2535] The server sends the collected data to the terminal.

[2536] Input: 3D model, audio data, related information

[2537] Output: Sending data to a terminal

[2538] Specific behavior:

[2539] The server transmits the collected data to the terminal.

[2540] Step 10:

[2541] The terminal uses the received data to render the virtual space and displays it to the user.

[2542] Input: 3D model, audio data, related information

[2543] Output: Rendering of the virtual space

[2544] Specific behavior:

[2545] The device uses the Unity engine to render the received 3D models and audio data, displaying the virtual space to the user.

[2546] Step 11:

[2547] Users interact with historical figures in a virtual space.

[2548] Input: Voice input (questions and instructions)

[2549] Output: Audio data

[2550] Specific behavior:

[2551] Users can ask questions and give instructions to historical figures by voice using a microphone in the virtual space.

[2552] Step 12:

[2553] The device recognizes the voice data, analyzes the emotion using an emotion engine, and sends it to the server.

[2554] Input: Audio data

[2555] Output: Text data, emotion data

[2556] Specific behavior:

[2557] The device converts the voice data into text using a speech recognition engine (e.g., Google Speech-to-Text API), analyzes the emotion using an emotion engine, and then sends the text data and emotion data to the server.

[2558] Step 13:

[2559] The server uses the received text data and emotion data to generate an answer using a generative AI model.

[2560] Input: Text data, emotion data

[2561] Output: Answer text

[2562] Specific behavior:

[2563] The server uses a generative AI model (e.g., GPT-3) to generate an appropriate answer based on the received text data and emotion data.

[2564] Step 14:

[2565] The server converts the generated response into voice data and transmits it to the terminal.

[2566] Input: Answer text

[2567] Output: Audio data

[2568] Specific behavior:

[2569] The server uses a service (e.g., Amazon Polly) to convert the response text into voice data and sends the generated voice data to the device.

[2570] Step 15:

[2571] The terminal plays back the received audio data and conveys it to the user.

[2572] Input: Audio data

[2573] Output: Audio playback

[2574] Specific behavior:

[2575] The terminal plays back the received voice data and conveys the answer to the user.

[2576] Step 16:

[2577] When the user finishes the experience, the device sends a completion notification to the server.

[2578] Input: Termination notification

[2579] Output: Sending notification of completion to the server

[2580] Specific behavior:

[2581] The user performs an end operation (e.g., clicking the "End" button), and the terminal sends an end notification to the server.

[2582] Step 17:

[2583] The server saves the user's session data to a database and ends the session.

[2584] Input: Termination notification, session data

[2585] Output: Save data, end session

[2586] Specific behavior:

[2587] The server stores the user's learning progress, questions, experience details, and emotional data in a database, invalidates the session ID, and ends the session.

[2588] (Application example 2)

[2589] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2590] Conventional virtual history experience systems have struggled to provide an interactive experience that reflects the user's emotions. Furthermore, due to a lack of efficient means for recording and utilizing the user's learning progress and emotions, it has been difficult to provide an optimal learning environment for each individual user. This has led to the issue of users being unable to learn about history and culture in depth.

[2591] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2592] In this invention, the server includes: a means for the user to select a virtual history experience scenario; a means for the terminal to receive 3D models and voice data from the server based on the selected scenario; a means for recognizing the user's voice in the virtual space and transmitting it to the server; a means for using a generation AI to generate answers to the user's questions and creating the voice data; a means for recognizing the user's emotions in real time and generating dialogue content using the emotion data; a means for recording the progress and learning content in a database when the virtual history experience ends; and a means for the server to end the session. This enables a more natural interactive virtual history experience that reflects the user's emotions.

[2593] A "user" is an entity that uses the system to experience virtual history.

[2594] A "virtual history experience scenario" is a historical or cultural scenario that a user can select and experience.

[2595] A "terminal" is a device used by a user for inputting and outputting information.

[2596] "3D model" refers to a three-dimensional object or environment displayed in a virtual space.

[2597] "Audio data" is digitized audio data used for recording, playing, transmitting, etc.

[2598] A "server" is a computer system that provides services to terminals over a network.

[2599] "Virtual space" refers to a three-dimensional digital space generated by a computer.

[2600] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate output data or content based on user input data.

[2601] A "session" refers to the period of time during which a user performs a series of operations or activities within a system.

[2602] "Emotion data" refers to data that analyzes and indicates the user's emotional state.

[2603] "Progress" is information that indicates how much learning and experience the user has achieved during the virtual history experience.

[2604] A "database" is a structured collection of data that a system manages and makes available.

[2605] "Dialogue content" refers to the content of questions and responses exchanged between the user and the system.

[2606] "Interactive" refers to the characteristic that operations and experiences progress as the user and the system influence each other.

[2607] This invention combines a system that allows users to interactively learn about history and culture through a virtual historical experience with an emotion engine that recognizes the user's emotions. Specific processing for each step of the embodiment of the invention will be described below.

[2608] Hardware and Software Configuration

[2609] Hardware

[2610] Device: Includes the smartphone, head-mounted display (HMD), microphone, and camera used by the user.

[2611] Server: A computer system that provides services to terminals over a network.

[2612] software

[2613] Django: a web application framework

[2614] OpenCV: A library for image processing

[2615] Google Cloud Speech-to-Text API: Speech Recognition

[2616] NVIDIA TensorFlow: A deep learning model for emotion recognition

[2617] Unity: 3D content rendering engine

[2618] DETAILED DESCRIPTION OF THE EMBODIMENTS

[2619] User Authentication and Login

[2620] The user enters login information on the terminal. The terminal encrypts this information and sends it to the server. The server checks the authentication information against a database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal stores the session ID, and the user is notified that login was successful.

[2621] Select and start your virtual history experience

[2622] The user checks a list of virtual history experience scenarios provided by the device and selects one that interests them. The device then sends the selection information to the server. The server then collects the 3D model, audio data, and related information corresponding to the selected scenario and sends this data to the device. The device then receives this data and renders the virtual space.

[2623] Interactive History Experience and Emotion Recognition

[2624] Within the virtual space, users interact with historical figures. They ask questions and give instructions through an interface. The device recognizes the user's voice, analyzes their emotions using an emotion engine, converts the text into text, and sends it to the server. The server then uses generative AI to analyze the user's question and generates an appropriate answer. The answer is adjusted taking into account the user's emotions recognized by the emotion engine. The generated answer is converted into audio data and sent to the device. The device then plays the audio and communicates it to the user.

[2625] Ending the experience and recording the results

[2626] When the user finishes the experience, the device notifies the server. The server compiles the user's session data and stores it in a database. The stored data includes the user's learning progress, questions, experience details, and emotional data. The server then ends the current session and invalidates the session ID.

[2627] (Example of a specific prompt)

[2628] As an example of a prompt sentence, the following can be entered:

[2629] Emotional state: Surprise, User: Who built this pyramid?

[2630] This allows the system to generate an appropriate response that takes into account the user's emotion when the user expresses surprise.

[2631] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2632] Step 1:

[2633] The user enters login information from the terminal. The entered username and password are encrypted and sent to the server. The server checks the authentication information against the database, and if authentication is successful, generates a new session ID and sends it to the terminal. The terminal saves the session ID and notifies the user that login was successful.

[2634] Input: Username, Password

[2635] Output: Session ID, login success notification

[2636] Step 2:

[2637] The user checks a list of virtual history experience scenarios provided by the devic...

Claims

1. A means for a user to select a virtual history experience scenario; means for receiving a 3D model and audio data from a server based on a scenario selected by the terminal; means for recognizing a user's voice in the virtual space and transmitting the voice to a server; A means for the server to analyze the received user voice data, generate an answer using the generation AI, and transmit the voice data to the terminal; means for playing back the audio data received by the terminal and transmitting it to the user; a means of recording progress and learning in a database upon completion of the virtual history experience; a means by which the server terminates the session; A system including:

2. 2. The system according to claim 1, wherein a generation AI is used to generate answers to user questions and create the voice data.

3. 10. The system of claim 1, further comprising means for enabling a user to interact with historical figures within the virtual space.

Images