System

Abstract

A system is provided.SOLUTION: A system for allowing a user to experience a virtual travel through a virtual reality device, the system comprising: virtual reality device means for tracking the user's location and movements; terminal means for loading and displaying virtual reality data based on a travel destination selection; artificial intelligence means for responding in real-time to user questions; and feedback device means for providing haptic feedback, such as wind and vibration, to the user's virtual travel experience.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many people find it difficult to travel casually due to illness, disability, advanced age, work, childcare, or financial reasons. Furthermore, physical and time constraints limit opportunities to have realistic experiences at travel destinations or learn about local culture and history. Conventional virtual travel methods do not adequately address these issues. The present invention aims to overcome these limitations and provide a new virtual travel system that provides users with a realistic travel experience that makes them feel as if they are actually there, while also having educational value. [Means for solving the problem]

[0005] The present invention provides a system for allowing a user to experience a virtual trip through a virtual reality device, comprising: a virtual reality device means for tracking the user's location and movement; a terminal means for loading and displaying virtual reality data based on a selected travel destination; an artificial intelligence means for responding to the user's questions in real time; and a feedback device means for providing tactile feedback such as wind and vibration to the user's virtual trip experience. The artificial intelligence means provides tourist information and guidance related to the user's selected travel destination, enabling a realistic travel experience without the user having to physically visit the destination. The artificial intelligence means also manages and provides educational content based on the user's interactions, providing an educational experience that allows the user to learn and have fun at the same time.

[0006] "User" refers to a person using a virtual reality device.

[0007] A "virtual reality device" is a device for tracking a user's position and movements and displaying a virtual reality environment.

[0008] "Terminal" refers to a computer or electronic device that loads and displays data in the Virtual Travel System.

[0009] "Artificial intelligence" refers to programs or software that respond to users' questions and provide information in real time.

[0010] A "feedback device" is a device for providing tactile feedback such as wind or vibration to a user's virtual travel experience.

[0011] "Virtual reality data" refers to information such as images, audio, and 3D models of travel destinations displayed on virtual reality devices.

[0012] "Tourist information" refers to information about the history, culture, and other aspects of the selected travel destination that is provided to the user by artificial intelligence.

[0013] "Educational content" refers to information such as quizzes, games, and historical commentary that allow users to learn through interaction. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system of the present invention allows users to experience virtual travel through a virtual reality device and obtain information in real time. The system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0036] 1. System startup and initial settings

[0037] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user then accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0038] 2. Select your destination and load your data

[0039] Next, the user operates the interface to select the desired travel destination. The device transmits this selection information to the server. The server retrieves data (video, audio, 3D models, etc.) related to the selected travel destination and transmits it to the device. The device receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0040] 3. Launching the AI ​​Assistant

[0041] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and responds to any questions or instructions the user may have.

[0042] 4. Real-time Guidance and Haptic Feedback

[0043] As the user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant a question. For example, they might ask, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that makes the user feel the wind. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0044] 5. Providing educational experiences

[0045] Furthermore, when a user wants to use educational content, they can select a children's educational mode. In this case, the device switches to educational mode and the server loads the educational content. The AI ​​assistant provides quizzes and games for children. For example, the AI ​​assistant may ask a question such as, "How many years did it take to build this pyramid?" and when the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0046] Example: A scenario in which a user visits the Egyptian pyramids

[0047] The user selects the Egyptian pyramids and the system loads the data. The AI ​​assistant begins the tour with a greeting, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then begins a detailed explanation. The terminal uses a feedback device to allow the user to experience the sensation of feeling the wind and walking on sand.

[0048] ---

[0049] This system allows users to experience real travel experiences through virtual reality devices without physically visiting the locations, and also provides educational content that allows users to have fun while learning.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0053] Step 2:

[0054] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0055] Step 3:

[0056] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0057] Step 4:

[0058] The user operates the interface to select the desired travel destination, and the information on the selected travel destination is sent from the terminal to the server as a request.

[0059] Step 5:

[0060] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0061] Step 6:

[0062] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0063] Step 7:

[0064] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data for the selected travel destination.

[0065] Step 8:

[0066] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0067] Step 9:

[0068] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0069] Step 10:

[0070] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0071] Step 11:

[0072] The device controls the haptic device to provide tactile feedback, such as the sensation of wind, by using the vibration motor of the VR device to transmit gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0073] Step 12:

[0074] The user (or parent) selects the educational mode for children. The device switches to educational mode, the server loads educational content, and the AI ​​assistant provides guidance for the child.

[0075] Step 13:

[0076] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0077] Step 14:

[0078] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0079] Step 15:

[0080] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] While conventional virtual reality systems allow users to experience virtual travel without physically visiting their destinations, they lack a system that provides real-time information. Furthermore, it is difficult to provide an experience that includes not only visual but also tactile sensations, and educational content is limited, limiting the user experience. Furthermore, it is difficult to introduce an artificial intelligence system that responds to user questions and instructions in real time, which hinders the provision of a comprehensive experience.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes a visual display device that tracks the user's location and movements, an information processing device that loads and displays virtual data based on the user's travel destination selection, a knowledge processing device that responds to the user's questions in real time, a sensory presentation device that provides tactile feedback such as wind and vibration to the user's virtual travel experience, a language processing system that provides tourist destination information and special information through speech, and a learning support system that manages and provides educational content to complement the user's experience. This makes it possible to provide detailed information in real time and provide the user with a tactile experience. It also enriches the educational content and supports the user's learning.

[0086] A "visual display device" is a device that tracks the user's position and movements in real time and displays images of a virtual reality space.

[0087] The "information processing device" is a device that processes data to load and display the necessary virtual data based on the selection of travel destinations.

[0088] A "knowledge processing device" is an artificial intelligence system that responds to user questions in real time and provides appropriate information.

[0089] A "sensory presentation device" is a device that provides tactile feedback such as wind or vibration during a user's virtual travel experience.

[0090] A "language processing system" is an artificial intelligence system that provides tourist information and special information to users through speech.

[0091] A "learning support system" is a system that complements the user experience and manages and provides educational content.

[0092] The present invention provides a system that allows users to experience virtual travel and acquire information in real time through a virtual reality device. The system includes, as its main components, a visual display device, an information processing device, a knowledge processing device, a sensory presentation device, a language processing system, and a learning support system.

[0093] When a user turns on the virtual reality device, the device automatically launches the software. Specifically, the visual display device tracks the user's position and movements and transmits the data to the device. Based on this, the device adjusts and displays the image in the virtual reality space appropriately.

[0094] Next, the user operates the interface to select a desired travel destination, and the terminal transmits this selection information to the server. The server receives this information, retrieves data (video, audio, 3D model, etc.) related to the selected travel destination, and transmits it to the terminal. The terminal receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0095] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). Using a language processing system, the AI ​​assistant can begin guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and respond to the user's questions and instructions.

[0096] For example, if a user asks, "Tell me about the history of this pyramid," the server analyzes the question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. The device also controls a sensory presentation device to provide haptic feedback to recreate the sensation of the user feeling the wind or walking on sand. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0097] Furthermore, when a user uses educational content, the learning support system switches to a children's education mode. In this case, the server loads educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant then asks the user educational questions, such as, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0098] As a concrete example, consider a scenario in which a user selects the pyramids of Egypt. The server sends data about the pyramids, including video, audio, and 3D models, to the terminal, which then loads it into a virtual reality device. The AI ​​assistant begins a tour with a greeting like "Welcome to the pyramids of Egypt," and provides detailed explanations in response to the user's questions. The terminal uses a sensory presentation device to give the user the sensation of feeling the wind or walking on sand.

[0099] By providing prompts like the following to the generative AI model, you can provide appropriate guidance and feedback:

[0100] Tell me about the Egyptian pyramids.

[0101] This system allows users to experience real travel through virtual reality devices without physically visiting the destination, and also provides educational content that allows users to have fun while learning.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] When a user powers on the virtual reality device, the device automatically launches the software. The visual display device collects data from built-in sensors to track the user's position and movements. This data serves as input for displaying a virtual space that updates in real time in response to the user's movements. The device processes this sensor data and sends it to the visual display device to generate output that displays the virtual reality environment to the user.

[0105] Step 2:

[0106] The user accesses the specified login screen and enters their user ID and password. This information is sent from the terminal to the server, which then verifies the credentials. The server compares the entered credentials with the user information in its database, and if authentication is successful, approves the login. This process allows the user to log in to the system, and the terminal generates output to move on to the next step.

[0107] Step 3:

[0108] The user operates the interface to select the desired travel destination. The user's selection information is sent from the terminal to the server. Based on this input information, the server searches the database for data (video, audio, 3D models, etc.) related to the corresponding travel destination. The server then sends this data to the terminal, which then loads the received data into the virtual reality device in real time. This process results in an output that displays the virtual environment of the travel destination in the user's field of vision.

[0109] Step 4:

[0110] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data (such as historical information) about the selected travel destination. This data is used as input for the AI ​​assistant, which analyzes it and generates a guide message such as "Hello, welcome to the Egyptian pyramids!". Depending on the user's input (questions or instructions), the AI ​​assistant generates an appropriate response. This response is output to the user as audio.

[0111] Step 5:

[0112] As the user explores the VR space and shows interest in a particular place or building, they can ask the AI ​​assistant a question. This input information is sent from the device to the server, which then analyzes the user's question. For example, if the user asks, "Tell me about the history of this pyramid," the server searches the database for the appropriate information and sends it to the AI ​​assistant. The AI ​​assistant then responds, "This pyramid was built in 2560 BC," and generates a detailed explanation. This answer is output to the user via voice.

[0113] Step 6:

[0114] The device controls the sensory presentation device to provide haptic feedback to the user, recreating the sensation of feeling the wind or walking on sand. For example, the device controls the vibration motor of the VR device to transmit gentle vibrations to the user's feet. Based on this input data, the device calculates vibration patterns and controls actual movements. This process generates haptic output that further immerses the user.

[0115] Step 7:

[0116] When a user uses educational content, the learning support system switches to a children's education mode. The server loads the educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant asks the user educational questions. Based on this input, the user enters an answer, and the AI ​​assistant provides immediate feedback. For example, it might ask a question like, "How many years did it take to build this pyramid?" and respond to the user's answer with a response like, "That's right!" or "Sorry, the correct answer is 20 years." This process generates an output that enhances the user's educational experience.

[0117] (Application example 1)

[0118] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] While existing virtual travel systems were effective in allowing users to virtually experience specific tourist destinations, there were no systems specifically designed for virtual shopping experiences. This made it difficult for users to check out products and obtain detailed information without visiting a real store. Furthermore, they lacked the ability to provide haptic feedback about products, creating a need for a more realistic shopping experience that cannot be achieved through online shopping.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0121] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for allowing the user to select a virtual store and for loading and displaying store information, an artificial intelligence means for responding to the user's questions in real time and providing detailed product information, and a feedback device means for providing haptic feedback about the products in the user's virtual shopping experience, thereby enabling the user to experience shopping in a virtual store without going to a real store and to obtain detailed product information and haptic feedback in real time.

[0122] "User" refers to an individual who utilizes the system to experience virtual reality.

[0123] A "virtual reality device" is a device for tracking a user's position and movements and displaying a virtual environment.

[0124] "Terminal means" refers to a device that allows a user to select a virtual store and load and display store information.

[0125] "Artificial intelligence means" refers to algorithms and software that respond to user questions in real time and provide detailed product information.

[0126] "Feedback device means" refers to devices or technology that provide tactile feedback about products in a user's virtual shopping experience.

[0127] A "virtual store" is a store that exists in a virtual space and provides a virtual environment where users can visit, check out products, and purchase them.

[0128] "Product information" refers to detailed descriptions and specifications of products that users can view in the virtual store.

[0129] "Haptic feedback" refers to technology that allows users to sense virtual physical properties (e.g., texture or temperature).

[0130] "Virtual shopping experience" refers to the experience of a user visiting a store in virtual reality, browsing, selecting, and purchasing products.

[0131] "Real-time response" means providing appropriate information and feedback immediately in response to a user's questions or operations.

[0132] The system of the present invention allows users to visit a virtual store through a virtual reality device and obtain detailed product information in real time. The system includes a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0133] When the system starts up, the user turns on the virtual reality device and the device launches the software. The various sensors that track the virtual reality device's position and movement are adjusted to ensure proper operation. The user accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0134] Next, the user operates the interface to select the desired virtual store. The terminal transmits this selection information to the server. The server retrieves data related to the selected virtual store (product information, video, audio, 3D models, etc.) and transmits it to the terminal. The terminal receives this data, loads it into the virtual reality device, and displays the virtual store environment in the user's field of view.

[0135] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with store information and special data about the selected product (such as material information). The AI ​​assistant begins guiding the user, saying, "Welcome to our bag specialty store!" and responds to any questions or instructions the user may have.

[0136] As a user explores the virtual store and shows interest in a specific product, they can ask the AI ​​assistant a question. For example, they might ask, "What material is this bag made of?" The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This bag is made of genuine leather," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that allows the user to feel the texture of the bag. For example, the feedback device may transmit soft vibrations to the user's hand, recreating the texture of the bag.

[0137] Furthermore, when a user wants to use educational content, they can select product education mode. In this case, the device switches to education mode and the server loads the product education content. The AI ​​assistant provides quizzes and games for the user. For example, the AI ​​assistant may ask a question such as, "How many steps are involved in the manufacturing process of this bag?". When the user enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's correct!" or "Sorry, the correct answer is five steps."

[0138] Specific examples

[0139] Consider a scenario in which a user uses a virtual reality device to visit a virtual bag store. The system verifies login information and loads the virtual store's data. The AI ​​assistant begins guiding the user with a greeting: "Welcome to our bag specialty store." When the user asks, "What material is this bag made of?", the server provides the material information to the AI ​​assistant, who then begins a detailed explanation, saying, "This bag is made of genuine leather." The feedback device also reproduces the texture of the bag, allowing the user to experience its feel.

[0140] Example prompt sentence:

[0141] Simulate a scenario where a user asks, "What material is this bag made of?" The AI ​​assistant will provide detailed material information and realistically recreate tactile feedback using a haptic device.

[0142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0143] Step 1:

[0144] The user turns on the virtual reality device, and the virtual reality device and the terminal start up. The various sensors of the virtual reality device (position, movement, etc.) are adjusted to operate normally.

[0145] Input: Power supply for virtual reality device, user movement information.

[0146] Data processing / computation: The virtual reality device tracks the user's position and movement in real time and calibrates the sensors.

[0147] Output: The user's location and movement information is sent to the device.

[0148] Specific actions: The user puts on the virtual reality device and sets it up.

[0149] Step 2:

[0150] The terminal starts the software and accesses the login screen specified by the user. The user logs in by entering their user ID and password. The server verifies the login information and performs authentication.

[0151] Input: User ID, Password.

[0152] Data processing / calculation: The server performs user authentication and returns the authentication result.

[0153] Output: Login confirmation or error message is displayed on the terminal.

[0154] Specific behavior: The user enters login information, and the device sends the entered information to the server.

[0155] Step 3:

[0156] The user operates the interface to select the desired virtual store, and the terminal transmits the selection information to the server.

[0157] Input: User's store selection information.

[0158] Data processing / calculation: The server searches for the selected virtual store data (product information, video, audio, 3D models, etc.) and sends it to the terminal.

[0159] Output: Store data is sent to the terminal and loaded into the virtual reality device.

[0160] Specific operation: The user operates the interface to select a store, and the terminal sends this to the server.

[0161] Step 4:

[0162] The device launches the AI ​​assistant software, and the server provides store information and product data to the AI ​​assistant.

[0163] Input: Store information and product data.

[0164] Data processing / calculation: The AI ​​assistant analyzes store information and product data and prepares information to provide in an interactive format.

[0165] Output: Your AI assistant is now ready.

[0166] Specific operation: The software starts and receives data from the server.

[0167] Step 5:

[0168] A user explores a virtual store and asks a question about a specific product. The terminal sends the question to the server.

[0169] Input: The user's question.

[0170] Data processing / calculation: The server analyzes the question and sends the corresponding product data to the AI ​​assistant.

[0171] Output: Product information is sent to the device and the AI ​​assistant provides an answer.

[0172] How it works: The user asks the AI ​​assistant a question, and the device sends it to the server.

[0173] Step 6:

[0174] The terminal controls the feedback device to provide haptic feedback to the user, for example, the feedback device reproduces the texture of a bag.

[0175] Input: User operation information, product information.

[0176] Data processing / calculation: The feedback device generates appropriate haptic feedback based on product data.

[0177] Output: Haptic feedback is provided to the user.

[0178] Specific action: The feedback device transmits vibrations or pressure to the user's hand.

[0179] Step 7:

[0180] If the user selects the product education mode, the terminal switches to the education mode and loads the product education content from the server.

[0181] Input: User mode selection.

[0182] Data processing / calculation: The server sends the product educational content to the terminal, and the terminal loads it.

[0183] Output: Educational content is displayed on a virtual reality device.

[0184] Specific behavior: The user selects the education mode, and the device loads the data accordingly.

[0185] Step 8:

[0186] The AI ​​assistant provides quizzes and games for users, making the virtual shopping experience fun.

[0187] Input: Educational content, user answers.

[0188] Data processing / calculation: The AI ​​assistant will ask questions and evaluate the answers, providing feedback on the results.

[0189] Output: The quiz results and explanations are provided to the user.

[0190] What happens: The user takes a quiz and the AI ​​assistant responds accordingly.

[0191] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0192] The system of the present invention allows users to experience virtual travel through a virtual reality device, and further combines it with an emotion engine that recognizes the user's emotions. This system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[0193] 1. System startup and initial settings

[0194] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. The user can update their profile information if necessary.

[0195] 2. Starting and initializing the emotion engine

[0196] Next, the device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition technology and voice analysis technology to determine the user's emotions.

[0197] 3. Select your destination and load your data

[0198] The user operates the interface to select the desired travel destination. Information about the selected travel destination is sent from the device to the server as a request. The server receives the request, searches the database for data about the selected travel destination (video, audio, 3D model, etc.), and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0199] 4. Launching the AI ​​Assistant

[0200] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data for the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0201] 5. Real-time guidance, haptic feedback, and emotion engine integration

[0202] The user explores the VR space and shows interest in a specific place or building. The user asks the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0203] The emotion engine also analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored. Based on this, the feedback device is adjusted. For example, if the user is excited, the haptic feedback that feels like wind is enhanced. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[0204] 6. Providing educational experiences

[0205] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the emotion engine analyzes the user's emotions and optimizes how the educational content is presented.

[0206] Example: A scenario in which a user visits the Egyptian pyramids

[0207] A user selects the Egyptian pyramids and begins the VR experience. The AI ​​assistant begins by guiding them, saying, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of these pyramids," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback device enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[0208] ---

[0209] The system allows users to have a personalized and realistic travel experience through virtual reality devices and emotion engines without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0213] Step 2:

[0214] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0215] Step 3:

[0216] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0217] Step 4:

[0218] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[0219] Step 5:

[0220] The user operates the interface to select the desired travel destination, and information about the selected destination is sent from the terminal to the server as a request.

[0221] Step 6:

[0222] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0223] Step 7:

[0224] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0225] Step 8:

[0226] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data (such as historical information) for the selected travel destination.

[0227] Step 9:

[0228] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0229] Step 10:

[0230] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0231] Step 11:

[0232] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0233] Step 12:

[0234] The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored, and adjusts the feedback device accordingly.

[0235] Step 13:

[0236] The terminal controls the haptic device to provide haptic feedback to the user, such as the sensation of wind. For example, if the user is excited, the haptic feedback of wind is enhanced.

[0237] Step 14:

[0238] When a user wants to use educational content, they select the educational mode for children. The device switches to educational mode, the server loads the educational content, and the AI ​​assistant provides guidance for children.

[0239] Step 15:

[0240] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0241] Step 16:

[0242] The emotion engine analyzes the child's emotions from their facial expressions and voice, and adjusts the difficulty and content of quizzes and games accordingly.

[0243] Step 17:

[0244] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0245] Step 18:

[0246] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0247] Example 2

[0248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0249] The present invention relates to a system for providing a virtual travel experience. Conventional systems have difficulty in properly recognizing a user's emotional state and providing a personalized experience. Furthermore, conventional systems lack sufficient educational content and lack means to maintain a user's interest in the experience. This makes it difficult for users to obtain a deep learning experience.

[0250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0251] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the user's selected travel destination, an artificial intelligence means for responding to the user's questions in real time, a feedback means for providing feedback such as wind and tactile sensations to the user's virtual travel experience, an emotion analysis engine means for analyzing the user's facial expressions and voice in real time to determine the user's emotional state, and an education mode means for providing educational content. This makes it possible to personalize the virtual travel experience based on the user's emotional state and further optimize and provide educational content.

[0252] "Virtual reality device means" is a general term for hardware that includes devices and sensors for tracking a user's position and movements.

[0253] "Terminal means" refers to an electronic device that allows a user to select a travel destination and has the function of loading and displaying virtual reality data based on that selection.

[0254] "Artificial Intelligence Means" refers collectively to software and algorithms that respond to a user's questions in real time and provide information and guidance about the selected travel destination.

[0255] "Feedback means" are devices and systems that provide physical feedback, such as wind or haptics, to enhance the user's virtual travel experience.

[0256] The "emotion analysis engine means" is software and algorithms for analyzing the user's facial expressions and voice in real time and determining the user's emotional state.

[0257] "Educational mode means" refers to software and systems that have functionality for managing and optimizing educational content as it is experienced by a user.

[0258] The system of the present invention is designed for users to experience virtual travel through a virtual reality device, and integrates an emotion engine that recognizes the user's emotional state. The system includes the following components: a virtual reality device means, a terminal means, a server means, an artificial intelligence means, a feedback means, an emotion analysis engine means, and an education mode means.

[0259] When a user turns on the virtual reality device, the terminal starts the software, which adjusts the virtual reality device's various sensors (position sensors, motion sensors, haptic feedback devices, etc.) to ensure they are working properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. If necessary, the user can update their profile information.

[0260] Next, the device launches an emotion analysis engine to prepare for real-time analysis of the user's facial expressions and voice. The emotion analysis engine uses facial recognition and voice analysis technologies (e.g., OpenCV or Google® Speech API) to determine the user's emotional state.

[0261] When a user operates the interface to select a desired travel destination, information about the selected travel destination is sent as a request from the terminal to the server. The server receives the request, searches a database (e.g., MySQL®) for data about the selected travel destination (video, audio, 3D model, etc.), and sends it to the terminal. The terminal receives this data and loads it into the virtual reality device, and the virtual environment of the travel destination is displayed in the user's field of view.

[0262] Furthermore, the device launches the AI ​​assistant software (e.g., Dialogflow). The server provides the AI ​​assistant with tourist information and special data for the selected travel destination. The AI ​​assistant begins guiding the user, saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0263] Additionally, if a user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. An emotion analysis engine analyzes the user's facial expressions and voice in real time to determine whether the user is excited, impressed, or bored. Based on this, the feedback method is adjusted. For example, if the user is excited, the haptic feedback device will enhance the haptic feedback that makes them feel the wind. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[0264] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the sentiment analysis engine analyzes the user's emotions and optimizes the way the educational content is presented.

[0265] As a concrete example, when a user selects the Egyptian pyramids and starts the VR experience, the AI ​​assistant begins by saying, "Welcome to the Egyptian pyramids." If the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback method enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[0266] Examples of prompts include:

[0267] "Tell me the history of the Egyptian pyramids."

[0268] "Tell me about the special exhibits."

[0269] "I want to know more about Kyoto festivals."

[0270] "I want to switch to education mode and learn."

[0271] The system allows users to have a personalized and realistic travel experience through virtual reality devices and an emotion analysis engine without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[0272] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0273] Step 1:

[0274] The user powers on the virtual reality device, which initializes the device's hardware and prepares the system to boot.

[0275] Input: Power-on by user operation

[0276] Data processing: Hardware initialization process

[0277] Output: System startup

[0278] Specific behavior:

[0279] The user presses and holds the power button on the VR headset, and the device starts up.

[0280] Step 2:

[0281] The device launches the connected software and adjusts the various sensors of the virtual reality device (position sensors, movement sensors, haptic feedback devices, etc.).

[0282] Input: Virtual reality device sensor data

[0283] Data processing: Sensor calibration

[0284] Output: Adjustment completion notification

[0285] Specific behavior:

[0286] The software on the device will automatically launch and calibrate the device's various sensors, using device drivers and calibration tools.

[0287] Step 3:

[0288] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[0289] Input: User ID and password

[0290] Data Processing: Credentials Verification

[0291] Output: Login approved or denied

[0292] Specific behavior:

[0293] The user enters their ID and password using the keyboard on the device display, and the input data is sent to the server, where authentication is performed by referencing a database.

[0294] Step 4:

[0295] The device launches an emotion analysis engine and prepares to analyze the user's facial expressions and voice in real time.

[0296] Input: facial expression data and voice data

[0297] Data processing: Real-time analysis

[0298] Output: Emotional state identification result

[0299] Specific behavior:

[0300] The device runs emotion analysis engine software, which collects data from the camera and microphone, and uses facial recognition and voice analysis technologies to determine the user's emotional state in real time.

[0301] Step 5:

[0302] The user operates the virtual travel interface to select a desired travel destination.

[0303] Input: User's travel destination selection

[0304] Data processing: Save selected data

[0305] Output: Generate a travel destination data request

[0306] Specific behavior:

[0307] The user operates the terminal interface and selects a destination from the list of travel destinations, which causes the data of the selected travel destination to be recorded in the terminal and prepares for sending a request to the server.

[0308] Step 6:

[0309] The device requests data on the selected travel destination from the server, the server searches for the corresponding data, and sends it to the device, which receives the data and loads it into the virtual reality device.

[0310] Input: Travel Destination Data Request

[0311] Data processing: Searching and sending travel destination data

[0312] Output: Loading travel destination data

[0313] Specific behavior:

[0314] The device sends an HTTP request to the server, which retrieves travel destination data from a database and returns a response in JSON format. The device receives the data, analyzes it, and displays it on the virtual reality device.

[0315] Step 7:

[0316] The device launches the AI ​​assistant software, the server provides tourist information and special data, and the AI ​​assistant starts guiding and interacting with the user.

[0317] Input: Destination data and tourist information

[0318] Data processing: scenario generation and dialogue management

[0319] Output: Guidance and dialogue start

[0320] Specific behavior:

[0321] The device launches the AI ​​assistant software and begins providing guidance based on the tourist information received from the server. The AI ​​assistant speaks to the user, saying, "Hello, welcome to the Egyptian pyramids!"

[0322] Step 8:

[0323] The user explores the VR space and asks questions at points of interest. The server analyzes the questions, and the AI ​​assistant provides answers. The emotion analysis engine analyzes the user's emotions, and the feedback method is adjusted accordingly.

[0324] Input: User question

[0325] Data processing: Question analysis and answer generation

[0326] Output: Answer to the question

[0327] Specific behavior:

[0328] When a user asks, "Tell me about the history of this pyramid," the server uses natural language processing technology to analyze the question and provide the AI ​​assistant with appropriate answer data. The emotion analysis engine analyzes facial expressions and tone of voice, and the feedback method adjusts haptic feedback. For example, it enhances the sensation of wind.

[0329] Step 9:

[0330] When a user selects education mode, the device switches to education mode and the server loads educational content. The AI ​​assistant provides quizzes and games for children and optimizes the delivery method using a sentiment analysis engine.

[0331] Input: Select Education Mode

[0332] Data processing: Loading and optimizing educational content

[0333] Output: Providing quizzes and games

[0334] Specific behavior:

[0335] When a user presses the education mode button, the device sends a mode change request to the server. The server loads educational content, and the AI ​​assistant asks a quiz question, such as "How many years did it take to build this pyramid?" and provides immediate feedback to the user after they answer. The sentiment analysis engine analyzes the user's level of interest in real time and selects the appropriate content delivery method.

[0336] (Application example 2)

[0337] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Conventional virtual travel systems provide realistic experiences without requiring users to physically visit the destination. However, they are unable to adjust feedback based on the user's emotions and lack appropriate responses to emotions such as boredom and excitement. Furthermore, in physical stores, product information is provided in a one-way manner, and there is no interactive shopping experience based on the user's emotions. This results in a poor user experience and limited sales promotion effectiveness in physical stores.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the travel destination selection, an artificial intelligence means for responding to the user's questions in real time, a feedback device means for providing tactile feedback such as wind and vibration to the user's virtual travel experience, and an emotion engine means for analyzing the user's emotions in real time and adjusting the feedback content based on the analysis results. This not only enables appropriate feedback based on the user's emotions to be provided in real time during the virtual travel, but also improves the shopping experience in physical stores, piques the user's interest, and increases sales promotion effects.

[0340] A "virtual reality device" is a device that allows a user to experience a virtual reality space.

[0341] A "terminal" is a computer terminal that is connected to a virtual reality device and that loads and displays data.

[0342] "Artificial intelligence" refers to interactive computer programs that respond to users' questions and provide information in real time.

[0343] A "feedback device" is a device that provides tactile feedback, such as wind or vibration, to a user's virtual travel experience.

[0344] The "emotion engine" is a system that analyzes the user's emotions in real time and adjusts the feedback content based on the analysis results.

[0345] The system of the present invention is composed of the following elements to enable users to experience virtual travel and virtual shopping in physical stores through a virtual reality device: It has the function of analyzing the user's emotions and adjusting feedback individually based on those emotions.

[0346] System Overview

[0347] The system consists of a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[0348] Hardware and software used

[0349] Virtual reality device: The primary device through which the user experiences the virtual space, providing visual and audio output.

[0350] Terminal: A computer terminal that tracks the user's movements and location and loads and displays virtual reality data.

[0351] Server: Manages virtual reality data, tourist information, product information, etc., and provides it to terminals and virtual reality devices.

[0352] Artificial intelligence: Interactive software that runs on a device or server and responds to user questions in real time. The software used includes natural language processing techniques and machine learning algorithms.

[0353] Feedback devices: These devices provide haptic feedback to enhance the user's virtual experience, providing physical sensations such as wind and vibration.

[0354] Emotion Engine: A system that analyzes the user's emotions and adjusts the feedback content. The technologies used include facial recognition and voice analysis.

[0355] System Operation

[0356] 1. System startup and initial setup:

[0357] The user turns on the virtual reality device, and the device launches the software, which calibrates the virtual reality device's sensors (position, movement, and haptic feedback) to ensure they function properly.

[0358] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[0359] 2. Emotion Engine startup and initial setup:

[0360] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[0361] 3. Select your destination or product information and load your data:

[0362] The user operates the interface to select the desired travel destination or product information, and the selected information is sent from the terminal to the server as a request.

[0363] The server receives the request, searches for the relevant data from the database, and sends it to the terminal, which then receives the data, loads it into the virtual reality device, and displays the virtual environment and product information in the user's field of view.

[0364] 4. Artificial Intelligence Guidance and Interaction:

[0365] The device activates artificial intelligence to provide tourist and product information, and users can interact with the AI ​​based on where they want to see next and what information they want to hear.

[0366] 5. Real-time feedback and emotion engine integration:

[0367] Users explore the VR space and obtain information related to places and products they are interested in. The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited or bored, and adjusts the feedback accordingly.

[0368] Specific examples

[0369] Consider a scenario where a user is wearing smart glasses and shopping in a brick-and-mortar store: if the emotion engine determines that the user is "excited," the AI ​​assistant will provide enhanced details about the product and promotions.

[0370] When a user asks, "Tell me about this product," the AI ​​assistant provides detailed information, such as, "This is the latest smartphone, and its main features are..." If the emotion engine detects boredom, it will provide interesting information in the form of a quiz, encouraging interaction.

[0371] Prompt Sentence Examples

[0372] The following script is a virtual shopping assistant for brick-and-mortar stores that uses smart glasses. It includes a function to adjust the way information is presented depending on the user's emotions. As a concrete example, it shows a scenario in which detailed product information is provided when the user is excited.

[0373] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0374] Step 1:

[0375] The user turns on the virtual reality device. The device launches the software and calibrates the virtual reality device's sensors to ensure proper operation. The device displays a login screen to the user, prompting them to enter their user ID and password. The server then verifies the credentials and, if successful, logs the user into the system. The user's profile information is loaded onto the device.

[0376] Step 2:

[0377] The device activates the emotion engine, which uses facial recognition and voice analysis technologies to capture and analyze the user's facial expressions and voice in real time. The camera captures the user's facial expressions, and the voice input system collects the user's voice. Based on this input data, the emotion engine determines the user's emotions and sends the results to the feedback system.

[0378] Step 3:

[0379] The user operates the interface to select the desired travel destination or product information. The device receives this selection and sends a request to the server. The server searches the database for the corresponding virtual reality data (video, audio, 3D models, etc.) and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays it in the user's field of view.

[0380] Step 4:

[0381] The terminal activates the AI. The server provides tourist information, product information, etc. to the AI, and the AI ​​starts providing guidance to the user, such as "Hello, welcome to the Egyptian pyramids." When the user asks a question, the terminal sends the question to the server, and the server generates an appropriate answer to the question and conveys it to the user via the AI.

[0382] Step 5:

[0383] As the user progresses through the virtual journey and shopping, they will show interest in specific places and products. The emotion engine analyzes the user's facial expressions and voice to determine whether they are excited or bored. If the user is excited, the feedback device will enhance tactile feedback such as wind, and the AI ​​will provide additional interesting information and suggestions. Conversely, if the user is bored, the AI ​​will provide information in the form of quizzes and games to pique the user's interest.

[0384] Step 6:

[0385] If the user selects education mode as they continue their experience, the device will load educational content. The server provides educational content for children, and the AI ​​interacts with the user (child) through quizzes and games. Again, the emotion engine analyzes the user's emotions and adjusts feedback at the appropriate time. For example, the AI ​​assistant might ask a quiz question such as, "How many years did it take to build this pyramid?" and provide immediate feedback to the user's answer.

[0386] These steps make the virtual travel and shopping experience more personalized and engaging for the user.

[0387] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0388] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0389] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0390] [Second embodiment]

[0391] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0392] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0393] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0394] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0395] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0397] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0398] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0399] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0400] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0401] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0402] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0403] The system of the present invention allows users to experience virtual travel through a virtual reality device and obtain information in real time. The system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0404] 1. System startup and initial settings

[0405] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user then accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0406] 2. Select your destination and load your data

[0407] Next, the user operates the interface to select the desired travel destination. The device transmits this selection information to the server. The server retrieves data (video, audio, 3D models, etc.) related to the selected travel destination and transmits it to the device. The device receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0408] 3. Launching the AI ​​Assistant

[0409] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and responds to any questions or instructions the user may have.

[0410] 4. Real-time Guidance and Haptic Feedback

[0411] As the user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant a question. For example, they might ask, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that makes the user feel the wind. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0412] 5. Providing educational experiences

[0413] Furthermore, when a user wants to use educational content, they can select a children's educational mode. In this case, the device switches to educational mode and the server loads the educational content. The AI ​​assistant provides quizzes and games for children. For example, the AI ​​assistant may ask a question such as, "How many years did it take to build this pyramid?" and when the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0414] Example: A scenario in which a user visits the Egyptian pyramids

[0415] The user selects the Egyptian pyramids and the system loads the data. The AI ​​assistant begins the tour with a greeting, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then begins a detailed explanation. The terminal uses a feedback device to allow the user to experience the sensation of feeling the wind and walking on sand.

[0416] ---

[0417] This system allows users to experience real travel experiences through virtual reality devices without physically visiting the locations, and also provides educational content that allows users to have fun while learning.

[0418] The processing flow will be explained below.

[0419] Step 1:

[0420] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0421] Step 2:

[0422] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0423] Step 3:

[0424] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0425] Step 4:

[0426] The user operates the interface to select the desired travel destination, and the information on the selected travel destination is sent from the terminal to the server as a request.

[0427] Step 5:

[0428] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0429] Step 6:

[0430] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0431] Step 7:

[0432] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data for the selected travel destination.

[0433] Step 8:

[0434] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0435] Step 9:

[0436] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0437] Step 10:

[0438] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0439] Step 11:

[0440] The device controls the haptic device to provide tactile feedback, such as the sensation of wind, by using the vibration motor of the VR device to transmit gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0441] Step 12:

[0442] The user (or parent) selects the educational mode for children. The device switches to educational mode, the server loads educational content, and the AI ​​assistant provides guidance for the child.

[0443] Step 13:

[0444] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0445] Step 14:

[0446] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0447] Step 15:

[0448] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0449] Example 1

[0450] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0451] While conventional virtual reality systems allow users to experience virtual travel without physically visiting their destinations, they lack a system that provides real-time information. Furthermore, it is difficult to provide an experience that includes not only visual but also tactile sensations, and educational content is limited, limiting the user experience. Furthermore, it is difficult to introduce an artificial intelligence system that responds to user questions and instructions in real time, which hinders the provision of a comprehensive experience.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0453] In this invention, the server includes a visual display device that tracks the user's location and movements, an information processing device that loads and displays virtual data based on the user's travel destination selection, a knowledge processing device that responds to the user's questions in real time, a sensory presentation device that provides tactile feedback such as wind and vibration to the user's virtual travel experience, a language processing system that provides tourist destination information and special information through speech, and a learning support system that manages and provides educational content to complement the user's experience. This makes it possible to provide detailed information in real time and provide the user with a tactile experience. It also enriches the educational content and supports the user's learning.

[0454] A "visual display device" is a device that tracks the user's position and movements in real time and displays images of a virtual reality space.

[0455] The "information processing device" is a device that processes data to load and display the necessary virtual data based on the selection of travel destinations.

[0456] A "knowledge processing device" is an artificial intelligence system that responds to user questions in real time and provides appropriate information.

[0457] A "sensory presentation device" is a device that provides tactile feedback such as wind or vibration during a user's virtual travel experience.

[0458] A "language processing system" is an artificial intelligence system that provides tourist information and special information to users through speech.

[0459] A "learning support system" is a system that complements the user experience and manages and provides educational content.

[0460] The present invention provides a system that allows users to experience virtual travel and acquire information in real time through a virtual reality device. The system includes, as its main components, a visual display device, an information processing device, a knowledge processing device, a sensory presentation device, a language processing system, and a learning support system.

[0461] When a user turns on the virtual reality device, the device automatically launches the software. Specifically, the visual display device tracks the user's position and movements and transmits the data to the device. Based on this, the device adjusts and displays the image in the virtual reality space appropriately.

[0462] Next, the user operates the interface to select a desired travel destination, and the terminal transmits this selection information to the server. The server receives this information, retrieves data (video, audio, 3D model, etc.) related to the selected travel destination, and transmits it to the terminal. The terminal receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0463] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). Using a language processing system, the AI ​​assistant can begin guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and respond to the user's questions and instructions.

[0464] For example, if a user asks, "Tell me about the history of this pyramid," the server analyzes the question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. The device also controls a sensory presentation device to provide haptic feedback to recreate the sensation of the user feeling the wind or walking on sand. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0465] Furthermore, when a user uses educational content, the learning support system switches to a children's education mode. In this case, the server loads educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant then asks the user educational questions, such as, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0466] As a concrete example, consider a scenario in which a user selects the pyramids of Egypt. The server sends data about the pyramids, including video, audio, and 3D models, to the terminal, which then loads it into a virtual reality device. The AI ​​assistant begins a tour with a greeting like "Welcome to the pyramids of Egypt," and provides detailed explanations in response to the user's questions. The terminal uses a sensory presentation device to give the user the sensation of feeling the wind or walking on sand.

[0467] By providing prompts like the following to the generative AI model, you can provide appropriate guidance and feedback:

[0468] Tell me about the Egyptian pyramids.

[0469] This system allows users to experience real travel through virtual reality devices without physically visiting the destination, and also provides educational content that allows users to have fun while learning.

[0470] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0471] Step 1:

[0472] When a user powers on the virtual reality device, the device automatically launches the software. The visual display device collects data from built-in sensors to track the user's position and movements. This data serves as input for displaying a virtual space that updates in real time in response to the user's movements. The device processes this sensor data and sends it to the visual display device to generate output that displays the virtual reality environment to the user.

[0473] Step 2:

[0474] The user accesses the specified login screen and enters their user ID and password. This information is sent from the terminal to the server, which then verifies the credentials. The server compares the entered credentials with the user information in its database, and if authentication is successful, approves the login. This process allows the user to log in to the system, and the terminal generates output to move on to the next step.

[0475] Step 3:

[0476] The user operates the interface to select the desired travel destination. The user's selection information is sent from the terminal to the server. Based on this input information, the server searches the database for data (video, audio, 3D models, etc.) related to the corresponding travel destination. The server then sends this data to the terminal, which then loads the received data into the virtual reality device in real time. This process results in an output that displays the virtual environment of the travel destination in the user's field of vision.

[0477] Step 4:

[0478] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data (such as historical information) about the selected travel destination. This data is used as input for the AI ​​assistant, which analyzes it and generates a guide message such as "Hello, welcome to the Egyptian pyramids!". Depending on the user's input (questions or instructions), the AI ​​assistant generates an appropriate response. This response is output to the user as audio.

[0479] Step 5:

[0480] As the user explores the VR space and shows interest in a particular place or building, they can ask the AI ​​assistant a question. This input information is sent from the device to the server, which then analyzes the user's question. For example, if the user asks, "Tell me about the history of this pyramid," the server searches the database for the appropriate information and sends it to the AI ​​assistant. The AI ​​assistant then responds, "This pyramid was built in 2560 BC," and generates a detailed explanation. This answer is output to the user via voice.

[0481] Step 6:

[0482] The device controls the sensory presentation device to provide haptic feedback to the user, recreating the sensation of feeling the wind or walking on sand. For example, the device controls the vibration motor of the VR device to transmit gentle vibrations to the user's feet. Based on this input data, the device calculates vibration patterns and controls actual movements. This process generates haptic output that further immerses the user.

[0483] Step 7:

[0484] When a user uses educational content, the learning support system switches to a children's education mode. The server loads the educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant asks the user educational questions. Based on this input, the user enters an answer, and the AI ​​assistant provides immediate feedback. For example, it might ask a question like, "How many years did it take to build this pyramid?" and respond to the user's answer with a response like, "That's right!" or "Sorry, the correct answer is 20 years." This process generates an output that enhances the user's educational experience.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] While existing virtual travel systems were effective in allowing users to virtually experience specific tourist destinations, there were no systems specifically designed for virtual shopping experiences. This made it difficult for users to check out products and obtain detailed information without visiting a real store. Furthermore, they lacked the ability to provide haptic feedback about products, creating a need for a more realistic shopping experience that cannot be achieved through online shopping.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for allowing the user to select a virtual store and for loading and displaying store information, an artificial intelligence means for responding to the user's questions in real time and providing detailed product information, and a feedback device means for providing haptic feedback about the products in the user's virtual shopping experience, thereby enabling the user to experience shopping in a virtual store without going to a real store and to obtain detailed product information and haptic feedback in real time.

[0490] "User" refers to an individual who utilizes the system to experience virtual reality.

[0491] A "virtual reality device" is a device for tracking a user's position and movements and displaying a virtual environment.

[0492] "Terminal means" refers to a device that allows a user to select a virtual store and load and display store information.

[0493] "Artificial intelligence means" refers to algorithms and software that respond to user questions in real time and provide detailed product information.

[0494] "Feedback device means" refers to devices or technology that provide tactile feedback about products in a user's virtual shopping experience.

[0495] A "virtual store" is a store that exists in a virtual space and provides a virtual environment where users can visit, check out products, and purchase them.

[0496] "Product information" refers to detailed descriptions and specifications of products that users can view in the virtual store.

[0497] "Haptic feedback" refers to technology that allows users to sense virtual physical properties (e.g., texture or temperature).

[0498] "Virtual shopping experience" refers to the experience of a user visiting a store in virtual reality, browsing, selecting, and purchasing products.

[0499] "Real-time response" means providing appropriate information and feedback immediately in response to a user's questions or operations.

[0500] The system of the present invention allows users to visit a virtual store through a virtual reality device and obtain detailed product information in real time. The system includes a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0501] When the system starts up, the user turns on the virtual reality device and the device launches the software. The various sensors that track the virtual reality device's position and movement are adjusted to ensure proper operation. The user accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0502] Next, the user operates the interface to select the desired virtual store. The terminal transmits this selection information to the server. The server retrieves data related to the selected virtual store (product information, video, audio, 3D models, etc.) and transmits it to the terminal. The terminal receives this data, loads it into the virtual reality device, and displays the virtual store environment in the user's field of view.

[0503] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with store information and special data about the selected product (such as material information). The AI ​​assistant begins guiding the user, saying, "Welcome to our bag specialty store!" and responds to any questions or instructions the user may have.

[0504] As a user explores the virtual store and shows interest in a specific product, they can ask the AI ​​assistant a question. For example, they might ask, "What material is this bag made of?" The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This bag is made of genuine leather," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that allows the user to feel the texture of the bag. For example, the feedback device may transmit soft vibrations to the user's hand, recreating the texture of the bag.

[0505] Furthermore, when a user wants to use educational content, they can select product education mode. In this case, the device switches to education mode and the server loads the product education content. The AI ​​assistant provides quizzes and games for the user. For example, the AI ​​assistant may ask a question such as, "How many steps are involved in the manufacturing process of this bag?". When the user enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's correct!" or "Sorry, the correct answer is five steps."

[0506] Specific examples

[0507] Consider a scenario in which a user uses a virtual reality device to visit a virtual bag store. The system verifies login information and loads the virtual store's data. The AI ​​assistant begins guiding the user with a greeting: "Welcome to our bag specialty store." When the user asks, "What material is this bag made of?", the server provides the material information to the AI ​​assistant, who then begins a detailed explanation, saying, "This bag is made of genuine leather." The feedback device also reproduces the texture of the bag, allowing the user to experience its feel.

[0508] Example prompt sentence:

[0509] Simulate a scenario where a user asks, "What material is this bag made of?" The AI ​​assistant will provide detailed material information and realistically recreate tactile feedback using a haptic device.

[0510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0511] Step 1:

[0512] The user turns on the virtual reality device, and the virtual reality device and the terminal start up. The various sensors of the virtual reality device (position, movement, etc.) are adjusted to operate normally.

[0513] Input: Power supply for virtual reality device, user movement information.

[0514] Data processing / computation: The virtual reality device tracks the user's position and movement in real time and calibrates the sensors.

[0515] Output: The user's location and movement information is sent to the device.

[0516] Specific actions: The user puts on the virtual reality device and sets it up.

[0517] Step 2:

[0518] The terminal starts the software and accesses the login screen specified by the user. The user logs in by entering their user ID and password. The server verifies the login information and performs authentication.

[0519] Input: User ID, Password.

[0520] Data processing / calculation: The server performs user authentication and returns the authentication result.

[0521] Output: Login confirmation or error message is displayed on the terminal.

[0522] Specific behavior: The user enters login information, and the device sends the entered information to the server.

[0523] Step 3:

[0524] The user operates the interface to select the desired virtual store, and the terminal transmits the selection information to the server.

[0525] Input: User's store selection information.

[0526] Data processing / calculation: The server searches for the selected virtual store data (product information, video, audio, 3D models, etc.) and sends it to the terminal.

[0527] Output: Store data is sent to the terminal and loaded into the virtual reality device.

[0528] Specific operation: The user operates the interface to select a store, and the terminal sends this to the server.

[0529] Step 4:

[0530] The device launches the AI ​​assistant software, and the server provides store information and product data to the AI ​​assistant.

[0531] Input: Store information and product data.

[0532] Data processing / calculation: The AI ​​assistant analyzes store information and product data and prepares information to provide in an interactive format.

[0533] Output: Your AI assistant is now ready.

[0534] Specific operation: The software starts and receives data from the server.

[0535] Step 5:

[0536] A user explores a virtual store and asks a question about a specific product. The terminal sends the question to the server.

[0537] Input: The user's question.

[0538] Data processing / calculation: The server analyzes the question and sends the corresponding product data to the AI ​​assistant.

[0539] Output: Product information is sent to the device and the AI ​​assistant provides an answer.

[0540] How it works: The user asks the AI ​​assistant a question, and the device sends it to the server.

[0541] Step 6:

[0542] The terminal controls the feedback device to provide haptic feedback to the user, for example, the feedback device reproduces the texture of a bag.

[0543] Input: User operation information, product information.

[0544] Data processing / calculation: The feedback device generates appropriate haptic feedback based on product data.

[0545] Output: Haptic feedback is provided to the user.

[0546] Specific action: The feedback device transmits vibrations or pressure to the user's hand.

[0547] Step 7:

[0548] If the user selects the product education mode, the terminal switches to the education mode and loads the product education content from the server.

[0549] Input: User mode selection.

[0550] Data processing / calculation: The server sends the product educational content to the terminal, and the terminal loads it.

[0551] Output: Educational content is displayed on a virtual reality device.

[0552] Specific behavior: The user selects the education mode, and the device loads the data accordingly.

[0553] Step 8:

[0554] The AI ​​assistant provides quizzes and games for users, making the virtual shopping experience fun.

[0555] Input: Educational content, user answers.

[0556] Data processing / calculation: The AI ​​assistant will ask questions and evaluate the answers, providing feedback on the results.

[0557] Output: The quiz results and explanations are provided to the user.

[0558] What happens: The user takes a quiz and the AI ​​assistant responds accordingly.

[0559] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0560] The system of the present invention allows users to experience virtual travel through a virtual reality device, and further combines it with an emotion engine that recognizes the user's emotions. This system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[0561] 1. System startup and initial settings

[0562] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. The user can update their profile information if necessary.

[0563] 2. Starting and initializing the emotion engine

[0564] Next, the device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition technology and voice analysis technology to determine the user's emotions.

[0565] 3. Select your destination and load your data

[0566] The user operates the interface to select the desired travel destination. Information about the selected travel destination is sent from the device to the server as a request. The server receives the request, searches the database for data about the selected travel destination (video, audio, 3D model, etc.), and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0567] 4. Launching the AI ​​Assistant

[0568] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data for the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0569] 5. Real-time guidance, haptic feedback, and emotion engine integration

[0570] The user explores the VR space and shows interest in a specific place or building. The user asks the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0571] The emotion engine also analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored. Based on this, the feedback device is adjusted. For example, if the user is excited, the haptic feedback that feels like wind is enhanced. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[0572] 6. Providing educational experiences

[0573] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the emotion engine analyzes the user's emotions and optimizes how the educational content is presented.

[0574] Example: A scenario in which a user visits the Egyptian pyramids

[0575] A user selects the Egyptian pyramids and begins the VR experience. The AI ​​assistant begins by guiding them, saying, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of these pyramids," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback device enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[0576] ---

[0577] The system allows users to have a personalized and realistic travel experience through virtual reality devices and emotion engines without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[0578] The processing flow will be explained below.

[0579] Step 1:

[0580] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0581] Step 2:

[0582] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0583] Step 3:

[0584] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0585] Step 4:

[0586] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[0587] Step 5:

[0588] The user operates the interface to select the desired travel destination, and information about the selected destination is sent from the terminal to the server as a request.

[0589] Step 6:

[0590] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0591] Step 7:

[0592] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0593] Step 8:

[0594] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data (such as historical information) for the selected travel destination.

[0595] Step 9:

[0596] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0597] Step 10:

[0598] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0599] Step 11:

[0600] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0601] Step 12:

[0602] The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored, and adjusts the feedback device accordingly.

[0603] Step 13:

[0604] The terminal controls the haptic device to provide haptic feedback to the user, such as the sensation of wind. For example, if the user is excited, the haptic feedback of wind is enhanced.

[0605] Step 14:

[0606] When a user wants to use educational content, they select the educational mode for children. The device switches to educational mode, the server loads the educational content, and the AI ​​assistant provides guidance for children.

[0607] Step 15:

[0608] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0609] Step 16:

[0610] The emotion engine analyzes the child's emotions from their facial expressions and voice, and adjusts the difficulty and content of quizzes and games accordingly.

[0611] Step 17:

[0612] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0613] Step 18:

[0614] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0615] Example 2

[0616] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] The present invention relates to a system for providing a virtual travel experience. Conventional systems have difficulty in properly recognizing a user's emotional state and providing a personalized experience. Furthermore, conventional systems lack sufficient educational content and lack means to maintain a user's interest in the experience. This makes it difficult for users to obtain a deep learning experience.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0619] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the user's selected travel destination, an artificial intelligence means for responding to the user's questions in real time, a feedback means for providing feedback such as wind and tactile sensations to the user's virtual travel experience, an emotion analysis engine means for analyzing the user's facial expressions and voice in real time to determine the user's emotional state, and an education mode means for providing educational content. This makes it possible to personalize the virtual travel experience based on the user's emotional state and further optimize and provide educational content.

[0620] "Virtual reality device means" is a general term for hardware that includes devices and sensors for tracking a user's position and movements.

[0621] "Terminal means" refers to an electronic device that allows a user to select a travel destination and has the function of loading and displaying virtual reality data based on that selection.

[0622] "Artificial Intelligence Means" refers collectively to software and algorithms that respond to a user's questions in real time and provide information and guidance about the selected travel destination.

[0623] "Feedback means" are devices and systems that provide physical feedback, such as wind or haptics, to enhance the user's virtual travel experience.

[0624] The "emotion analysis engine means" is software and algorithms for analyzing the user's facial expressions and voice in real time and determining the user's emotional state.

[0625] "Educational mode means" refers to software and systems that have functionality for managing and optimizing educational content as it is experienced by a user.

[0626] The system of the present invention is designed for users to experience virtual travel through a virtual reality device, and integrates an emotion engine that recognizes the user's emotional state. The system includes the following components: a virtual reality device means, a terminal means, a server means, an artificial intelligence means, a feedback means, an emotion analysis engine means, and an education mode means.

[0627] When a user turns on the virtual reality device, the terminal starts the software, which adjusts the virtual reality device's various sensors (position sensors, motion sensors, haptic feedback devices, etc.) to ensure they are working properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. If necessary, the user can update their profile information.

[0628] Next, the device launches an emotion analysis engine to prepare for real-time analysis of the user's facial expressions and voice. The emotion analysis engine uses facial recognition and voice analysis technologies (e.g., OpenCV and Google Speech API) to determine the user's emotional state.

[0629] When a user operates the interface to select a desired travel destination, information about the selected destination is sent as a request from the device to the server. The server receives the request, searches for data about the selected destination (video, audio, 3D model, etc.) from a database (e.g., MySQL), and sends it to the device. The device receives this data and loads it into the virtual reality device, and the virtual environment of the destination is displayed in the user's field of view.

[0630] Furthermore, the device launches the AI ​​assistant software (e.g., Dialogflow). The server provides the AI ​​assistant with tourist information and special data for the selected travel destination. The AI ​​assistant begins guiding the user, saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0631] Additionally, if a user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. An emotion analysis engine analyzes the user's facial expressions and voice in real time to determine whether the user is excited, impressed, or bored. Based on this, the feedback method is adjusted. For example, if the user is excited, the haptic feedback device will enhance the haptic feedback that makes them feel the wind. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[0632] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the sentiment analysis engine analyzes the user's emotions and optimizes the way the educational content is presented.

[0633] As a concrete example, when a user selects the Egyptian pyramids and starts the VR experience, the AI ​​assistant begins by saying, "Welcome to the Egyptian pyramids." If the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback method enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[0634] Examples of prompts include:

[0635] "Tell me the history of the Egyptian pyramids."

[0636] "Tell me about the special exhibits."

[0637] "I want to know more about Kyoto festivals."

[0638] "I want to switch to education mode and learn."

[0639] The system allows users to have a personalized and realistic travel experience through virtual reality devices and an emotion analysis engine without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[0640] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0641] Step 1:

[0642] The user powers on the virtual reality device, which initializes the device's hardware and prepares the system to boot.

[0643] Input: Power-on by user operation

[0644] Data processing: Hardware initialization process

[0645] Output: System startup

[0646] Specific behavior:

[0647] The user presses and holds the power button on the VR headset, and the device starts up.

[0648] Step 2:

[0649] The device launches the connected software and adjusts the various sensors of the virtual reality device (position sensors, movement sensors, haptic feedback devices, etc.).

[0650] Input: Virtual reality device sensor data

[0651] Data processing: Sensor calibration

[0652] Output: Adjustment completion notification

[0653] Specific behavior:

[0654] The software on the device will automatically launch and calibrate the device's various sensors, using device drivers and calibration tools.

[0655] Step 3:

[0656] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[0657] Input: User ID and password

[0658] Data Processing: Credentials Verification

[0659] Output: Login approved or denied

[0660] Specific behavior:

[0661] The user enters their ID and password using the keyboard on the device display, and the input data is sent to the server, where authentication is performed by referencing a database.

[0662] Step 4:

[0663] The device launches an emotion analysis engine and prepares to analyze the user's facial expressions and voice in real time.

[0664] Input: facial expression data and voice data

[0665] Data processing: Real-time analysis

[0666] Output: Emotional state identification result

[0667] Specific behavior:

[0668] The device runs emotion analysis engine software, which collects data from the camera and microphone, and uses facial recognition and voice analysis technologies to determine the user's emotional state in real time.

[0669] Step 5:

[0670] The user operates the virtual travel interface to select a desired travel destination.

[0671] Input: User's travel destination selection

[0672] Data processing: Save selected data

[0673] Output: Generate a travel destination data request

[0674] Specific behavior:

[0675] The user operates the terminal interface and selects a destination from the list of travel destinations, which causes the data of the selected travel destination to be recorded in the terminal and prepares for sending a request to the server.

[0676] Step 6:

[0677] The device requests data on the selected travel destination from the server, the server searches for the corresponding data, and sends it to the device, which receives the data and loads it into the virtual reality device.

[0678] Input: Travel Destination Data Request

[0679] Data processing: Searching and sending travel destination data

[0680] Output: Loading travel destination data

[0681] Specific behavior:

[0682] The device sends an HTTP request to the server, which retrieves travel destination data from a database and returns a response in JSON format. The device receives the data, analyzes it, and displays it on the virtual reality device.

[0683] Step 7:

[0684] The device launches the AI ​​assistant software, the server provides tourist information and special data, and the AI ​​assistant starts guiding and interacting with the user.

[0685] Input: Destination data and tourist information

[0686] Data processing: scenario generation and dialogue management

[0687] Output: Guidance and dialogue start

[0688] Specific behavior:

[0689] The device launches the AI ​​assistant software and begins providing guidance based on the tourist information received from the server. The AI ​​assistant speaks to the user, saying, "Hello, welcome to the Egyptian pyramids!"

[0690] Step 8:

[0691] The user explores the VR space and asks questions at points of interest. The server analyzes the questions, and the AI ​​assistant provides answers. The emotion analysis engine analyzes the user's emotions, and the feedback method is adjusted accordingly.

[0692] Input: User question

[0693] Data processing: Question analysis and answer generation

[0694] Output: Answer to the question

[0695] Specific behavior:

[0696] When a user asks, "Tell me about the history of this pyramid," the server uses natural language processing technology to analyze the question and provide the AI ​​assistant with appropriate answer data. The emotion analysis engine analyzes facial expressions and tone of voice, and the feedback method adjusts haptic feedback. For example, it enhances the sensation of wind.

[0697] Step 9:

[0698] When a user selects education mode, the device switches to education mode and the server loads educational content. The AI ​​assistant provides quizzes and games for children and optimizes the delivery method using a sentiment analysis engine.

[0699] Input: Select Education Mode

[0700] Data processing: Loading and optimizing educational content

[0701] Output: Providing quizzes and games

[0702] Specific behavior:

[0703] When a user presses the education mode button, the device sends a mode change request to the server. The server loads educational content, and the AI ​​assistant asks a quiz question, such as "How many years did it take to build this pyramid?" and provides immediate feedback to the user after they answer. The sentiment analysis engine analyzes the user's level of interest in real time and selects the appropriate content delivery method.

[0704] (Application example 2)

[0705] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0706] Conventional virtual travel systems provide realistic experiences without requiring users to physically visit the destination. However, they are unable to adjust feedback based on the user's emotions and lack appropriate responses to emotions such as boredom and excitement. Furthermore, in physical stores, product information is provided in a one-way manner, and there is no interactive shopping experience based on the user's emotions. This results in a poor user experience and limited sales promotion effectiveness in physical stores.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the travel destination selection, an artificial intelligence means for responding to the user's questions in real time, a feedback device means for providing tactile feedback such as wind and vibration to the user's virtual travel experience, and an emotion engine means for analyzing the user's emotions in real time and adjusting the feedback content based on the analysis results. This not only enables appropriate feedback based on the user's emotions to be provided in real time during the virtual travel, but also improves the shopping experience in physical stores, piques the user's interest, and increases sales promotion effects.

[0708] A "virtual reality device" is a device that allows a user to experience a virtual reality space.

[0709] A "terminal" is a computer terminal that is connected to a virtual reality device and that loads and displays data.

[0710] "Artificial intelligence" refers to interactive computer programs that respond to users' questions and provide information in real time.

[0711] A "feedback device" is a device that provides tactile feedback, such as wind or vibration, to a user's virtual travel experience.

[0712] The "emotion engine" is a system that analyzes the user's emotions in real time and adjusts the feedback content based on the analysis results.

[0713] The system of the present invention is composed of the following elements to enable users to experience virtual travel and virtual shopping in physical stores through a virtual reality device: It has the function of analyzing the user's emotions and adjusting feedback individually based on those emotions.

[0714] System Overview

[0715] The system consists of a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[0716] Hardware and software used

[0717] Virtual reality device: The primary device through which the user experiences the virtual space, providing visual and audio output.

[0718] Terminal: A computer terminal that tracks the user's movements and location and loads and displays virtual reality data.

[0719] Server: Manages virtual reality data, tourist information, product information, etc., and provides it to terminals and virtual reality devices.

[0720] Artificial intelligence: Interactive software that runs on a device or server and responds to user questions in real time. The software used includes natural language processing techniques and machine learning algorithms.

[0721] Feedback devices: These devices provide haptic feedback to enhance the user's virtual experience, providing physical sensations such as wind and vibration.

[0722] Emotion Engine: A system that analyzes the user's emotions and adjusts the feedback content. The technologies used include facial recognition and voice analysis.

[0723] System Operation

[0724] 1. System startup and initial setup:

[0725] The user turns on the virtual reality device, and the device launches the software, which calibrates the virtual reality device's sensors (position, movement, and haptic feedback) to ensure they function properly.

[0726] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[0727] 2. Emotion Engine startup and initial setup:

[0728] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[0729] 3. Select your destination or product information and load your data:

[0730] The user operates the interface to select the desired travel destination or product information, and the selected information is sent from the terminal to the server as a request.

[0731] The server receives the request, searches for the relevant data from the database, and sends it to the terminal, which then receives the data, loads it into the virtual reality device, and displays the virtual environment and product information in the user's field of view.

[0732] 4. Artificial Intelligence Guidance and Interaction:

[0733] The device activates artificial intelligence to provide tourist and product information, and users can interact with the AI ​​based on where they want to see next and what information they want to hear.

[0734] 5. Real-time feedback and emotion engine integration:

[0735] Users explore the VR space and obtain information related to places and products they are interested in. The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited or bored, and adjusts the feedback accordingly.

[0736] Specific examples

[0737] Consider a scenario where a user is wearing smart glasses and shopping in a brick-and-mortar store: if the emotion engine determines that the user is "excited," the AI ​​assistant will provide enhanced details about the product and promotions.

[0738] When a user asks, "Tell me about this product," the AI ​​assistant provides detailed information, such as, "This is the latest smartphone, and its main features are..." If the emotion engine detects boredom, it will provide interesting information in the form of a quiz, encouraging interaction.

[0739] Prompt Sentence Examples

[0740] The following script is a virtual shopping assistant for brick-and-mortar stores that uses smart glasses. It includes a function to adjust the way information is presented depending on the user's emotions. As a concrete example, it shows a scenario in which detailed product information is provided when the user is excited.

[0741] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0742] Step 1:

[0743] The user turns on the virtual reality device. The device launches the software and calibrates the virtual reality device's sensors to ensure proper operation. The device displays a login screen to the user, prompting them to enter their user ID and password. The server then verifies the credentials and, if successful, logs the user into the system. The user's profile information is loaded onto the device.

[0744] Step 2:

[0745] The device activates the emotion engine, which uses facial recognition and voice analysis technologies to capture and analyze the user's facial expressions and voice in real time. The camera captures the user's facial expressions, and the voice input system collects the user's voice. Based on this input data, the emotion engine determines the user's emotions and sends the results to the feedback system.

[0746] Step 3:

[0747] The user operates the interface to select the desired travel destination or product information. The device receives this selection and sends a request to the server. The server searches the database for the corresponding virtual reality data (video, audio, 3D models, etc.) and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays it in the user's field of view.

[0748] Step 4:

[0749] The terminal activates the AI. The server provides tourist information, product information, etc. to the AI, and the AI ​​starts providing guidance to the user, such as "Hello, welcome to the Egyptian pyramids." When the user asks a question, the terminal sends the question to the server, and the server generates an appropriate answer to the question and conveys it to the user via the AI.

[0750] Step 5:

[0751] As the user progresses through the virtual journey and shopping, they will show interest in specific places and products. The emotion engine analyzes the user's facial expressions and voice to determine whether they are excited or bored. If the user is excited, the feedback device will enhance tactile feedback such as wind, and the AI ​​will provide additional interesting information and suggestions. Conversely, if the user is bored, the AI ​​will provide information in the form of quizzes and games to pique the user's interest.

[0752] Step 6:

[0753] If the user selects education mode as they continue their experience, the device will load educational content. The server provides educational content for children, and the AI ​​interacts with the user (child) through quizzes and games. Again, the emotion engine analyzes the user's emotions and adjusts feedback at the appropriate time. For example, the AI ​​assistant might ask a quiz question such as, "How many years did it take to build this pyramid?" and provide immediate feedback to the user's answer.

[0754] These steps make the virtual travel and shopping experience more personalized and engaging for the user.

[0755] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0756] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0757] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0758] [Third embodiment]

[0759] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0760] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0761] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0762] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0763] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0764] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0765] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0766] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0767] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0768] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0769] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0770] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0771] The system of the present invention allows users to experience virtual travel through a virtual reality device and obtain information in real time. The system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0772] 1. System startup and initial settings

[0773] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user then accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0774] 2. Select your destination and load your data

[0775] Next, the user operates the interface to select the desired travel destination. The device transmits this selection information to the server. The server retrieves data (video, audio, 3D models, etc.) related to the selected travel destination and transmits it to the device. The device receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0776] 3. Launching the AI ​​Assistant

[0777] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and responds to any questions or instructions the user may have.

[0778] 4. Real-time Guidance and Haptic Feedback

[0779] As the user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant a question. For example, they might ask, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that makes the user feel the wind. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0780] 5. Providing educational experiences

[0781] Furthermore, when a user wants to use educational content, they can select a children's educational mode. In this case, the device switches to educational mode and the server loads the educational content. The AI ​​assistant provides quizzes and games for children. For example, the AI ​​assistant may ask a question such as, "How many years did it take to build this pyramid?" and when the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0782] Example: A scenario in which a user visits the Egyptian pyramids

[0783] The user selects the Egyptian pyramids and the system loads the data. The AI ​​assistant begins the tour with a greeting, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then begins a detailed explanation. The terminal uses a feedback device to allow the user to experience the sensation of feeling the wind and walking on sand.

[0784] ---

[0785] This system allows users to experience real travel experiences through virtual reality devices without physically visiting the locations, and also provides educational content that allows users to have fun while learning.

[0786] The processing flow will be explained below.

[0787] Step 1:

[0788] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0789] Step 2:

[0790] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0791] Step 3:

[0792] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0793] Step 4:

[0794] The user operates the interface to select the desired travel destination, and the information on the selected travel destination is sent from the terminal to the server as a request.

[0795] Step 5:

[0796] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0797] Step 6:

[0798] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0799] Step 7:

[0800] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data for the selected travel destination.

[0801] Step 8:

[0802] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0803] Step 9:

[0804] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0805] Step 10:

[0806] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0807] Step 11:

[0808] The device controls the haptic device to provide tactile feedback, such as the sensation of wind, by using the vibration motor of the VR device to transmit gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0809] Step 12:

[0810] The user (or parent) selects the educational mode for children. The device switches to educational mode, the server loads educational content, and the AI ​​assistant provides guidance for the child.

[0811] Step 13:

[0812] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0813] Step 14:

[0814] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0815] Step 15:

[0816] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0817] Example 1

[0818] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0819] While conventional virtual reality systems allow users to experience virtual travel without physically visiting their destinations, they lack a system that provides real-time information. Furthermore, it is difficult to provide an experience that includes not only visual but also tactile sensations, and educational content is limited, limiting the user experience. Furthermore, it is difficult to introduce an artificial intelligence system that responds to user questions and instructions in real time, which hinders the provision of a comprehensive experience.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0821] In this invention, the server includes a visual display device that tracks the user's location and movements, an information processing device that loads and displays virtual data based on the user's travel destination selection, a knowledge processing device that responds to the user's questions in real time, a sensory presentation device that provides tactile feedback such as wind and vibration to the user's virtual travel experience, a language processing system that provides tourist destination information and special information through speech, and a learning support system that manages and provides educational content to complement the user's experience. This makes it possible to provide detailed information in real time and provide the user with a tactile experience. It also enriches the educational content and supports the user's learning.

[0822] A "visual display device" is a device that tracks the user's position and movements in real time and displays images of a virtual reality space.

[0823] The "information processing device" is a device that processes data to load and display the necessary virtual data based on the selection of travel destinations.

[0824] A "knowledge processing device" is an artificial intelligence system that responds to user questions in real time and provides appropriate information.

[0825] A "sensory presentation device" is a device that provides tactile feedback such as wind or vibration during a user's virtual travel experience.

[0826] A "language processing system" is an artificial intelligence system that provides tourist information and special information to users through speech.

[0827] A "learning support system" is a system that complements the user experience and manages and provides educational content.

[0828] The present invention provides a system that allows users to experience virtual travel and acquire information in real time through a virtual reality device. The system includes, as its main components, a visual display device, an information processing device, a knowledge processing device, a sensory presentation device, a language processing system, and a learning support system.

[0829] When a user turns on the virtual reality device, the device automatically launches the software. Specifically, the visual display device tracks the user's position and movements and transmits the data to the device. Based on this, the device adjusts and displays the image in the virtual reality space appropriately.

[0830] Next, the user operates the interface to select a desired travel destination, and the terminal transmits this selection information to the server. The server receives this information, retrieves data (video, audio, 3D model, etc.) related to the selected travel destination, and transmits it to the terminal. The terminal receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0831] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). Using a language processing system, the AI ​​assistant can begin guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and respond to the user's questions and instructions.

[0832] For example, if a user asks, "Tell me about the history of this pyramid," the server analyzes the question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. The device also controls a sensory presentation device to provide haptic feedback to recreate the sensation of the user feeling the wind or walking on sand. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[0833] Furthermore, when a user uses educational content, the learning support system switches to a children's education mode. In this case, the server loads educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant then asks the user educational questions, such as, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[0834] As a concrete example, consider a scenario in which a user selects the pyramids of Egypt. The server sends data about the pyramids, including video, audio, and 3D models, to the terminal, which then loads it into a virtual reality device. The AI ​​assistant begins a tour with a greeting like "Welcome to the pyramids of Egypt," and provides detailed explanations in response to the user's questions. The terminal uses a sensory presentation device to give the user the sensation of feeling the wind or walking on sand.

[0835] By providing prompts like the following to the generative AI model, you can provide appropriate guidance and feedback:

[0836] Tell me about the Egyptian pyramids.

[0837] This system allows users to experience real travel through virtual reality devices without physically visiting the destination, and also provides educational content that allows users to have fun while learning.

[0838] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0839] Step 1:

[0840] When a user powers on the virtual reality device, the device automatically launches the software. The visual display device collects data from built-in sensors to track the user's position and movements. This data serves as input for displaying a virtual space that updates in real time in response to the user's movements. The device processes this sensor data and sends it to the visual display device to generate output that displays the virtual reality environment to the user.

[0841] Step 2:

[0842] The user accesses the specified login screen and enters their user ID and password. This information is sent from the terminal to the server, which then verifies the credentials. The server compares the entered credentials with the user information in its database, and if authentication is successful, approves the login. This process allows the user to log in to the system, and the terminal generates output to move on to the next step.

[0843] Step 3:

[0844] The user operates the interface to select the desired travel destination. The user's selection information is sent from the terminal to the server. Based on this input information, the server searches the database for data (video, audio, 3D models, etc.) related to the corresponding travel destination. The server then sends this data to the terminal, which then loads the received data into the virtual reality device in real time. This process results in an output that displays the virtual environment of the travel destination in the user's field of vision.

[0845] Step 4:

[0846] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data (such as historical information) about the selected travel destination. This data is used as input for the AI ​​assistant, which analyzes it and generates a guide message such as "Hello, welcome to the Egyptian pyramids!". Depending on the user's input (questions or instructions), the AI ​​assistant generates an appropriate response. This response is output to the user as audio.

[0847] Step 5:

[0848] As the user explores the VR space and shows interest in a particular place or building, they can ask the AI ​​assistant a question. This input information is sent from the device to the server, which then analyzes the user's question. For example, if the user asks, "Tell me about the history of this pyramid," the server searches the database for the appropriate information and sends it to the AI ​​assistant. The AI ​​assistant then responds, "This pyramid was built in 2560 BC," and generates a detailed explanation. This answer is output to the user via voice.

[0849] Step 6:

[0850] The device controls the sensory presentation device to provide haptic feedback to the user, recreating the sensation of feeling the wind or walking on sand. For example, the device controls the vibration motor of the VR device to transmit gentle vibrations to the user's feet. Based on this input data, the device calculates vibration patterns and controls actual movements. This process generates haptic output that further immerses the user.

[0851] Step 7:

[0852] When a user uses educational content, the learning support system switches to a children's education mode. The server loads the educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant asks the user educational questions. Based on this input, the user enters an answer, and the AI ​​assistant provides immediate feedback. For example, it might ask a question like, "How many years did it take to build this pyramid?" and respond to the user's answer with a response like, "That's right!" or "Sorry, the correct answer is 20 years." This process generates an output that enhances the user's educational experience.

[0853] (Application example 1)

[0854] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0855] While existing virtual travel systems were effective in allowing users to virtually experience specific tourist destinations, there were no systems specifically designed for virtual shopping experiences. This made it difficult for users to check out products and obtain detailed information without visiting a real store. Furthermore, they lacked the ability to provide haptic feedback about products, creating a need for a more realistic shopping experience that cannot be achieved through online shopping.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0857] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for allowing the user to select a virtual store and for loading and displaying store information, an artificial intelligence means for responding to the user's questions in real time and providing detailed product information, and a feedback device means for providing haptic feedback about the products in the user's virtual shopping experience, thereby enabling the user to experience shopping in a virtual store without going to a real store and to obtain detailed product information and haptic feedback in real time.

[0858] "User" refers to an individual who utilizes the system to experience virtual reality.

[0859] A "virtual reality device" is a device for tracking a user's position and movements and displaying a virtual environment.

[0860] "Terminal means" refers to a device that allows a user to select a virtual store and load and display store information.

[0861] "Artificial intelligence means" refers to algorithms and software that respond to user questions in real time and provide detailed product information.

[0862] "Feedback device means" refers to devices or technology that provide tactile feedback about products in a user's virtual shopping experience.

[0863] A "virtual store" is a store that exists in a virtual space and provides a virtual environment where users can visit, check out products, and purchase them.

[0864] "Product information" refers to detailed descriptions and specifications of products that users can view in the virtual store.

[0865] "Haptic feedback" refers to technology that allows users to sense virtual physical properties (e.g., texture or temperature).

[0866] "Virtual shopping experience" refers to the experience of a user visiting a store in virtual reality, browsing, selecting, and purchasing products.

[0867] "Real-time response" means providing appropriate information and feedback immediately in response to a user's questions or operations.

[0868] The system of the present invention allows users to visit a virtual store through a virtual reality device and obtain detailed product information in real time. The system includes a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[0869] When the system starts up, the user turns on the virtual reality device and the device launches the software. The various sensors that track the virtual reality device's position and movement are adjusted to ensure proper operation. The user accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[0870] Next, the user operates the interface to select the desired virtual store. The terminal transmits this selection information to the server. The server retrieves data related to the selected virtual store (product information, video, audio, 3D models, etc.) and transmits it to the terminal. The terminal receives this data, loads it into the virtual reality device, and displays the virtual store environment in the user's field of view.

[0871] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with store information and special data about the selected product (such as material information). The AI ​​assistant begins guiding the user, saying, "Welcome to our bag specialty store!" and responds to any questions or instructions the user may have.

[0872] As a user explores the virtual store and shows interest in a specific product, they can ask the AI ​​assistant a question. For example, they might ask, "What material is this bag made of?" The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This bag is made of genuine leather," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that allows the user to feel the texture of the bag. For example, the feedback device may transmit soft vibrations to the user's hand, recreating the texture of the bag.

[0873] Furthermore, when a user wants to use educational content, they can select product education mode. In this case, the device switches to education mode and the server loads the product education content. The AI ​​assistant provides quizzes and games for the user. For example, the AI ​​assistant may ask a question such as, "How many steps are involved in the manufacturing process of this bag?". When the user enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's correct!" or "Sorry, the correct answer is five steps."

[0874] Specific examples

[0875] Consider a scenario in which a user uses a virtual reality device to visit a virtual bag store. The system verifies login information and loads the virtual store's data. The AI ​​assistant begins guiding the user with a greeting: "Welcome to our bag specialty store." When the user asks, "What material is this bag made of?", the server provides the material information to the AI ​​assistant, who then begins a detailed explanation, saying, "This bag is made of genuine leather." The feedback device also reproduces the texture of the bag, allowing the user to experience its feel.

[0876] Example prompt sentence:

[0877] Simulate a scenario where a user asks, "What material is this bag made of?" The AI ​​assistant will provide detailed material information and realistically recreate tactile feedback using a haptic device.

[0878] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0879] Step 1:

[0880] The user turns on the virtual reality device, and the virtual reality device and the terminal start up. The various sensors of the virtual reality device (position, movement, etc.) are adjusted to operate normally.

[0881] Input: Power supply for virtual reality device, user movement information.

[0882] Data processing / computation: The virtual reality device tracks the user's position and movement in real time and calibrates the sensors.

[0883] Output: The user's location and movement information is sent to the device.

[0884] Specific actions: The user puts on the virtual reality device and sets it up.

[0885] Step 2:

[0886] The terminal starts the software and accesses the login screen specified by the user. The user logs in by entering their user ID and password. The server verifies the login information and performs authentication.

[0887] Input: User ID, Password.

[0888] Data processing / calculation: The server performs user authentication and returns the authentication result.

[0889] Output: Login confirmation or error message is displayed on the terminal.

[0890] Specific behavior: The user enters login information, and the device sends the entered information to the server.

[0891] Step 3:

[0892] The user operates the interface to select the desired virtual store, and the terminal transmits the selection information to the server.

[0893] Input: User's store selection information.

[0894] Data processing / calculation: The server searches for the selected virtual store data (product information, video, audio, 3D models, etc.) and sends it to the terminal.

[0895] Output: Store data is sent to the terminal and loaded into the virtual reality device.

[0896] Specific operation: The user operates the interface to select a store, and the terminal sends this to the server.

[0897] Step 4:

[0898] The device launches the AI ​​assistant software, and the server provides store information and product data to the AI ​​assistant.

[0899] Input: Store information and product data.

[0900] Data processing / calculation: The AI ​​assistant analyzes store information and product data and prepares information to provide in an interactive format.

[0901] Output: Your AI assistant is now ready.

[0902] Specific operation: The software starts and receives data from the server.

[0903] Step 5:

[0904] A user explores a virtual store and asks a question about a specific product. The terminal sends the question to the server.

[0905] Input: The user's question.

[0906] Data processing / calculation: The server analyzes the question and sends the corresponding product data to the AI ​​assistant.

[0907] Output: Product information is sent to the device and the AI ​​assistant provides an answer.

[0908] How it works: The user asks the AI ​​assistant a question, and the device sends it to the server.

[0909] Step 6:

[0910] The terminal controls the feedback device to provide haptic feedback to the user, for example, the feedback device reproduces the texture of a bag.

[0911] Input: User operation information, product information.

[0912] Data processing / calculation: The feedback device generates appropriate haptic feedback based on product data.

[0913] Output: Haptic feedback is provided to the user.

[0914] Specific action: The feedback device transmits vibrations or pressure to the user's hand.

[0915] Step 7:

[0916] If the user selects the product education mode, the terminal switches to the education mode and loads the product education content from the server.

[0917] Input: User mode selection.

[0918] Data processing / calculation: The server sends the product educational content to the terminal, and the terminal loads it.

[0919] Output: Educational content is displayed on a virtual reality device.

[0920] Specific behavior: The user selects the education mode, and the device loads the data accordingly.

[0921] Step 8:

[0922] The AI ​​assistant provides quizzes and games for users, making the virtual shopping experience fun.

[0923] Input: Educational content, user answers.

[0924] Data processing / calculation: The AI ​​assistant will ask questions and evaluate the answers, providing feedback on the results.

[0925] Output: The quiz results and explanations are provided to the user.

[0926] What happens: The user takes a quiz and the AI ​​assistant responds accordingly.

[0927] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0928] The system of the present invention allows users to experience virtual travel through a virtual reality device, and further combines it with an emotion engine that recognizes the user's emotions. This system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[0929] 1. System startup and initial settings

[0930] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. The user can update their profile information if necessary.

[0931] 2. Starting and initializing the emotion engine

[0932] Next, the device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition technology and voice analysis technology to determine the user's emotions.

[0933] 3. Select your destination and load your data

[0934] The user operates the interface to select the desired travel destination. Information about the selected travel destination is sent from the device to the server as a request. The server receives the request, searches the database for data about the selected travel destination (video, audio, 3D model, etc.), and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[0935] 4. Launching the AI ​​Assistant

[0936] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data for the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0937] 5. Real-time guidance, haptic feedback, and emotion engine integration

[0938] The user explores the VR space and shows interest in a specific place or building. The user asks the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0939] The emotion engine also analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored. Based on this, the feedback device is adjusted. For example, if the user is excited, the haptic feedback that feels like wind is enhanced. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[0940] 6. Providing educational experiences

[0941] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the emotion engine analyzes the user's emotions and optimizes how the educational content is presented.

[0942] Example: A scenario in which a user visits the Egyptian pyramids

[0943] A user selects the Egyptian pyramids and begins the VR experience. The AI ​​assistant begins by guiding them, saying, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of these pyramids," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback device enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[0944] ---

[0945] The system allows users to have a personalized and realistic travel experience through virtual reality devices and emotion engines without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[0946] The processing flow will be explained below.

[0947] Step 1:

[0948] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[0949] Step 2:

[0950] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[0951] Step 3:

[0952] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[0953] Step 4:

[0954] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[0955] Step 5:

[0956] The user operates the interface to select the desired travel destination, and information about the selected destination is sent from the terminal to the server as a request.

[0957] Step 6:

[0958] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[0959] Step 7:

[0960] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[0961] Step 8:

[0962] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data (such as historical information) for the selected travel destination.

[0963] Step 9:

[0964] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[0965] Step 10:

[0966] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[0967] Step 11:

[0968] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[0969] Step 12:

[0970] The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored, and adjusts the feedback device accordingly.

[0971] Step 13:

[0972] The terminal controls the haptic device to provide haptic feedback to the user, such as the sensation of wind. For example, if the user is excited, the haptic feedback of wind is enhanced.

[0973] Step 14:

[0974] When a user wants to use educational content, they select the educational mode for children. The device switches to educational mode, the server loads the educational content, and the AI ​​assistant provides guidance for children.

[0975] Step 15:

[0976] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[0977] Step 16:

[0978] The emotion engine analyzes the child's emotions from their facial expressions and voice, and adjusts the difficulty and content of quizzes and games accordingly.

[0979] Step 17:

[0980] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[0981] Step 18:

[0982] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[0983] Example 2

[0984] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] The present invention relates to a system for providing a virtual travel experience. Conventional systems have difficulty in properly recognizing a user's emotional state and providing a personalized experience. Furthermore, conventional systems lack sufficient educational content and lack means to maintain a user's interest in the experience. This makes it difficult for users to obtain a deep learning experience.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0987] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the user's selected travel destination, an artificial intelligence means for responding to the user's questions in real time, a feedback means for providing feedback such as wind and tactile sensations to the user's virtual travel experience, an emotion analysis engine means for analyzing the user's facial expressions and voice in real time to determine the user's emotional state, and an education mode means for providing educational content. This makes it possible to personalize the virtual travel experience based on the user's emotional state and further optimize and provide educational content.

[0988] "Virtual reality device means" is a general term for hardware that includes devices and sensors for tracking a user's position and movements.

[0989] "Terminal means" refers to an electronic device that allows a user to select a travel destination and has the function of loading and displaying virtual reality data based on that selection.

[0990] "Artificial Intelligence Means" refers collectively to software and algorithms that respond to a user's questions in real time and provide information and guidance about the selected travel destination.

[0991] "Feedback means" are devices and systems that provide physical feedback, such as wind or haptics, to enhance the user's virtual travel experience.

[0992] The "emotion analysis engine means" is software and algorithms for analyzing the user's facial expressions and voice in real time and determining the user's emotional state.

[0993] "Educational mode means" refers to software and systems that have functionality for managing and optimizing educational content as it is experienced by a user.

[0994] The system of the present invention is designed for users to experience virtual travel through a virtual reality device, and integrates an emotion engine that recognizes the user's emotional state. The system includes the following components: a virtual reality device means, a terminal means, a server means, an artificial intelligence means, a feedback means, an emotion analysis engine means, and an education mode means.

[0995] When a user turns on the virtual reality device, the terminal starts the software, which adjusts the virtual reality device's various sensors (position sensors, motion sensors, haptic feedback devices, etc.) to ensure they are working properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. If necessary, the user can update their profile information.

[0996] Next, the device launches an emotion analysis engine to prepare for real-time analysis of the user's facial expressions and voice. The emotion analysis engine uses facial recognition and voice analysis technologies (e.g., OpenCV and Google Speech API) to determine the user's emotional state.

[0997] When a user operates the interface to select a desired travel destination, information about the selected destination is sent as a request from the device to the server. The server receives the request, searches for data about the selected destination (video, audio, 3D model, etc.) from a database (e.g., MySQL), and sends it to the device. The device receives this data and loads it into the virtual reality device, and the virtual environment of the destination is displayed in the user's field of view.

[0998] Furthermore, the device launches the AI ​​assistant software (e.g., Dialogflow). The server provides the AI ​​assistant with tourist information and special data for the selected travel destination. The AI ​​assistant begins guiding the user, saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[0999] Additionally, if a user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. An emotion analysis engine analyzes the user's facial expressions and voice in real time to determine whether the user is excited, impressed, or bored. Based on this, the feedback method is adjusted. For example, if the user is excited, the haptic feedback device will enhance the haptic feedback that makes them feel the wind. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[1000] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the sentiment analysis engine analyzes the user's emotions and optimizes the way the educational content is presented.

[1001] As a concrete example, when a user selects the Egyptian pyramids and starts the VR experience, the AI ​​assistant begins by saying, "Welcome to the Egyptian pyramids." If the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback method enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[1002] Examples of prompts include:

[1003] "Tell me the history of the Egyptian pyramids."

[1004] "Tell me about the special exhibits."

[1005] "I want to know more about Kyoto festivals."

[1006] "I want to switch to education mode and learn."

[1007] The system allows users to have a personalized and realistic travel experience through virtual reality devices and an emotion analysis engine without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[1008] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1009] Step 1:

[1010] The user powers on the virtual reality device, which initializes the device's hardware and prepares the system to boot.

[1011] Input: Power-on by user operation

[1012] Data processing: Hardware initialization process

[1013] Output: System startup

[1014] Specific behavior:

[1015] The user presses and holds the power button on the VR headset, and the device starts up.

[1016] Step 2:

[1017] The device launches the connected software and adjusts the various sensors of the virtual reality device (position sensors, movement sensors, haptic feedback devices, etc.).

[1018] Input: Virtual reality device sensor data

[1019] Data processing: Sensor calibration

[1020] Output: Adjustment completion notification

[1021] Specific behavior:

[1022] The software on the device will automatically launch and calibrate the device's various sensors, using device drivers and calibration tools.

[1023] Step 3:

[1024] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[1025] Input: User ID and password

[1026] Data Processing: Credentials Verification

[1027] Output: Login approved or denied

[1028] Specific behavior:

[1029] The user enters their ID and password using the keyboard on the device display, and the input data is sent to the server, where authentication is performed by referencing a database.

[1030] Step 4:

[1031] The device launches an emotion analysis engine and prepares to analyze the user's facial expressions and voice in real time.

[1032] Input: facial expression data and voice data

[1033] Data processing: Real-time analysis

[1034] Output: Emotional state identification result

[1035] Specific behavior:

[1036] The device runs emotion analysis engine software, which collects data from the camera and microphone, and uses facial recognition and voice analysis technologies to determine the user's emotional state in real time.

[1037] Step 5:

[1038] The user operates the virtual travel interface to select a desired travel destination.

[1039] Input: User's travel destination selection

[1040] Data processing: Save selected data

[1041] Output: Generate a travel destination data request

[1042] Specific behavior:

[1043] The user operates the terminal interface and selects a destination from the list of travel destinations, which causes the data of the selected travel destination to be recorded in the terminal and prepares for sending a request to the server.

[1044] Step 6:

[1045] The device requests data on the selected travel destination from the server, the server searches for the corresponding data, and sends it to the device, which receives the data and loads it into the virtual reality device.

[1046] Input: Travel Destination Data Request

[1047] Data processing: Searching and sending travel destination data

[1048] Output: Loading travel destination data

[1049] Specific behavior:

[1050] The device sends an HTTP request to the server, which retrieves travel destination data from a database and returns a response in JSON format. The device receives the data, analyzes it, and displays it on the virtual reality device.

[1051] Step 7:

[1052] The device launches the AI ​​assistant software, the server provides tourist information and special data, and the AI ​​assistant starts guiding and interacting with the user.

[1053] Input: Destination data and tourist information

[1054] Data processing: scenario generation and dialogue management

[1055] Output: Guidance and dialogue start

[1056] Specific behavior:

[1057] The device launches the AI ​​assistant software and begins providing guidance based on the tourist information received from the server. The AI ​​assistant speaks to the user, saying, "Hello, welcome to the Egyptian pyramids!"

[1058] Step 8:

[1059] The user explores the VR space and asks questions at points of interest. The server analyzes the questions, and the AI ​​assistant provides answers. The emotion analysis engine analyzes the user's emotions, and the feedback method is adjusted accordingly.

[1060] Input: User question

[1061] Data processing: Question analysis and answer generation

[1062] Output: Answer to the question

[1063] Specific behavior:

[1064] When a user asks, "Tell me about the history of this pyramid," the server uses natural language processing technology to analyze the question and provide the AI ​​assistant with appropriate answer data. The emotion analysis engine analyzes facial expressions and tone of voice, and the feedback method adjusts haptic feedback. For example, it enhances the sensation of wind.

[1065] Step 9:

[1066] When a user selects education mode, the device switches to education mode and the server loads educational content. The AI ​​assistant provides quizzes and games for children and optimizes the delivery method using a sentiment analysis engine.

[1067] Input: Select Education Mode

[1068] Data processing: Loading and optimizing educational content

[1069] Output: Providing quizzes and games

[1070] Specific behavior:

[1071] When a user presses the education mode button, the device sends a mode change request to the server. The server loads educational content, and the AI ​​assistant asks a quiz question, such as "How many years did it take to build this pyramid?" and provides immediate feedback to the user after they answer. The sentiment analysis engine analyzes the user's level of interest in real time and selects the appropriate content delivery method.

[1072] (Application example 2)

[1073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1074] Conventional virtual travel systems provide realistic experiences without requiring users to physically visit the destination. However, they are unable to adjust feedback based on the user's emotions and lack appropriate responses to emotions such as boredom and excitement. Furthermore, in physical stores, product information is provided in a one-way manner, and there is no interactive shopping experience based on the user's emotions. This results in a poor user experience and limited sales promotion effectiveness in physical stores.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the travel destination selection, an artificial intelligence means for responding to the user's questions in real time, a feedback device means for providing tactile feedback such as wind and vibration to the user's virtual travel experience, and an emotion engine means for analyzing the user's emotions in real time and adjusting the feedback content based on the analysis results. This not only enables appropriate feedback based on the user's emotions to be provided in real time during the virtual travel, but also improves the shopping experience in physical stores, piques the user's interest, and increases sales promotion effects.

[1076] A "virtual reality device" is a device that allows a user to experience a virtual reality space.

[1077] A "terminal" is a computer terminal that is connected to a virtual reality device and that loads and displays data.

[1078] "Artificial intelligence" refers to interactive computer programs that respond to users' questions and provide information in real time.

[1079] A "feedback device" is a device that provides tactile feedback, such as wind or vibration, to a user's virtual travel experience.

[1080] The "emotion engine" is a system that analyzes the user's emotions in real time and adjusts the feedback content based on the analysis results.

[1081] The system of the present invention is composed of the following elements to enable users to experience virtual travel and virtual shopping in physical stores through a virtual reality device: It has the function of analyzing the user's emotions and adjusting feedback individually based on those emotions.

[1082] System Overview

[1083] The system consists of a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[1084] Hardware and software used

[1085] Virtual reality device: The primary device through which the user experiences the virtual space, providing visual and audio output.

[1086] Terminal: A computer terminal that tracks the user's movements and location and loads and displays virtual reality data.

[1087] Server: Manages virtual reality data, tourist information, product information, etc., and provides it to terminals and virtual reality devices.

[1088] Artificial intelligence: Interactive software that runs on a device or server and responds to user questions in real time. The software used includes natural language processing techniques and machine learning algorithms.

[1089] Feedback devices: These devices provide haptic feedback to enhance the user's virtual experience, providing physical sensations such as wind and vibration.

[1090] Emotion Engine: A system that analyzes the user's emotions and adjusts the feedback content. The technologies used include facial recognition and voice analysis.

[1091] System Operation

[1092] 1. System startup and initial setup:

[1093] The user turns on the virtual reality device, and the device launches the software, which calibrates the virtual reality device's sensors (position, movement, and haptic feedback) to ensure they function properly.

[1094] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[1095] 2. Emotion Engine startup and initial setup:

[1096] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[1097] 3. Select your destination or product information and load your data:

[1098] The user operates the interface to select the desired travel destination or product information, and the selected information is sent from the terminal to the server as a request.

[1099] The server receives the request, searches for the relevant data from the database, and sends it to the terminal, which then receives the data, loads it into the virtual reality device, and displays the virtual environment and product information in the user's field of view.

[1100] 4. Artificial Intelligence Guidance and Interaction:

[1101] The device activates artificial intelligence to provide tourist and product information, and users can interact with the AI ​​based on where they want to see next and what information they want to hear.

[1102] 5. Real-time feedback and emotion engine integration:

[1103] Users explore the VR space and obtain information related to places and products they are interested in. The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited or bored, and adjusts the feedback accordingly.

[1104] Specific examples

[1105] Consider a scenario where a user is wearing smart glasses and shopping in a brick-and-mortar store: if the emotion engine determines that the user is "excited," the AI ​​assistant will provide enhanced details about the product and promotions.

[1106] When a user asks, "Tell me about this product," the AI ​​assistant provides detailed information, such as, "This is the latest smartphone, and its main features are..." If the emotion engine detects boredom, it will provide interesting information in the form of a quiz, encouraging interaction.

[1107] Prompt Sentence Examples

[1108] The following script is a virtual shopping assistant for brick-and-mortar stores that uses smart glasses. It includes a function to adjust the way information is presented depending on the user's emotions. As a concrete example, it shows a scenario in which detailed product information is provided when the user is excited.

[1109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1110] Step 1:

[1111] The user turns on the virtual reality device. The device launches the software and calibrates the virtual reality device's sensors to ensure proper operation. The device displays a login screen to the user, prompting them to enter their user ID and password. The server then verifies the credentials and, if successful, logs the user into the system. The user's profile information is loaded onto the device.

[1112] Step 2:

[1113] The device activates the emotion engine, which uses facial recognition and voice analysis technologies to capture and analyze the user's facial expressions and voice in real time. The camera captures the user's facial expressions, and the voice input system collects the user's voice. Based on this input data, the emotion engine determines the user's emotions and sends the results to the feedback system.

[1114] Step 3:

[1115] The user operates the interface to select the desired travel destination or product information. The device receives this selection and sends a request to the server. The server searches the database for the corresponding virtual reality data (video, audio, 3D models, etc.) and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays it in the user's field of view.

[1116] Step 4:

[1117] The terminal activates the AI. The server provides tourist information, product information, etc. to the AI, and the AI ​​starts providing guidance to the user, such as "Hello, welcome to the Egyptian pyramids." When the user asks a question, the terminal sends the question to the server, and the server generates an appropriate answer to the question and conveys it to the user via the AI.

[1118] Step 5:

[1119] As the user progresses through the virtual journey and shopping, they will show interest in specific places and products. The emotion engine analyzes the user's facial expressions and voice to determine whether they are excited or bored. If the user is excited, the feedback device will enhance tactile feedback such as wind, and the AI ​​will provide additional interesting information and suggestions. Conversely, if the user is bored, the AI ​​will provide information in the form of quizzes and games to pique the user's interest.

[1120] Step 6:

[1121] If the user selects education mode as they continue their experience, the device will load educational content. The server provides educational content for children, and the AI ​​interacts with the user (child) through quizzes and games. Again, the emotion engine analyzes the user's emotions and adjusts feedback at the appropriate time. For example, the AI ​​assistant might ask a quiz question such as, "How many years did it take to build this pyramid?" and provide immediate feedback to the user's answer.

[1122] These steps make the virtual travel and shopping experience more personalized and engaging for the user.

[1123] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1124] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1125] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1126] [Fourth embodiment]

[1127] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1128] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1129] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1130] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1131] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1133] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1134] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1135] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1136] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1137] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1138] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1139] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1140] The system of the present invention allows users to experience virtual travel through a virtual reality device and obtain information in real time. The system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[1141] 1. System startup and initial settings

[1142] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user then accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[1143] 2. Select your destination and load your data

[1144] Next, the user operates the interface to select the desired travel destination. The device transmits this selection information to the server. The server retrieves data (video, audio, 3D models, etc.) related to the selected travel destination and transmits it to the device. The device receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[1145] 3. Launching the AI ​​Assistant

[1146] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and responds to any questions or instructions the user may have.

[1147] 4. Real-time Guidance and Haptic Feedback

[1148] As the user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant a question. For example, they might ask, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that makes the user feel the wind. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[1149] 5. Providing educational experiences

[1150] Furthermore, when a user wants to use educational content, they can select a children's educational mode. In this case, the device switches to educational mode and the server loads the educational content. The AI ​​assistant provides quizzes and games for children. For example, the AI ​​assistant may ask a question such as, "How many years did it take to build this pyramid?" and when the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[1151] Example: A scenario in which a user visits the Egyptian pyramids

[1152] The user selects the Egyptian pyramids and the system loads the data. The AI ​​assistant begins the tour with a greeting, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then begins a detailed explanation. The terminal uses a feedback device to allow the user to experience the sensation of feeling the wind and walking on sand.

[1153] ---

[1154] This system allows users to experience real travel experiences through virtual reality devices without physically visiting the locations, and also provides educational content that allows users to have fun while learning.

[1155] The processing flow will be explained below.

[1156] Step 1:

[1157] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[1158] Step 2:

[1159] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[1160] Step 3:

[1161] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[1162] Step 4:

[1163] The user operates the interface to select the desired travel destination, and the information on the selected travel destination is sent from the terminal to the server as a request.

[1164] Step 5:

[1165] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[1166] Step 6:

[1167] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[1168] Step 7:

[1169] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data for the selected travel destination.

[1170] Step 8:

[1171] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[1172] Step 9:

[1173] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[1174] Step 10:

[1175] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[1176] Step 11:

[1177] The device controls the haptic device to provide tactile feedback, such as the sensation of wind, by using the vibration motor of the VR device to transmit gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[1178] Step 12:

[1179] The user (or parent) selects the educational mode for children. The device switches to educational mode, the server loads educational content, and the AI ​​assistant provides guidance for the child.

[1180] Step 13:

[1181] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[1182] Step 14:

[1183] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[1184] Step 15:

[1185] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[1186] Example 1

[1187] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1188] While conventional virtual reality systems allow users to experience virtual travel without physically visiting their destinations, they lack a system that provides real-time information. Furthermore, it is difficult to provide an experience that includes not only visual but also tactile sensations, and educational content is limited, limiting the user experience. Furthermore, it is difficult to introduce an artificial intelligence system that responds to user questions and instructions in real time, which hinders the provision of a comprehensive experience.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1190] In this invention, the server includes a visual display device that tracks the user's location and movements, an information processing device that loads and displays virtual data based on the user's travel destination selection, a knowledge processing device that responds to the user's questions in real time, a sensory presentation device that provides tactile feedback such as wind and vibration to the user's virtual travel experience, a language processing system that provides tourist destination information and special information through speech, and a learning support system that manages and provides educational content to complement the user's experience. This makes it possible to provide detailed information in real time and provide the user with a tactile experience. It also enriches the educational content and supports the user's learning.

[1191] A "visual display device" is a device that tracks the user's position and movements in real time and displays images of a virtual reality space.

[1192] The "information processing device" is a device that processes data to load and display the necessary virtual data based on the selection of travel destinations.

[1193] A "knowledge processing device" is an artificial intelligence system that responds to user questions in real time and provides appropriate information.

[1194] A "sensory presentation device" is a device that provides tactile feedback such as wind or vibration during a user's virtual travel experience.

[1195] A "language processing system" is an artificial intelligence system that provides tourist information and special information to users through speech.

[1196] A "learning support system" is a system that complements the user experience and manages and provides educational content.

[1197] The present invention provides a system that allows users to experience virtual travel and acquire information in real time through a virtual reality device. The system includes, as its main components, a visual display device, an information processing device, a knowledge processing device, a sensory presentation device, a language processing system, and a learning support system.

[1198] When a user turns on the virtual reality device, the device automatically launches the software. Specifically, the visual display device tracks the user's position and movements and transmits the data to the device. Based on this, the device adjusts and displays the image in the virtual reality space appropriately.

[1199] Next, the user operates the interface to select a desired travel destination, and the terminal transmits this selection information to the server. The server receives this information, retrieves data (video, audio, 3D model, etc.) related to the selected travel destination, and transmits it to the terminal. The terminal receives this data, loads it into a virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[1200] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data about the selected travel destination (such as historical information). Using a language processing system, the AI ​​assistant can begin guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" and respond to the user's questions and instructions.

[1201] For example, if a user asks, "Tell me about the history of this pyramid," the server analyzes the question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. The device also controls a sensory presentation device to provide haptic feedback to recreate the sensation of the user feeling the wind or walking on sand. For example, the vibration motor in the VR device transmits gentle vibrations to the user's feet, recreating the sensation of walking on sand.

[1202] Furthermore, when a user uses educational content, the learning support system switches to a children's education mode. In this case, the server loads educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant then asks the user educational questions, such as, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years."

[1203] As a concrete example, consider a scenario in which a user selects the pyramids of Egypt. The server sends data about the pyramids, including video, audio, and 3D models, to the terminal, which then loads it into a virtual reality device. The AI ​​assistant begins a tour with a greeting like "Welcome to the pyramids of Egypt," and provides detailed explanations in response to the user's questions. The terminal uses a sensory presentation device to give the user the sensation of feeling the wind or walking on sand.

[1204] By providing prompts like the following to the generative AI model, you can provide appropriate guidance and feedback:

[1205] Tell me about the Egyptian pyramids.

[1206] This system allows users to experience real travel through virtual reality devices without physically visiting the destination, and also provides educational content that allows users to have fun while learning.

[1207] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1208] Step 1:

[1209] When a user powers on the virtual reality device, the device automatically launches the software. The visual display device collects data from built-in sensors to track the user's position and movements. This data serves as input for displaying a virtual space that updates in real time in response to the user's movements. The device processes this sensor data and sends it to the visual display device to generate output that displays the virtual reality environment to the user.

[1210] Step 2:

[1211] The user accesses the specified login screen and enters their user ID and password. This information is sent from the terminal to the server, which then verifies the credentials. The server compares the entered credentials with the user information in its database, and if authentication is successful, approves the login. This process allows the user to log in to the system, and the terminal generates output to move on to the next step.

[1212] Step 3:

[1213] The user operates the interface to select the desired travel destination. The user's selection information is sent from the terminal to the server. Based on this input information, the server searches the database for data (video, audio, 3D models, etc.) related to the corresponding travel destination. The server then sends this data to the terminal, which then loads the received data into the virtual reality device in real time. This process results in an output that displays the virtual environment of the travel destination in the user's field of vision.

[1214] Step 4:

[1215] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data (such as historical information) about the selected travel destination. This data is used as input for the AI ​​assistant, which analyzes it and generates a guide message such as "Hello, welcome to the Egyptian pyramids!". Depending on the user's input (questions or instructions), the AI ​​assistant generates an appropriate response. This response is output to the user as audio.

[1216] Step 5:

[1217] As the user explores the VR space and shows interest in a particular place or building, they can ask the AI ​​assistant a question. This input information is sent from the device to the server, which then analyzes the user's question. For example, if the user asks, "Tell me about the history of this pyramid," the server searches the database for the appropriate information and sends it to the AI ​​assistant. The AI ​​assistant then responds, "This pyramid was built in 2560 BC," and generates a detailed explanation. This answer is output to the user via voice.

[1218] Step 6:

[1219] The device controls the sensory presentation device to provide haptic feedback to the user, recreating the sensation of feeling the wind or walking on sand. For example, the device controls the vibration motor of the VR device to transmit gentle vibrations to the user's feet. Based on this input data, the device calculates vibration patterns and controls actual movements. This process generates haptic output that further immerses the user.

[1220] Step 7:

[1221] When a user uses educational content, the learning support system switches to a children's education mode. The server loads the educational content (such as quizzes and games) and sends it to the device. The AI ​​assistant asks the user educational questions. Based on this input, the user enters an answer, and the AI ​​assistant provides immediate feedback. For example, it might ask a question like, "How many years did it take to build this pyramid?" and respond to the user's answer with a response like, "That's right!" or "Sorry, the correct answer is 20 years." This process generates an output that enhances the user's educational experience.

[1222] (Application example 1)

[1223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] While existing virtual travel systems were effective in allowing users to virtually experience specific tourist destinations, there were no systems specifically designed for virtual shopping experiences. This made it difficult for users to check out products and obtain detailed information without visiting a real store. Furthermore, they lacked the ability to provide haptic feedback about products, creating a need for a more realistic shopping experience that cannot be achieved through online shopping.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1226] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for allowing the user to select a virtual store and for loading and displaying store information, an artificial intelligence means for responding to the user's questions in real time and providing detailed product information, and a feedback device means for providing haptic feedback about the products in the user's virtual shopping experience, thereby enabling the user to experience shopping in a virtual store without going to a real store and to obtain detailed product information and haptic feedback in real time.

[1227] "User" refers to an individual who utilizes the system to experience virtual reality.

[1228] A "virtual reality device" is a device for tracking a user's position and movements and displaying a virtual environment.

[1229] "Terminal means" refers to a device that allows a user to select a virtual store and load and display store information.

[1230] "Artificial intelligence means" refers to algorithms and software that respond to user questions in real time and provide detailed product information.

[1231] "Feedback device means" refers to devices or technology that provide tactile feedback about products in a user's virtual shopping experience.

[1232] A "virtual store" is a store that exists in a virtual space and provides a virtual environment where users can visit, check out products, and purchase them.

[1233] "Product information" refers to detailed descriptions and specifications of products that users can view in the virtual store.

[1234] "Haptic feedback" refers to technology that allows users to sense virtual physical properties (e.g., texture or temperature).

[1235] "Virtual shopping experience" refers to the experience of a user visiting a store in virtual reality, browsing, selecting, and purchasing products.

[1236] "Real-time response" means providing appropriate information and feedback immediately in response to a user's questions or operations.

[1237] The system of the present invention allows users to visit a virtual store through a virtual reality device and obtain detailed product information in real time. The system includes a virtual reality device, a terminal, a server, artificial intelligence, and a feedback device.

[1238] When the system starts up, the user turns on the virtual reality device and the device launches the software. The various sensors that track the virtual reality device's position and movement are adjusted to ensure proper operation. The user accesses the designated login screen and logs in by entering their user ID and password. The server then verifies the user's credentials and approves the login. The user can then update their profile information if necessary.

[1239] Next, the user operates the interface to select the desired virtual store. The terminal transmits this selection information to the server. The server retrieves data related to the selected virtual store (product information, video, audio, 3D models, etc.) and transmits it to the terminal. The terminal receives this data, loads it into the virtual reality device, and displays the virtual store environment in the user's field of view.

[1240] The device then launches the AI ​​assistant software. The server provides the AI ​​assistant with store information and special data about the selected product (such as material information). The AI ​​assistant begins guiding the user, saying, "Welcome to our bag specialty store!" and responds to any questions or instructions the user may have.

[1241] As a user explores the virtual store and shows interest in a specific product, they can ask the AI ​​assistant a question. For example, they might ask, "What material is this bag made of?" The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This bag is made of genuine leather," and begins to provide a detailed explanation. The device also controls a feedback device to provide haptic feedback that allows the user to feel the texture of the bag. For example, the feedback device may transmit soft vibrations to the user's hand, recreating the texture of the bag.

[1242] Furthermore, when a user wants to use educational content, they can select product education mode. In this case, the device switches to education mode and the server loads the product education content. The AI ​​assistant provides quizzes and games for the user. For example, the AI ​​assistant may ask a question such as, "How many steps are involved in the manufacturing process of this bag?". When the user enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's correct!" or "Sorry, the correct answer is five steps."

[1243] Specific examples

[1244] Consider a scenario in which a user uses a virtual reality device to visit a virtual bag store. The system verifies login information and loads the virtual store's data. The AI ​​assistant begins guiding the user with a greeting: "Welcome to our bag specialty store." When the user asks, "What material is this bag made of?", the server provides the material information to the AI ​​assistant, who then begins a detailed explanation, saying, "This bag is made of genuine leather." The feedback device also reproduces the texture of the bag, allowing the user to experience its feel.

[1245] Example prompt sentence:

[1246] Simulate a scenario where a user asks, "What material is this bag made of?" The AI ​​assistant will provide detailed material information and realistically recreate tactile feedback using a haptic device.

[1247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1248] Step 1:

[1249] The user turns on the virtual reality device, and the virtual reality device and the terminal start up. The various sensors of the virtual reality device (position, movement, etc.) are adjusted to operate normally.

[1250] Input: Power supply for virtual reality device, user movement information.

[1251] Data processing / computation: The virtual reality device tracks the user's position and movement in real time and calibrates the sensors.

[1252] Output: The user's location and movement information is sent to the device.

[1253] Specific actions: The user puts on the virtual reality device and sets it up.

[1254] Step 2:

[1255] The terminal starts the software and accesses the login screen specified by the user. The user logs in by entering their user ID and password. The server verifies the login information and performs authentication.

[1256] Input: User ID, Password.

[1257] Data processing / calculation: The server performs user authentication and returns the authentication result.

[1258] Output: Login confirmation or error message is displayed on the terminal.

[1259] Specific behavior: The user enters login information, and the device sends the entered information to the server.

[1260] Step 3:

[1261] The user operates the interface to select the desired virtual store, and the terminal transmits the selection information to the server.

[1262] Input: User's store selection information.

[1263] Data processing / calculation: The server searches for the selected virtual store data (product information, video, audio, 3D models, etc.) and sends it to the terminal.

[1264] Output: Store data is sent to the terminal and loaded into the virtual reality device.

[1265] Specific operation: The user operates the interface to select a store, and the terminal sends this to the server.

[1266] Step 4:

[1267] The device launches the AI ​​assistant software, and the server provides store information and product data to the AI ​​assistant.

[1268] Input: Store information and product data.

[1269] Data processing / calculation: The AI ​​assistant analyzes store information and product data and prepares information to provide in an interactive format.

[1270] Output: Your AI assistant is now ready.

[1271] Specific operation: The software starts and receives data from the server.

[1272] Step 5:

[1273] A user explores a virtual store and asks a question about a specific product. The terminal sends the question to the server.

[1274] Input: The user's question.

[1275] Data processing / calculation: The server analyzes the question and sends the corresponding product data to the AI ​​assistant.

[1276] Output: Product information is sent to the device and the AI ​​assistant provides an answer.

[1277] How it works: The user asks the AI ​​assistant a question, and the device sends it to the server.

[1278] Step 6:

[1279] The terminal controls the feedback device to provide haptic feedback to the user, for example, the feedback device reproduces the texture of a bag.

[1280] Input: User operation information, product information.

[1281] Data processing / calculation: The feedback device generates appropriate haptic feedback based on product data.

[1282] Output: Haptic feedback is provided to the user.

[1283] Specific action: The feedback device transmits vibrations or pressure to the user's hand.

[1284] Step 7:

[1285] If the user selects the product education mode, the terminal switches to the education mode and loads the product education content from the server.

[1286] Input: User mode selection.

[1287] Data processing / calculation: The server sends the product educational content to the terminal, and the terminal loads it.

[1288] Output: Educational content is displayed on a virtual reality device.

[1289] Specific behavior: The user selects the education mode, and the device loads the data accordingly.

[1290] Step 8:

[1291] The AI ​​assistant provides quizzes and games for users, making the virtual shopping experience fun.

[1292] Input: Educational content, user answers.

[1293] Data processing / calculation: The AI ​​assistant will ask questions and evaluate the answers, providing feedback on the results.

[1294] Output: The quiz results and explanations are provided to the user.

[1295] What happens: The user takes a quiz and the AI ​​assistant responds accordingly.

[1296] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1297] The system of the present invention allows users to experience virtual travel through a virtual reality device, and further combines it with an emotion engine that recognizes the user's emotions. This system mainly comprises the following means: a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[1298] 1. System startup and initial settings

[1299] First, when a user turns on the virtual reality device, the device launches software that adjusts the virtual reality device's various sensors (position, movement, haptic feedback, etc.) to ensure they function properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. The user can update their profile information if necessary.

[1300] 2. Starting and initializing the emotion engine

[1301] Next, the device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition technology and voice analysis technology to determine the user's emotions.

[1302] 3. Select your destination and load your data

[1303] The user operates the interface to select the desired travel destination. Information about the selected travel destination is sent from the device to the server as a request. The server receives the request, searches the database for data about the selected travel destination (video, audio, 3D model, etc.), and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays the virtual environment of the travel destination in the user's field of view.

[1304] 4. Launching the AI ​​Assistant

[1305] The device launches the AI ​​assistant software. The server provides the AI ​​assistant with tourist information and special data for the selected travel destination (such as historical information). The AI ​​assistant begins guiding the user, saying, "Hello, and welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[1306] 5. Real-time guidance, haptic feedback, and emotion engine integration

[1307] The user explores the VR space and shows interest in a specific place or building. The user asks the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[1308] The emotion engine also analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored. Based on this, the feedback device is adjusted. For example, if the user is excited, the haptic feedback that feels like wind is enhanced. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[1309] 6. Providing educational experiences

[1310] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the emotion engine analyzes the user's emotions and optimizes how the educational content is presented.

[1311] Example: A scenario in which a user visits the Egyptian pyramids

[1312] A user selects the Egyptian pyramids and begins the VR experience. The AI ​​assistant begins by guiding them, saying, "Welcome to the Egyptian pyramids." When the user asks, "Tell me about the history of these pyramids," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback device enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[1313] ---

[1314] The system allows users to have a personalized and realistic travel experience through virtual reality devices and emotion engines without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[1315] The processing flow will be explained below.

[1316] Step 1:

[1317] The device powers on the virtual reality device, automatically launching the internal software and verifying that the virtual reality device's sensors (position, movement, haptic feedback, etc.) are working properly.

[1318] Step 2:

[1319] The user accesses the specified login screen and enters their user ID and password to log in. The server verifies the user's credentials and approves the login.

[1320] Step 3:

[1321] The device acquires the user's profile information and, if it has been updated, performs the initial settings for the virtual trip based on that information. The user confirms or updates the profile information as necessary.

[1322] Step 4:

[1323] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[1324] Step 5:

[1325] The user operates the interface to select the desired travel destination, and information about the selected destination is sent from the terminal to the server as a request.

[1326] Step 6:

[1327] The server receives the request, searches the database for data (video, audio, 3D models, etc.) for the selected travel destination, and sends the search results to the device.

[1328] Step 7:

[1329] The terminal loads the received data into the virtual reality device, displays the virtual environment of the travel destination in the user's field of view, and completes the initial setup of the virtual reality environment, making it ready for the user to interactively explore.

[1330] Step 8:

[1331] The device launches the AI ​​assistant software, and the server provides the AI ​​assistant with tourist information and special data (such as historical information) for the selected travel destination.

[1332] Step 9:

[1333] The AI ​​assistant begins guiding the user by saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the next place they want to see and the information they want to hear.

[1334] Step 10:

[1335] The user explores the VR space and shows interest in specific places and buildings. The user asks the AI ​​assistant, "Tell me about the history of this pyramid."

[1336] Step 11:

[1337] The server analyzes the user's question and sends the appropriate information to the AI ​​assistant, which responds, "This pyramid was built in 2560 BC," and provides a detailed explanation.

[1338] Step 12:

[1339] The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited, impressed, or bored, and adjusts the feedback device accordingly.

[1340] Step 13:

[1341] The terminal controls the haptic device to provide haptic feedback to the user, such as the sensation of wind. For example, if the user is excited, the haptic feedback of wind is enhanced.

[1342] Step 14:

[1343] When a user wants to use educational content, they select the educational mode for children. The device switches to educational mode, the server loads the educational content, and the AI ​​assistant provides guidance for children.

[1344] Step 15:

[1345] The AI ​​assistant asks a question like, "How many years did it take to build this pyramid?" and the user (child) inputs the answer. The AI ​​assistant provides immediate feedback, saying, "That's right!" or "Sorry, the correct answer is 20 years."

[1346] Step 16:

[1347] The emotion engine analyzes the child's emotions from their facial expressions and voice, and adjusts the difficulty and content of quizzes and games accordingly.

[1348] Step 17:

[1349] The user indicates their intention to end the travel experience and selects to end the session on their device. The device then ends the VR device session and prepares to save the user's experience data.

[1350] Step 18:

[1351] The server receives the experience data sent from the device and stores it in a database as the user's travel history. The device then terminates all processes and powers off the VR device.

[1352] Example 2

[1353] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1354] The present invention relates to a system for providing a virtual travel experience. Conventional systems have difficulty in properly recognizing a user's emotional state and providing a personalized experience. Furthermore, conventional systems lack sufficient educational content and lack means to maintain a user's interest in the experience. This makes it difficult for users to obtain a deep learning experience.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1356] In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the user's selected travel destination, an artificial intelligence means for responding to the user's questions in real time, a feedback means for providing feedback such as wind and tactile sensations to the user's virtual travel experience, an emotion analysis engine means for analyzing the user's facial expressions and voice in real time to determine the user's emotional state, and an education mode means for providing educational content. This makes it possible to personalize the virtual travel experience based on the user's emotional state and further optimize and provide educational content.

[1357] "Virtual reality device means" is a general term for hardware that includes devices and sensors for tracking a user's position and movements.

[1358] "Terminal means" refers to an electronic device that allows a user to select a travel destination and has the function of loading and displaying virtual reality data based on that selection.

[1359] "Artificial Intelligence Means" refers collectively to software and algorithms that respond to a user's questions in real time and provide information and guidance about the selected travel destination.

[1360] "Feedback means" are devices and systems that provide physical feedback, such as wind or haptics, to enhance the user's virtual travel experience.

[1361] The "emotion analysis engine means" is software and algorithms for analyzing the user's facial expressions and voice in real time and determining the user's emotional state.

[1362] "Educational mode means" refers to software and systems that have functionality for managing and optimizing educational content as it is experienced by a user.

[1363] The system of the present invention is designed for users to experience virtual travel through a virtual reality device, and integrates an emotion engine that recognizes the user's emotional state. The system includes the following components: a virtual reality device means, a terminal means, a server means, an artificial intelligence means, a feedback means, an emotion analysis engine means, and an education mode means.

[1364] When a user turns on the virtual reality device, the terminal starts the software, which adjusts the virtual reality device's various sensors (position sensors, motion sensors, haptic feedback devices, etc.) to ensure they are working properly. The user accesses the designated login screen and logs in by entering their user ID and password. The server verifies the user's credentials and approves the login. If necessary, the user can update their profile information.

[1365] Next, the device launches an emotion analysis engine to prepare for real-time analysis of the user's facial expressions and voice. The emotion analysis engine uses facial recognition and voice analysis technologies (e.g., OpenCV and Google Speech API) to determine the user's emotional state.

[1366] When a user operates the interface to select a desired travel destination, information about the selected destination is sent as a request from the device to the server. The server receives the request, searches for data about the selected destination (video, audio, 3D model, etc.) from a database (e.g., MySQL), and sends it to the device. The device receives this data and loads it into the virtual reality device, and the virtual environment of the destination is displayed in the user's field of view.

[1367] Furthermore, the device launches the AI ​​assistant software (e.g., Dialogflow). The server provides the AI ​​assistant with tourist information and special data for the selected travel destination. The AI ​​assistant begins guiding the user, saying, "Hello, welcome to the Egyptian pyramids!" The user interacts with the AI ​​assistant based on the places they want to see next and the information they want to hear.

[1368] Additionally, if a user explores the VR space and shows interest in a specific place or building, they can ask the AI ​​assistant, "Tell me about the history of this pyramid." The server analyzes the user's question and sends the appropriate information to the AI ​​assistant. The AI ​​assistant responds, "This pyramid was built in 2560 BC," and provides a detailed explanation. An emotion analysis engine analyzes the user's facial expressions and voice in real time to determine whether the user is excited, impressed, or bored. Based on this, the feedback method is adjusted. For example, if the user is excited, the haptic feedback device will enhance the haptic feedback that makes them feel the wind. If the user is bored, the AI ​​assistant will provide more interesting information and entertainment elements.

[1369] Furthermore, when a user wants to use educational content, they can select an educational mode for children. The device switches to educational mode, and the server loads the educational content. The AI ​​assistant then provides quizzes and games for children. For example, the AI ​​assistant might ask a question like, "How many years did it take to build this pyramid?" When the user (child) enters the answer, the AI ​​assistant provides immediate feedback, such as, "That's right!" or "Sorry, the correct answer is 20 years." In this case, too, the sentiment analysis engine analyzes the user's emotions and optimizes the way the educational content is presented.

[1370] As a concrete example, when a user selects the Egyptian pyramids and starts the VR experience, the AI ​​assistant begins by saying, "Welcome to the Egyptian pyramids." If the user asks, "Tell me about the history of this pyramid," the server provides historical information to the AI ​​assistant, which then gives a detailed explanation. If the user becomes excited, the feedback method enhances the haptic feedback of the wind, further deepening the sense of being there. If the user becomes bored, the AI ​​assistant refreshes the experience by providing new information or quizzes.

[1371] Examples of prompts include:

[1372] "Tell me the history of the Egyptian pyramids."

[1373] "Tell me about the special exhibits."

[1374] "I want to know more about Kyoto festivals."

[1375] "I want to switch to education mode and learn."

[1376] The system allows users to have a personalized and realistic travel experience through virtual reality devices and an emotion analysis engine without physically visiting the destination, and also provides an educational and entertaining experience through educational content.

[1377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1378] Step 1:

[1379] The user powers on the virtual reality device, which initializes the device's hardware and prepares the system to boot.

[1380] Input: Power-on by user operation

[1381] Data processing: Hardware initialization process

[1382] Output: System startup

[1383] Specific behavior:

[1384] The user presses and holds the power button on the VR headset, and the device starts up.

[1385] Step 2:

[1386] The device launches the connected software and adjusts the various sensors of the virtual reality device (position sensors, movement sensors, haptic feedback devices, etc.).

[1387] Input: Virtual reality device sensor data

[1388] Data processing: Sensor calibration

[1389] Output: Adjustment completion notification

[1390] Specific behavior:

[1391] The software on the device will automatically launch and calibrate the device's various sensors, using device drivers and calibration tools.

[1392] Step 3:

[1393] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[1394] Input: User ID and password

[1395] Data Processing: Credentials Verification

[1396] Output: Login approved or denied

[1397] Specific behavior:

[1398] The user enters their ID and password using the keyboard on the device display, and the input data is sent to the server, where authentication is performed by referencing a database.

[1399] Step 4:

[1400] The device launches an emotion analysis engine and prepares to analyze the user's facial expressions and voice in real time.

[1401] Input: facial expression data and voice data

[1402] Data processing: Real-time analysis

[1403] Output: Emotional state identification result

[1404] Specific behavior:

[1405] The device runs emotion analysis engine software, which collects data from the camera and microphone, and uses facial recognition and voice analysis technologies to determine the user's emotional state in real time.

[1406] Step 5:

[1407] The user operates the virtual travel interface to select a desired travel destination.

[1408] Input: User's travel destination selection

[1409] Data processing: Save selected data

[1410] Output: Generate a travel destination data request

[1411] Specific behavior:

[1412] The user operates the terminal interface and selects a destination from the list of travel destinations, which causes the data of the selected travel destination to be recorded in the terminal and prepares for sending a request to the server.

[1413] Step 6:

[1414] The device requests data on the selected travel destination from the server, the server searches for the corresponding data, and sends it to the device, which receives the data and loads it into the virtual reality device.

[1415] Input: Travel Destination Data Request

[1416] Data processing: Searching and sending travel destination data

[1417] Output: Loading travel destination data

[1418] Specific behavior:

[1419] The device sends an HTTP request to the server, which retrieves travel destination data from a database and returns a response in JSON format. The device receives the data, analyzes it, and displays it on the virtual reality device.

[1420] Step 7:

[1421] The device launches the AI ​​assistant software, the server provides tourist information and special data, and the AI ​​assistant starts guiding and interacting with the user.

[1422] Input: Destination data and tourist information

[1423] Data processing: scenario generation and dialogue management

[1424] Output: Guidance and dialogue start

[1425] Specific behavior:

[1426] The device launches the AI ​​assistant software and begins providing guidance based on the tourist information received from the server. The AI ​​assistant speaks to the user, saying, "Hello, welcome to the Egyptian pyramids!"

[1427] Step 8:

[1428] The user explores the VR space and asks questions at points of interest. The server analyzes the questions, and the AI ​​assistant provides answers. The emotion analysis engine analyzes the user's emotions, and the feedback method is adjusted accordingly.

[1429] Input: User question

[1430] Data processing: Question analysis and answer generation

[1431] Output: Answer to the question

[1432] Specific behavior:

[1433] When a user asks, "Tell me about the history of this pyramid," the server uses natural language processing technology to analyze the question and provide the AI ​​assistant with appropriate answer data. The emotion analysis engine analyzes facial expressions and tone of voice, and the feedback method adjusts haptic feedback. For example, it enhances the sensation of wind.

[1434] Step 9:

[1435] When a user selects education mode, the device switches to education mode and the server loads educational content. The AI ​​assistant provides quizzes and games for children and optimizes the delivery method using a sentiment analysis engine.

[1436] Input: Select Education Mode

[1437] Data processing: Loading and optimizing educational content

[1438] Output: Providing quizzes and games

[1439] Specific behavior:

[1440] When a user presses the education mode button, the device sends a mode change request to the server. The server loads educational content, and the AI ​​assistant asks a quiz question, such as "How many years did it take to build this pyramid?" and provides immediate feedback to the user after they answer. The sentiment analysis engine analyzes the user's level of interest in real time and selects the appropriate content delivery method.

[1441] (Application example 2)

[1442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1443] Conventional virtual travel systems provide realistic experiences without requiring users to physically visit the destination. However, they are unable to adjust feedback based on the user's emotions and lack appropriate responses to emotions such as boredom and excitement. Furthermore, in physical stores, product information is provided in a one-way manner, and there is no interactive shopping experience based on the user's emotions. This results in a poor user experience and limited sales promotion effectiveness in physical stores.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a virtual reality device means for tracking the user's location and movement, a terminal means for loading and displaying virtual reality data based on the travel destination selection, an artificial intelligence means for responding to the user's questions in real time, a feedback device means for providing tactile feedback such as wind and vibration to the user's virtual travel experience, and an emotion engine means for analyzing the user's emotions in real time and adjusting the feedback content based on the analysis results. This not only enables appropriate feedback based on the user's emotions to be provided in real time during the virtual travel, but also improves the shopping experience in physical stores, piques the user's interest, and increases sales promotion effects.

[1445] A "virtual reality device" is a device that allows a user to experience a virtual reality space.

[1446] A "terminal" is a computer terminal that is connected to a virtual reality device and that loads and displays data.

[1447] "Artificial intelligence" refers to interactive computer programs that respond to users' questions and provide information in real time.

[1448] A "feedback device" is a device that provides tactile feedback, such as wind or vibration, to a user's virtual travel experience.

[1449] The "emotion engine" is a system that analyzes the user's emotions in real time and adjusts the feedback content based on the analysis results.

[1450] The system of the present invention is composed of the following elements to enable users to experience virtual travel and virtual shopping in physical stores through a virtual reality device: It has the function of analyzing the user's emotions and adjusting feedback individually based on those emotions.

[1451] System Overview

[1452] The system consists of a virtual reality device, a terminal, a server, artificial intelligence, a feedback device, and an emotion engine.

[1453] Hardware and software used

[1454] Virtual reality device: The primary device through which the user experiences the virtual space, providing visual and audio output.

[1455] Terminal: A computer terminal that tracks the user's movements and location and loads and displays virtual reality data.

[1456] Server: Manages virtual reality data, tourist information, product information, etc., and provides it to terminals and virtual reality devices.

[1457] Artificial intelligence: Interactive software that runs on a device or server and responds to user questions in real time. The software used includes natural language processing techniques and machine learning algorithms.

[1458] Feedback devices: These devices provide haptic feedback to enhance the user's virtual experience, providing physical sensations such as wind and vibration.

[1459] Emotion Engine: A system that analyzes the user's emotions and adjusts the feedback content. The technologies used include facial recognition and voice analysis.

[1460] System Operation

[1461] 1. System startup and initial setup:

[1462] The user turns on the virtual reality device, and the device launches the software, which calibrates the virtual reality device's sensors (position, movement, and haptic feedback) to ensure they function properly.

[1463] The user accesses the login screen and enters their user ID and password to log in. The server verifies the credentials and approves the login.

[1464] 2. Emotion Engine startup and initial setup:

[1465] The device activates the emotion engine and prepares to analyze the user's facial expressions and voice in real time. The emotion engine uses facial recognition and voice analysis technologies to determine the user's emotions.

[1466] 3. Select your destination or product information and load your data:

[1467] The user operates the interface to select the desired travel destination or product information, and the selected information is sent from the terminal to the server as a request.

[1468] The server receives the request, searches for the relevant data from the database, and sends it to the terminal, which then receives the data, loads it into the virtual reality device, and displays the virtual environment and product information in the user's field of view.

[1469] 4. Artificial Intelligence Guidance and Interaction:

[1470] The device activates artificial intelligence to provide tourist and product information, and users can interact with the AI ​​based on where they want to see next and what information they want to hear.

[1471] 5. Real-time feedback and emotion engine integration:

[1472] Users explore the VR space and obtain information related to places and products they are interested in. The emotion engine analyzes the user's facial expressions and voice in real time to determine whether they are excited or bored, and adjusts the feedback accordingly.

[1473] Specific examples

[1474] Consider a scenario where a user is wearing smart glasses and shopping in a brick-and-mortar store: if the emotion engine determines that the user is "excited," the AI ​​assistant will provide enhanced details about the product and promotions.

[1475] When a user asks, "Tell me about this product," the AI ​​assistant provides detailed information, such as, "This is the latest smartphone, and its main features are..." If the emotion engine detects boredom, it will provide interesting information in the form of a quiz, encouraging interaction.

[1476] Prompt Sentence Examples

[1477] The following script is a virtual shopping assistant for brick-and-mortar stores that uses smart glasses. It includes a function to adjust the way information is presented depending on the user's emotions. As a concrete example, it shows a scenario in which detailed product information is provided when the user is excited.

[1478] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1479] Step 1:

[1480] The user turns on the virtual reality device. The device launches the software and calibrates the virtual reality device's sensors to ensure proper operation. The device displays a login screen to the user, prompting them to enter their user ID and password. The server then verifies the credentials and, if successful, logs the user into the system. The user's profile information is loaded onto the device.

[1481] Step 2:

[1482] The device activates the emotion engine, which uses facial recognition and voice analysis technologies to capture and analyze the user's facial expressions and voice in real time. The camera captures the user's facial expressions, and the voice input system collects the user's voice. Based on this input data, the emotion engine determines the user's emotions and sends the results to the feedback system.

[1483] Step 3:

[1484] The user operates the interface to select the desired travel destination or product information. The device receives this selection and sends a request to the server. The server searches the database for the corresponding virtual reality data (video, audio, 3D models, etc.) and sends it to the device. The device receives this data, loads it into the virtual reality device, and displays it in the user's field of view.

[1485] Step 4:

[1486] The terminal activates the AI. The server provides tourist information, product information, etc. to the AI, and the AI ​​starts providing guidance to the user, such as "Hello, welcome to the Egyptian pyramids." When the user asks a question, the terminal sends the question to the server, and the server generates an appropriate answer to the question and conveys it to the user via the AI.

[1487] Step 5:

[1488] As the user progresses through the virtual journey and shopping, they will show interest in specific places and products. The emotion engine analyzes the user's facial expressions and voice to determine whether they are excited or bored. If the user is excited, the feedback device will enhance tactile feedback such as wind, and the AI ​​will provide additional interesting information and suggestions. Conversely, if the user is bored, the AI ​​will provide information in the form of quizzes and games to pique the user's interest.

[1489] Step 6:

[1490] If the user selects education mode as they continue their experience, the device will load educational content. The server provides educational content for children, and the AI ​​interacts with the user (child) through quizzes and games. Again, the emotion engine analyzes the user's emotions and adjusts feedback at the appropriate time. For example, the AI ​​assistant might ask a quiz question such as, "How many years did it take to build this pyramid?" and provide immediate feedback to the user's answer.

[1491] These steps make the virtual travel and shopping experience more personalized and engaging for the user.

[1492] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1493] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1494] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1495] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1496] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1497] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1498] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1499] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1500] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1501] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1502] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1503] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1504] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1505] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1506] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1507] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1508] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1509] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1510] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1511] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1512] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1513] The following is further disclosed regarding the above embodiment.

[1514] (Claim 1)

[1515] A system for allowing a user to experience a virtual journey through a virtual reality device, comprising:

[1516] a virtual reality device means for tracking a user's position and movement;

[1517] terminal means for loading and displaying virtual reality data based on the travel destination selection;

[1518] an artificial intelligence means for responding to user questions in real time;

[1519] a feedback device means for providing haptic feedback such as wind or vibration to the user's virtual travel experience;

[1520] A system including:

[1521] (Claim 2)

[1522] 2. The system according to claim 1, wherein the artificial intelligence means provides tourist information and guidance for a travel destination selected by the user.

[1523] (Claim 3)

[1524] 10. The system of claim 1, wherein the artificial intelligence means manages and provides educational content based on user interaction.

[1525] "Example 1"

[1526] (Claim 1)

[1527] A system for allowing a user to experience a virtual journey through a virtual reality device, comprising:

[1528] visual display means for tracking the user's position and movement;

[1529] an information processing device means for loading and displaying virtual data based on the travel destination selection;

[1530] a knowledge processing device that responds to user questions in real time;

[1531] a sensory presentation device means for providing tactile feedback such as wind and vibration to the user's virtual travel experience;

[1532] a language processing system means for providing tourist destination information and special information through speech;

[1533] a learning support system means for managing and providing educational content to complement the user's experience;

[1534] A system including:

[1535] (Claim 2)

[1536] 2. The system according to claim 1, wherein the knowledge processing device provides tourist information and guidance relating to a travel destination selected by the user.

[1537] (Claim 3)

[1538] 10. The system according to claim 1, wherein the learning support system means manages and provides educational content based on user interaction.

[1539] "Application Example 1"

[1540] (Claim 1)

[1541] A system for allowing a user to experience a virtual journey through a virtual reality device, comprising:

[1542] a virtual reality device means for tracking a user's position and movement;

[1543] a terminal means for allowing a user to select a virtual store and for loading and displaying store information;

[1544] An artificial intelligence means for responding to user questions in real time and providing detailed product information;

[1545] a feedback device means for providing haptic feedback about the products to the user's virtual shopping experience;

[1546] A system including:

[1547] (Claim 2)

[1548] 2. The system according to claim 1, wherein the artificial intelligence means provides product information and guidance regarding the virtual store selected by the user.

[1549] (Claim 3)

[1550] 10. The system of claim 1, wherein the artificial intelligence means manages and provides product education content based on user interaction.

[1551] "Example 2: Combining Emotion Engines"

[1552] (Claim 1)

[1553] A system for allowing a user to experience a virtual journey through a virtual reality device, comprising:

[1554] a virtual reality device means for tracking a user's position and movement;

[1555] terminal means for loading and displaying virtual reality data based on the travel destination selection;

[1556] an artificial intelligence means for responding to user questions in real time;

[1557] a feedback means for providing feedback such as wind or tactile sensations to the user's virtual travel experience;

[1558] an emotion analysis engine means for analyzing the user's facial expressions and voice in real time and determining the user's emotional state;

[1559] an educational mode means for providing educational content;

[1560] A system including:

[1561] (Claim 2)

[1562] 2. The system according to claim 1, wherein the arti...

Claims

1. A system for allowing a user to experience a virtual journey through a virtual reality device, comprising: a virtual reality device means for tracking a user's position and movement; terminal means for loading and displaying virtual reality data based on the travel destination selection; an artificial intelligence means for responding to user questions in real time; a feedback device means for providing haptic feedback such as wind or vibration to the user's virtual travel experience; A system including:

2. 2. The system of claim 1, wherein the artificial intelligence means provides tourist information and guidance for a travel destination selected by the user.

3. 10. The system of claim 1, wherein the artificial intelligence means manages and provides educational content based on user interaction.

Images