System

Abstract

To provide a system capable of eliminating various problems for travels in an actual world.SOLUTION: The system includes: means for generating an artificial world using digital twins; means for allowing experience of the artificial world in virtual reality; means for allowing communication with generative AI by producing a person in the artificial world using AI; mean for allowing purchasing activity in the artificial world to be realized in e-commerce; means for linking purchasing activity in the artificial world with transportation means or accommodation in an actual world; means for paying a part of a usage fee as an information provision fee; means for allowing delivery in the actual world when a user selects a product in the artificial world and confirms purchasing it; and means for carrying out reservation procedures in the actual world when the user selects the transportation means or the accommodation in the artificial world.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traveling in the real world presents various challenges, such as time, cost, and physical constraints. Furthermore, language barriers and lack of information at the destination can hinder the travel experience. [Means for solving the problem]

[0005] This invention uses a digital twin to generate a virtual world, which can then be experienced in VR, providing a new travel experience that transcends the constraints of time, cost, and physical space. Furthermore, by generating humans in the virtual world using AI and enabling conversations with the generative AI, language barriers and lack of information are eliminated. Furthermore, by enabling purchases within the virtual world via e-commerce and linking them to real-world transportation and accommodations, the virtual world is linked to the real world. This allows users to have a new travel experience that transcends the constraints of the real world. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on places a user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, data on the city's streets, buildings, and scenery is acquired, and a simulated world of Paris is generated based on that data.

[0029] "Example 2"

[0030] The generated simulated world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their own home. The people in the simulated world are also generated by AI, and can converse with the generative AI. This allows the user to ask for directions or enjoy everyday conversations.

[0031] "Example 3"

[0032] Furthermore, purchasing within the virtual world is realized through e-commerce. When a user selects and purchases a product at a store within the virtual world, the product is delivered to the user in the real world. It is also linked to real-world transportation and accommodation options, so when a user selects transportation or accommodation within the virtual world, that option is reflected in their reservation in the real world.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user inputs the place they want to visit into the system. For example, if the user wants to visit Paris, they input the information Paris into the system.

[0036] Step 2: The system retrieves data for the relevant location based on the input information. This data includes information on streetscapes, buildings, landscapes, etc.

[0037] Step 3: A digital twin is created based on the acquired data. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[0038] "Example 2"

[0039] Step 1: The user puts on the VR device.

[0040] Step 2: Experience the simulated world created through the VR device. Users can experience walking through the streets of Paris from the comfort of their own home.

[0041] Step 3: Humans in the virtual world are generated by AI and can converse with the generated AI. Users can ask the AI-generated humans for directions or enjoy everyday conversations.

[0042] "Example 3"

[0043] Step 1: The user selects and purchases a product from a store in the virtual world.

[0044] Step 2: The purchased product is delivered to the user in the real world via the e-commerce system.

[0045] Step 3: The user selects transportation and accommodation in the virtual world, which is then reflected in the booking in the real world.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] Conventional digital twin technology requires a complex process for users to virtually experience the places they want to visit, and lacks a system that users can easily access. Furthermore, there is a lack of means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This makes it difficult for users to have a virtual experience in real time.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes means for inputting a place a user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin, thereby enabling the user to easily virtually experience the places they want to visit.

[0051] A "user" is an individual or group who uses the system to specify the place they want to visit and virtually experience it.

[0052] "Input" refers to the act of a user specifying to the system the places they wish to visit, or the information therein.

[0053] "Geographical data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layout.

[0054] "Analysis" is the process of calculations and data processing used to process collected geographic data and convert it into a 3D model.

[0055] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographic data.

[0056] A "digital twin" is a 3D model that recreates a specific location in the real world in a virtual space.

[0057] "Sending" is the act of transferring the digital twin generated by the server to the user's device.

[0058] "Operation" refers to the act of a user interactively using a digital twin on a terminal.

[0059] This invention is a system that allows users to virtually experience places they would like to visit. The system involves a series of processes: a user inputs the place they would like to visit, collects geographical data about the place, analyzes it, converts it into a 3D model, generates a digital twin, and transmits it to the user's device.

[0060] Hardware and software used

[0061] server

[0062] The server receives user input, collects and analyzes geographic data, and generates the 3D model. Specifically, it uses the following software and services:

[0063] Google® Maps API: Used to collect geographic data.

[0064] OpenStreetMap: Used to collect complementary geographic data.

[0065] Python: Run scripts to analyze collected geographic data and convert it into 3D models.

[0066] Unity: A game engine for generating 3D models based on the analyzed data.

[0067] Terminal

[0068] The terminal is a device that allows users to operate the digital twin and virtually experience the place they want to visit. Specifically, it uses the following software:

[0069] Unity application: displays the 3D model sent from the server and allows the user to manipulate it.

[0070] Data processing and calculation

[0071] Data collection

[0072] The server collects geographical data from Google Maps API and OpenStreetMap based on the place the user wants to visit. For example, if the user enters "Paris," the server retrieves data such as Paris's latitude and longitude, building shapes, and road layout.

[0073] Data analysis

[0074] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and converts the building shape into polygon data. This process generates the basic data for the digital twin.

[0075] 3D model generation

[0076] The server uses Unity to generate a 3D model based on the analyzed data. Specifically, it runs Unity C scripts to create 3D models of Paris's streets and buildings. This creates a digital twin that allows users to virtually visit Paris.

[0077] Sending data

[0078] The server then sends the generated 3D model to the user's device, where the user can launch a Unity application and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll down the Champs-Élysées.

[0079] Specific examples

[0080] If a user inputs "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographic data for Paris. Then, it runs a Python script to analyze the retrieved geographic data and convert it into a 3D model. Using Unity, it generates a 3D model of Paris's streets and buildings based on the analyzed data. Finally, it sends the generated 3D model to the user's device, where the user can launch the Unity application and virtually experience Paris.

[0081] Prompt Sentence Examples

[0082] "Create a digital twin of Paris. Use Google Maps API to get geographic data of Paris and generate a 3D model using Unity."

[0083] In this way, a system is constructed that allows users to virtually experience places they would like to visit.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] The user enters the location they want to visit.

[0087] Specifically, a user accesses the system's web interface, enters "Paris" in the search box, and clicks the search button.

[0088] Input: A place you would like to visit (e.g. Paris)

[0089] Output: Places you want to visit (e.g. Paris)

[0090] Step 2:

[0091] The server receives user input and collects geographic data.

[0092] Specifically, the server sends a request to the Google Maps API to retrieve geographic data for Paris. For example, it sends a request to the URL https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[0093] Input: Information about the place you want to visit (e.g. Paris)

[0094] Output: Collected geographical data (e.g., Paris's latitude and longitude, building shapes, road layout)

[0095] Step 3:

[0096] The server analyzes the collected geographic data and converts it into a 3D model.

[0097] Specifically, the server runs a Python script to parse the retrieved geographic data. For example, it runs a script called parse_geodata.py to convert the latitude and longitude information into 3D coordinates.

[0098] Input: Collected geographical data (e.g., latitude and longitude information of Paris, building shapes, road layout)

[0099] Output: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[0100] Step 4:

[0101] The server generates a 3D model based on the analyzed data.

[0102] Specifically, the server runs a Unity C script to create a 3D model of the Paris cityscape and buildings. For example, it runs a script called GenerateParisModel.cs.

[0103] Input: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[0104] Output: Generated 3D model (e.g. 3D model of Paris cityscape and buildings)

[0105] Step 5:

[0106] The server sends the generated 3D model to the user's device.

[0107] Specifically, the server transfers the generated 3D model to the user's device.

[0108] Input: Generated 3D model (e.g., 3D model of Paris cityscape or buildings)

[0109] Output: The 3D model sent to the user's device

[0110] Step 6:

[0111] The user operates the digital twin on the device.

[0112] Specifically, users can launch the Unity application on their device and interact with the digital twin, for example, virtually visiting the Eiffel Tower or strolling down the Champs-Élysées.

[0113] Input: The 3D model sent to the user's device

[0114] Output: A digital twin operated by the user

[0115] (Application example 1)

[0116] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has difficulty recreating the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to get detailed information about the places you want to visit and experience their atmosphere in advance. This often leaves consumers feeling anxious because they cannot check the details of products and places before purchasing. Furthermore, the lack of ways to link with real-world transportation and accommodations can lead to a lack of consistency in travel plans.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0119] In this invention, the server includes: a means for generating a simulated world using a digital twin; a means for experiencing the simulated world in VR; a means for generating a person in the simulated world using AI and enabling conversation with the generating AI; a means for making purchases in the simulated world via e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for acquiring data on places the user wants to visit and generating a digital twin based on the data; a means for providing a virtual shopping tour based on the digital twin; and a means for experiencing the virtual shopping tour via a smartphone or head-mounted display. This allows users to experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[0120] A "digital twin" is a virtual representation of a physical place or object in the real world as digital data.

[0121] A "pseudo-world" is a virtual environment or space created using a digital twin.

[0122] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[0123] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0124] "Generative AI" is a technology that uses artificial intelligence to generate people and objects in virtual space.

[0125] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the Internet.

[0126] "Transportation" refers to the means of transportation or method by which a user physically travels.

[0127] "Accommodation" means a facility or place where users can stay temporarily.

[0128] "Information Fee" means the fee paid by the User for the information provided.

[0129] A "virtual shopping tour" is a tour that allows you to experience shopping in a virtual space created using a digital twin.

[0130] A "smartphone" is a type of mobile phone, a multi-function device that can connect to the Internet and use applications.

[0131] A "head-mounted display" is a display device worn on the head, and is used to experience virtual reality and augmented reality.

[0132] A system for implementing this invention generates a simulated world using a digital twin and allows users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[0133] System Configuration

[0134] The system mainly consists of the following components:

[0135] 1. Server: Generates the digital twin and provides data upon user request.

[0136] 2. Device: The smartphone or head-mounted display used by the user.

[0137] 3. User: Use the system to experience a virtual shopping tour.

[0138] Program processing

[0139] The server implements the system using the following means:

[0140] Digital twin generation method: Obtain data on the place the user wants to visit and generate a digital twin based on that data. Specifically, the location data is obtained through an API and a virtual space is generated using a 3D rendering library.

[0141] VR experience: The generated digital twin can be experienced in VR. Users can move freely within the virtual space using a smartphone or head-mounted display.

[0142] AI generation: AI generates humans in virtual space and enables conversations with users. Generative AI models are used to achieve natural conversations in real time.

[0143] EC implementation: E-commerce is used to enable purchasing within a virtual space. Users can select products in a virtual store and complete the purchase process.

[0144] Linking: Linking to real-world transportation and accommodations, allowing users to make reservations and arrangements within the virtual space.

[0145] Payment method for information provision fee: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[0146] Hardware and software used

[0147] Hardware: Smartphone, head-mounted display

[0148] Software: Python, 3D rendering library (e.g. some_3d_rendering_library), generative AI model, API

[0149] Specific examples

[0150] If a user wants to go shopping on the Champs-Élysées in Paris, the system works as follows:

[0151] 1. The user accesses the system using a smartphone or head-mounted display.

[0152] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[0153] 3. Users can experience the generated digital twin in VR and move freely within the virtual space.

[0154] 4. Talk to an AI-generated human in the virtual space and receive directions and product explanations.

[0155] 5. The user selects products in the virtual store and completes the purchase through e-commerce.

[0156] 6. Book real-world transportation and accommodations, if necessary.

[0157] Prompt Sentence Examples

[0158] Users can enter the name of a city they want to visit, for example, "Paris," which will generate a digital twin of the Champs-Élysées in Paris and allow them to start a virtual shopping tour.

[0159] In this way, users can experience the detailed information and atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[0160] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0161] Step 1:

[0162] The user accesses the system using a smartphone or head-mounted display. The user inputs the name of the place they want to visit. The inputted name of the place is sent to the server.

[0163] Step 2:

[0164] The server retrieves the corresponding location data based on the input location name. Specifically, it requests the location data through the API and parses the retrieved data. The input is the location name, and the output is the location details.

[0165] Step 3:

[0166] The server generates a digital twin based on the acquired data. It uses a 3D rendering library to recreate detailed location data as a virtual space. The input is detailed location data, and the output is a digital twin.

[0167] Step 4:

[0168] The server sends the generated digital twin to the user's device, where the user experiences the digital twin in VR via a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[0169] Step 5:

[0170] The server generates a human in the virtual space using a generative AI model. The generated AI can converse with the user. The input is the generative AI model and the virtual space data, and the output is an AI-generated human.

[0171] Step 6:

[0172] The user interacts with an AI-generated human in a virtual space, receiving directions and product explanations. The input is the user's question or request, and the output is a response from the AI-generated human.

[0173] Step 7:

[0174] The user selects a product in a virtual store and completes the purchase procedure. The server then executes the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[0175] Step 8:

[0176] Users book transportation and accommodation in the real world as needed. The server processes this reservation information and provides it to the user. The input is the user's reservation information, and the output is a reservation confirmation.

[0177] Step 9:

[0178] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is confirmation of fee collection.

[0179] In this way, users can experience detailed information and the atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[0180] Example 2

[0181] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] Conventional virtual reality systems have limited the simulated worlds that users experience, making it difficult to have natural conversations and interactions in real time. Furthermore, purchasing within the simulated world and links to the real world are insufficient, limiting the user experience. Furthermore, there is a lack of a way to effectively use the user's voice to have natural conversations with generative AI.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0184] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating people in the simulated world using artificial intelligence and enabling them to converse with the generative artificial intelligence, means for making purchases in the simulated world through e-commerce, means for linking to transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for converting a user's voice into text and sending it to the generative artificial intelligence as a prompt, and means for converting a response generated by the generative artificial intelligence into voice and returning it to the user. This allows users to enjoy natural conversations in real time, improving interaction within the simulated world and strengthening the link with the real world.

[0185] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[0186] A "pseudo-world" is a virtual environment different from the real world that a user experiences through a virtual reality device.

[0187] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment.

[0188] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[0189] "Generative AI" is an AI model that generates natural responses based on user input.

[0190] "Electronic commerce" refers to the buying and selling of goods and services over the Internet.

[0191] A "prompt sentence" is a text-based instruction sentence sent to a generative artificial intelligence based on user input.

[0192] "Speech recognition software" is software for converting speech into text.

[0193] "Speech synthesis software" is software for converting text into speech.

[0194] The present invention provides a system for generating a simulated world that a user experiences through a virtual reality device, and for enabling natural conversation with a generative artificial intelligence. Specific embodiments of this system are described below.

[0195] First, the server generates a virtual world using the digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then provided to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[0196] When a user wears a virtual reality device, the device connects to a server and receives data about the simulated world. Based on the received data, the device renders the simulated world in real time on the device's display, allowing the user to experience the world as if they were actually walking through it.

[0197] The humans in the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the generative artificial intelligence as a prompt.

[0198] A generative AI (e.g., OpenAI's GPT-4) generates an appropriate response based on the received prompt. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI generates the response, "The Eiffel Tower is just down this road, turn right at the next intersection."

[0199] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech), which is then transmitted back to the user through the virtual reality device's speakers, allowing the user to hear the response from a human in the simulated world.

[0200] As a concrete example, imagine a user wearing a virtual reality device is walking through the streets of Paris. If the user is near the Eiffel Tower and asks a person in the virtual world, "Where is the Eiffel Tower?", the following prompt is sent to the generative AI:

[0201] Example prompt sentence:

[0202] User: "Where is the Eiffel Tower?"

[0203] The generative AI receives this prompt and generates an appropriate response, which the device converts into speech and returns to the user, allowing the user to enjoy natural conversation in real time.

[0204] This system improves user interaction within the virtual world and strengthens the link with the real world. For example, it enables purchases within the virtual world to be made through e-commerce, and links to real-world transportation and accommodation options. Furthermore, it includes a means to pay a portion of the usage fee as an information provision fee, further enhancing the user experience.

[0205] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0206] Step 1:

[0207] The user puts on the virtual reality device.

[0208] Input: The action of the user putting on the virtual reality device.

[0209] Output: The virtual reality device is activated and the user's vision and hearing are switched to the virtual reality device.

[0210] Specific operation: The user puts the virtual reality device on their head and turns it on, which activates the device and switches the user's vision and hearing to the information provided through the virtual reality device.

[0211] Step 2:

[0212] The device connects to the server and receives data from the virtual world.

[0213] Input: Virtual reality device activation signal.

[0214] Output: Data from the virtual world is sent from the server to the device.

[0215] How it works: The device connects to a server via the Internet, and the server sends data about the virtual world created with Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[0216] Step 3:

[0217] The device renders the simulated world and displays it to the user.

[0218] Input: Pseudo-world data received from the server.

[0219] Output: The simulated world is displayed on the display of the virtual reality device.

[0220] How it works: Based on the received data, the device renders the city of Paris in real time on the display of the virtual reality device, giving the user a visual experience that makes them feel as if they are actually walking through the city of Paris.

[0221] Step 4:

[0222] The user acts within the virtual world and talks to people within the virtual world.

[0223] Input: User's voice input.

[0224] Output: The user's voice is captured on the device.

[0225] Specific operation: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[0226] Step 5:

[0227] The device converts the user's speech into text and sends it to the generative AI model as a prompt.

[0228] Input: User's voice data.

[0229] Output: The prompt in text format.

[0230] How it works: The device captures the user's voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text as a prompt to the generative AI model.

[0231] Step 6:

[0232] The server generates a response using a generative AI model.

[0233] Input: A prompt in text format.

[0234] Output: The generated response text.

[0235] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt, such as "The Eiffel Tower is just down this road, turn right at the next intersection."

[0236] Step 7:

[0237] The terminal converts the generated response into speech and returns it to the user.

[0238] Input: The generated response text.

[0239] Output: The response in audio format.

[0240] How it works: The device converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then sent back to the user through the speaker of the virtual reality device. The user can then hear the response from the person in the virtual world.

[0241] (Application example 2)

[0242] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0243] Today's consumers want to enjoy shopping without visiting a physical store, but traditional online shopping has the drawback of not being able to actually touch and check the product. It's also difficult to obtain detailed product information through conversations with store staff. Furthermore, there are limited ways for users to experience the streets of foreign countries and enjoy the local atmosphere from the comfort of their own homes. To address these challenges, a system that offers a more realistic shopping experience and interactive communication with users is needed.

[0244] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0245] In this invention, the server includes: a means for generating a virtual world using a digital twin; a means for experiencing the virtual world in VR; a means for generating a human in the virtual world using AI and enabling conversation with the generated AI; a means for enabling purchases in the virtual world through e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for tracking the user's location and initiating a conversation with an AI-generated character; a means for processing the user's input and generating a response from the AI; and a means for displaying the response. This allows users to enjoy a realistic shopping experience from the comfort of their own home, obtaining detailed information about products through interactive communication with store clerks. Users can also experience the cityscapes of foreign countries and enjoy the local atmosphere.

[0246] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[0247] A "pseudo-world" is a virtual environment or space created using a digital twin.

[0248] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[0249] "AI" stands for artificial intelligence, a computer system that mimics human intelligence.

[0250] "Generative AI" is artificial intelligence that interacts with users and generates content.

[0251] "EC" is an abbreviation for electronic commerce, a system for buying and selling goods and services over the Internet.

[0252] "Real-world transportation" refers to the means of transportation used by users to travel in the real world.

[0253] "Accommodation" refers to a facility where users can stay.

[0254] "Information Fee" means the fee paid by a user for the information provided by the user.

[0255] "Tracking user location" means tracking the user's current location in real time.

[0256] An "AI-generated character" is a virtual person or character generated by artificial intelligence.

[0257] "Processing user input" means analyzing input data from the user and generating an appropriate response.

[0258] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[0259] "Displaying the response" means visually presenting the generated response to the user.

[0260] To implement the present invention, the following system configuration and program are required.

[0261] System Configuration

[0262] 1. Hardware:

[0263] VR devices (e.g. Oculus Rift, HTC Vive)

[0264] Smartphones (e.g. iPhone (registered trademark), ANDROID (registered trademark))

[0265] Server (with high-performance computing capabilities)

[0266] 2. Software:

[0267] VR engine (e.g. Unity, Unreal Engine)

[0268] AI conversation engine (e.g., OpenAI GPT-3 (registered trademark))

[0269] Program processing explanation

[0270] The server first uses the digital twin to generate a virtual world that recreates the streets of Paris. The user then wears a VR device and experiences this virtual world. The user's location is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[0271] User input is sent to the server via the VR device or smartphone, which then analyzes the input and generates an appropriate response using an AI conversation engine, which then displays the response in the user's field of view.

[0272] Specific examples

[0273] When a user walks into a virtual store and asks an AI-generated sales associate, "Do you have this dress in my size?", the server receives this input and uses its AI conversation engine to generate a response: "Yes, this dress is available in sizes S, M, and L." This response is displayed in the user's field of view.

[0274] Prompt Sentence Examples

[0275] User: Do you have this dress in your size?

[0276] AI: Yes, this dress is available in sizes S, M, and L.

[0277] In this way, users can enjoy a realistic shopping experience from the comfort of their own home, obtain detailed information about products through interactive communication with store staff, and even experience the local atmosphere of a foreign city.

[0278] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0279] Step 1:

[0280] The server uses a digital twin to generate a simulated world that recreates the streets of Paris.

[0281] Input: Paris streetscape data

[0282] Data processing: Using digital twin technology, the real-world streets of Paris are recreated in a virtual space.

[0283] Output: A simulated world of Parisian streets

[0284] Step 2:

[0285] The user wears a VR device and experiences a simulated world transmitted from the server.

[0286] Input: Pseudo-world data sent from the server

[0287] Data calculation: The VR device tracks the user's viewpoint and movements in real time and displays a simulated world.

[0288] Output: A simulated world displayed to the user.

[0289] Step 3:

[0290] The server tracks the user's location in real time.

[0291] Input: User's location

[0292] Data calculation: Calculates the user's position in real time and updates their position in the virtual world.

[0293] Output: Updated user location

[0294] Step 4:

[0295] When a user approaches an AI-generated character, the server initiates a conversation.

[0296] Input: User location, AI-generated character location

[0297] Data calculation: Calculates the distance between the user and the character, and starts conversation when they come within a certain distance.

[0298] Output: Trigger to start a conversation

[0299] Step 5:

[0300] Users send input to the server through their VR device or smartphone.

[0301] Input: User voice and text input

[0302] Data calculation: Sends input data to the server.

[0303] Output: User input data sent to the server

[0304] Step 6:

[0305] The server analyzes the user's input and generates an appropriate response using an AI conversation engine.

[0306] Input: User-entered data

[0307] Data Computing: Using an AI conversation engine (e.g., OpenAI GPT-3) to generate appropriate responses to user input.

[0308] Output: The generated response data

[0309] Step 7:

[0310] The server displays the generated response in the user's view.

[0311] Input: Generated response data

[0312] Data calculation: Send the response data to the VR device and display it in the user's field of view.

[0313] Output: The response displayed in the user's view

[0314] In this way, users can enjoy a realistic shopping experience from the comfort of their own home. They can also obtain detailed information about products through interactive communication with store staff. They can also experience the streets of foreign countries and enjoy the local atmosphere.

[0315] Example 3

[0316] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] There is a demand for a system that allows efficient purchasing and reservation procedures within a virtual world linked to the real world. In conventional systems, the process of reflecting actions in the virtual world in the real world is complicated, resulting in low user convenience. There is also a need for a system that makes the experience in the virtual world more realistic and allows users to freely choose the location and time.

[0318] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in virtual reality, means for generating characters in the pseudo-world using artificial intelligence and enabling conversations with the generative artificial intelligence, means for making purchases in the pseudo-world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for arranging delivery in the real world when a user selects a product in the pseudo-world and confirms the purchase, and means for arranging reservations in the real world when a user selects transportation and accommodation in the pseudo-world. This allows the user's actions in the pseudo-world to be reflected in the real world quickly and efficiently.

[0319] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[0320] A "pseudo-world" is a virtual reality environment created using a digital twin.

[0321] "Virtual reality" is a technology that allows users to experience an artificial environment generated using computer technology.

[0322] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[0323] "Generative AI" is a system that uses artificial intelligence technology to generate characters in a virtual world and enable them to converse with the user.

[0324] "Electronic commerce" is a form of transaction in which goods and services are bought and sold over the Internet.

[0325] "Transportation" is the means by which a user travels from one place to another.

[0326] "Accommodation" means a facility where users can stay temporarily.

[0327] "Information provision fee" is part of the fee paid by users when they use the system, and is paid in exchange for the provision of information.

[0328] "Shipping procedure" refers to the procedure for shipping the products purchased by the user in the real world.

[0329] "Booking Process" means the process by which a User makes a real-world reservation for their chosen mode of transportation and accommodation.

[0330] MODE FOR CARRYING OUT THE INVENTION

[0331] This invention is a system that reflects the actions of a user in a virtual world, such as purchasing products and selecting transportation and accommodations, in the real world. Specific embodiments of this system are described below.

[0332] Hardware and software used

[0333] Server: Database management system (e.g., MySQL (registered trademark)), Web server (e.g., Apache (registered trademark))

[0334] Device: PC or smartphone used by the user

[0335] Generative AI models: Natural language processing models (e.g., GPT-4)

[0336] Data processing and calculation

[0337] 1. The user selects a product in a virtual world

[0338] Users use their devices to visit stores in the virtual world.

[0339] When a user selects a product, the information is sent to the server.

[0340] The server stores the data of the selected products in a database.

[0341] 2. Completing the purchase process

[0342] Once the user confirms the purchase, the server processes the purchase through the e-commerce system (e.g., Shopify).

[0343] The purchase information is sent to a real-world delivery system, and the product is delivered to the user.

[0344] 3. Choosing transportation and accommodation

[0345] Users choose their means of transportation and accommodation within the virtual world.

[0346] The selection information is sent to the server, which makes the real-world reservation through a reservation system (e.g., Booking.com API).

[0347] Specific examples

[0348] Specific examples of product purchases

[0349] A user chooses a smartphone in a virtual electronics store.

[0350] When the purchase button is clicked, the server completes the purchase process through the e-commerce system and the smartphone is delivered to the user's address.

[0351] Specific examples of transportation options

[0352] The user chooses a taxi within the virtual world.

[0353] The server makes taxi reservations in the real world through a taxi reservation system.

[0354] Prompt Sentence Examples

[0355] Purchase prompts

[0356] A user wants to purchase a smartphone in a simulated world. Please explain the steps to complete the purchase and have it shipped to a real-world address.

[0357] Prompt for transportation selection

[0358] A user wants to book a taxi in a simulated world. Please explain the steps to go through the booking process and complete the taxi booking in the real world.

[0359] This system allows the user's actions in the virtual world to be reflected in the real world quickly and efficiently. The server receives the selection information from the user and performs the procedure in the real world through the appropriate system. This significantly improves user convenience. The flow of the specific processing in Example 3 will be explained using Figure 15.

[0360] Step 1:

[0361] The user visits a store in the virtual world using a terminal. The user selects a product and clicks the select button. The input is the product information selected by the user, and the output is that information being sent to the server. In concrete terms, the user selects a smartphone at an electronics store in the virtual world.

[0362] Step 2:

[0363] The server receives the product selection information sent by the user. The input is the product information sent by the user, and the output is that information is saved in a database. Specifically, the server receives the selection information from the "smartphone" and saves it in the database.

[0364] Step 3:

[0365] The user clicks the button to confirm the purchase on the purchase confirmation screen. The input is the user's purchase confirmation operation, and the output is that information being sent to the server. The specific action is when the user clicks the "Confirm Purchase" button on the purchase confirmation screen.

[0366] Step 4:

[0367] The server connects to the EC system and starts the purchase procedure. The input is the user's purchase confirmation information, and the output is the start of the purchase procedure to the EC system. Specifically, the server connects to the EC system and performs the purchase procedure on the smartphone.

[0368] Step 5:

[0369] The server sends the user's payment information to the e-commerce system to complete the purchase. The input is the user's payment information, and the output is a confirmation of the purchase completion.

[0370] Step 6:

[0371] The server sends purchase information to the delivery system. The input is purchase completion information, and the output is sending information to the delivery system. Specifically, the server sends the smartphone's delivery information to the delivery system.

[0372] Step 7:

[0373] The delivery system delivers the product to the user's address. The input is delivery information, and the output is that the product will be delivered to the user's address. The specific operation is that the delivery system delivers a smartphone to the user's address.

[0374] Step 8:

[0375] The user uses a terminal to select transportation and accommodation within the virtual world. The input is the user's selection information, and the output is that information is sent to the server. The specific operation is for the user to select a taxi within the virtual world.

[0376] Step 9:

[0377] The server receives the selection information sent by the user. The input is the selection information sent by the user, and the output is that the information is saved in the database. Specifically, the server receives the selection information for "taxi" and saves it in the database.

[0378] Step 10:

[0379] The server connects to the reservation system and starts the reservation procedure. The input is the user's selection information, and the output is the start of the reservation procedure to the reservation system. In concrete terms, the server connects to the reservation system and performs the taxi reservation procedure.

[0380] Step 11:

[0381] The server sends the reservation information to the reservation system and completes the reservation in the real world. The input is the reservation information and the output is confirmation of the reservation completion. In concrete terms, the server sends the reservation information to the reservation system and completes the taxi reservation in the real world.

[0382] (Application example 3)

[0383] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] There is a demand for a system that allows users to seamlessly perform purchasing and reservation actions in the virtual world in the same way as they would in the real world. In conventional systems, the experience in the virtual world is separated from the actions in the real world, and users have to perform separate manual operations to use the products and services they selected in the virtual world in the real world. This has led to issues such as a poor user experience and a loss of convenience.

[0385] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a virtual world using a digital twin, means for experiencing the virtual world in VR, means for generating a person in the virtual world using AI and enabling conversation with the generative AI, means for realizing purchases in the virtual world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for selecting a product in the virtual world and completing a purchase in the real world by pressing a purchase button, and means for selecting transportation and accommodation in the virtual world and completing a reservation in the real world by pressing a reservation button. This enables purchases and reservations made by a user in the virtual world to be seamlessly reflected in the real world.

[0386] A "digital twin" is a digital representation of a real-world physical object or environment.

[0387] A "pseudo-world" is a virtual space constructed using virtual reality or augmented reality.

[0388] "VR" stands for virtual reality, a technology that allows users to immerse themselves in a virtual space.

[0389] "AI" stands for artificial intelligence, a computer system that mimics human intelligence.

[0390] "Generative AI" is artificial intelligence that generates human-like characters in virtual space and interacts with users.

[0391] "EC" stands for electronic commerce, a system for buying and selling goods and services over the Internet.

[0392] "Transportation" refers to real-world transportation, including, for example, airplanes, trains, and taxis.

[0393] "Accommodation facilities" are facilities such as hotels and inns where users can stay.

[0394] "Information provision fee" is part of the fee paid by the user when using the service, and is paid as compensation for the provision of information.

[0395] "A means of completing a purchase in the real world by selecting a product in the virtual world and pressing the purchase button" is a system in which a user selects a product in the virtual space and presses the purchase button, causing that product to be purchased in the real world.

[0396] "A means of completing a reservation in the real world by selecting a means of transportation and accommodation within the virtual world and pressing a reservation button" is a system in which a user selects a means of transportation and accommodation within the virtual space and presses a reservation button, completing the reservation in the real world.

[0397] A system for implementing this invention uses a digital twin to generate a simulated world that users can experience in virtual reality (VR). The system includes a means for realizing purchasing and reservation actions in the virtual world in the real world.

[0398] Hardware and software used

[0399] Hardware: Smartphone, Head-Mounted Display (HMD)

[0400] Software: Unity (game engine), Python (backend processing), Firebase (database), Stripe (electronic payment)

[0401] Data processing and calculation

[0402] Building a Virtual World

[0403] The server uses Unity to build a virtual world that users can move freely through, populating it with 3D models of goods, transportation, and accommodations.

[0404] Product selection and purchase

[0405] When the user selects an item, details about that item are displayed within Unity. When the user presses the purchase button, the Python backend accesses the Firebase database to record the purchase, and Stripe is used to process the electronic payment and complete the purchase.

[0406] Transportation and accommodation bookings

[0407] Similarly, once a user selects their transportation and accommodation, a Python backend accesses a Firebase database to record the reservation information, which is then reflected in a real-world reservation system.

[0408] Specific examples

[0409] Product purchase: The user selects a smartphone in a store in the virtual world, presses the purchase button, and the smartphone is delivered to them in the real world.

[0410] Booking accommodation: The user selects a hotel in the virtual world and presses the reservation button, completing the hotel reservation in the real world.

[0411] Prompt Sentence Examples

[0412] Product Purchasing: "Build a system that allows you to purchase and deliver products selected in the virtual world in the real world."

[0413] Accommodation Booking: "Build a system that allows you to book accommodation selected in the virtual world in the real world."

[0414] In this way, a system can be realized in which actions in the virtual world are seamlessly reflected in the real world.

[0415] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0416] Step 1:

[0417] The server uses Unity to build the virtual world. As input, it receives data on real-world physical objects and environments, generates a digital twin, and as output, provides a virtual world in which users can freely move.

[0418] Step 2:

[0419] The user wears a head-mounted display (HMD) and accesses the virtual world. As input, the system receives positional and motion information from the HMD, which reflects the user's viewpoint and movement within the virtual world in real time. As output, the system provides the user with an immersive virtual experience.

[0420] Step 3:

[0421] The user selects a product in the virtual world. The system receives the user's selection as input and displays detailed information about the selected product. The system provides detailed information about the product to the user as output.

[0422] Step 4:

[0423] The user presses the Buy button. As input, the Buy button press information is received and sent to the Python backend. As output, the purchase information is provided to the Python backend.

[0424] Step 5:

[0425] The server uses a Python backend to access the Firebase database and record the purchase information. As input, it receives the purchase information and stores it in the Firebase database. As output, it provides the database with the purchase information recorded.

[0426] Step 6:

[0427] The server processes electronic payments using Stripe. As input, it receives purchase information and the user's payment information, calls the Stripe API to complete the payment, and as output, provides confirmation that the payment has been completed.

[0428] Step 7:

[0429] The user selects a means of transportation or accommodation in the virtual world. The system receives the user's selection as input and displays detailed information about the selected means of transportation or accommodation. The system provides detailed information about the selected means of transportation or accommodation to the user as output.

[0430] Step 8:

[0431] The user presses the reservation button. As input, the system receives the reservation button press information and sends it to the Python backend. As output, the system provides the reservation information to the Python backend.

[0432] Step 9:

[0433] The server uses a Python backend to access the Firebase database and record the reservation information. As input, it receives the reservation information and stores it in the Firebase database. As output, it provides the database with the reservation information.

[0434] Step 10:

[0435] The server reflects the reservation information to the real-world reservation system. As input, it receives the reservation information and sends it to the real-world reservation system. As output, it provides confirmation that the reservation has been completed.

[0436] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0437] "Example 1"

[0438] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and adjusts the experience in the simulated world according to the emotions. Specifically, when the user feels joy, the system adjusts the experience in the simulated world to be more enjoyable. For example, the system may adjust the weather in the simulated world to be sunny, or make the AI-generated humans behave more friendly toward the user.

[0439] "Example 2"

[0440] The emotion engine also adjusts the content of the conversation with the AI-generated human depending on the user's emotions. For example, if the user is sad, the AI-generated human will offer words of comfort. If the user is angry, the AI-generated human will offer words of apology. This allows users to experience appropriate responses in the virtual world according to their emotions.

[0441] "Example 3"

[0442] Furthermore, the emotion engine recognizes the user's emotions and supports purchasing behavior within the virtual world according to those emotions. For example, when a user feels happy, the system will recommend products that will increase that joy. When a user feels sad, the system will recommend products that will ease that sadness. This allows users to purchase appropriate products within the virtual world according to their emotions.

[0443] The processing flow of each embodiment will be described below.

[0444] "Example 1"

[0445] Step 1: The user logs into the virtual world through a VR device.

[0446] Step 2: The emotion engine recognizes the user's emotions, based on their facial expressions, tone of voice, and choice of words.

[0447] Step 3: The system adjusts the simulated world experience based on the emotions it recognizes. For example, if the user feels happy, the system might make the weather in the simulated world sunny or make the AI-generated human behave more friendly toward the user.

[0448] "Example 2"

[0449] Step 1: The user initiates a conversation with an AI-generated human.

[0450] Step 2: The emotion engine recognizes the user's emotions and adjusts the conversation with the AI-generated human accordingly. For example, if the user is sad, the AI-generated human will offer words of comfort; if the user is angry, the AI-generated human will offer words of apology.

[0451] "Example 3"

[0452] Step 1: The user begins shopping in the virtual world.

[0453] Step 2: The emotion engine recognizes the user's emotions and adjusts the product recommendations accordingly. For example, if the user is feeling happy, the system will recommend products that will increase that joy. If the user is sad, the system will recommend products that will alleviate that sadness.

[0454] Example 1

[0455] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional virtual reality systems have the problem of being difficult to generate detailed digital twins of places users want to visit, and they do not adjust the experience according to the user's emotions, resulting in low user satisfaction. Furthermore, purchasing within the virtual world and linking it to the real world are insufficient, resulting in a lack of user convenience.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0458] In this invention, the server includes: means for generating a virtual world using a digital twin; means for experiencing the virtual world in virtual reality; means for generating characters in the virtual world using artificial intelligence and enabling conversations with the generated artificial intelligence; means for enabling purchases in the virtual world through e-commerce; means for linking to real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means for recognizing a user's emotions and adjusting the virtual world experience in response to those emotions; and an emotion recognition engine for analyzing the user's emotions. This allows for the generation of a detailed digital twin of a place the user wants to visit and for the adjustment of the experience in response to the user's emotions. Furthermore, the link between the virtual world and the real world and the purchases in the virtual world is strengthened, improving user convenience.

[0459] A "digital twin" is a digital model that recreates a real-world physical place or object in a virtual space.

[0460] A "pseudo-world" is a virtual reality environment created using a digital twin.

[0461] "Virtual reality" is a technology that allows users to experience an artificial environment generated using computer technology.

[0462] "Artificial intelligence" is a technology that allows computers to act in a way that mimics human intelligence.

[0463] "Electronic commerce" refers to the buying and selling of goods and services over the Internet.

[0464] An "emotion recognition engine" is software that analyzes emotions from a user's facial expressions, voice, etc.

[0465] "Emotional experience tailoring" refers to modifying the virtual reality environment and interactions based on the user's emotional state.

[0466] "Information Fee" means the fee paid by a user for the information provided by the user.

[0467] "Transportation" refers to the means of transportation that a user uses to move around in the real world.

[0468] "Accommodation" refers to a facility where users can stay.

[0469] MODE FOR CARRYING OUT THE INVENTION

[0470] This invention is a system that uses digital twins to generate a simulated world that users can experience in virtual reality. It also has the ability to recognize the user's emotions and adjust the experience of the simulated world according to those emotions.

[0471] Digital Twin Generation

[0472] The server collects data on the location the user wants to visit. This data includes streetscapes, buildings, and scenery. Specifically, it uses the Google Maps API or OpenStreetMap API to obtain a 3D model of the specified location and surrounding landscape data. The server then uses this data to generate a digital twin using Unity or Unreal Engine.

[0473] Example: If a user requests to "visit the Eiffel Tower in Paris," the server uses the Google Maps API to collect a 3D model of the Eiffel Tower and surrounding landscape data, and then uses Unity to generate a digital twin of the Eiffel Tower.

[0474] Example prompt sentence:

[0475] A user wants to visit the Eiffel Tower in Paris. Collect a 3D model of the Eiffel Tower and surrounding landscape data, and generate a simulated world using Unity.

[0476] Virtual reality experience

[0477] The device then presents the generated digital twin to the user, who can then experience the virtual world through a VR headset or monitor. The device tracks the user's movements and gaze, updating the virtual world in real time.

[0478] AI-powered conversation

[0479] The server uses artificial intelligence to generate characters in the virtual world and enables them to converse with users. For example, if a user asks for directions within the virtual world, the AI ​​will provide an appropriate response.

[0480] Enabling e-commerce

[0481] The server realizes purchasing actions within the virtual world through e-commerce. Users can select products in a virtual store and complete the purchase process. Purchase data is securely managed by the server.

[0482] Emotion recognition and experience regulation

[0483] The device uses a camera and microphone to capture the user's facial expressions and voice. The server analyzes the user's emotions from the captured data using Microsoft® Azure® Emotion API or Google Cloud's Vision API. Based on the analyzed emotions, the server adjusts the simulated world experience. For example, if the user is feeling happy, the server sets the weather in the simulated world to sunny and makes the AI-generated character greet the user with a smile.

[0484] Example: If the server detects that the user is having fun in the simulated world, it will set the weather to sunny and adjust the AI-generated characters to greet the user with a smile.

[0485] Example prompt sentence:

[0486] The user is recognized as having fun in the simulated world. Please set the weather in the simulated world to sunny and adjust the AI-generated characters to greet the user with a smile.

[0487] The system allows users to experience a detailed digital twin of the place they want to visit, providing a personalized experience tailored to their emotions, and enhancing user convenience through enhanced purchasing behavior within the virtual world and links to the real world.

[0488] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0489] Step 1:

[0490] The user inputs the location they want to visit into the terminal, for example, "Eiffel Tower in Paris." The input data is sent to the server.

[0491] Input: Places the user wants to visit

[0492] Output: Places to visit data sent to the server

[0493] Step 2:

[0494] The server collects data for the specified location via the Internet, specifically using the Google Maps API and OpenStreetMap API to obtain a 3D model of the Eiffel Tower and surrounding landscape data.

[0495] Input: Data about places the user wants to visit

[0496] Output: 3D models and landscape data of collected locations

[0497] Step 3:

[0498] The server generates a digital twin using Unity or Unreal Engine based on the collected data. Specifically, it imports a 3D model of the Eiffel Tower into Unity and places the surrounding landscape data.

[0499] Input: 3D model and landscape data of collected locations

[0500] Output: Digital twin generated in Unity or Unreal Engine

[0501] Step 4:

[0502] The server then sends the generated digital twin to the device, allowing the user to experience the virtual world. The device then displays the virtual world to the user through a VR headset or monitor.

[0503] Input: Digital twin generated in Unity or Unreal Engine

[0504] Output: Digital twin sent to the device

[0505] Step 5:

[0506] The device uses a camera and microphone to capture the user's facial expressions and voice. Specifically, the camera takes a picture of the user's face and the microphone records the user's voice.

[0507] Input: User's facial expressions and voice

[0508] Output: Captured facial and voice data of the user

[0509] Step 6:

[0510] The server uses Microsoft Azure's Emotion API or Google Cloud's Vision API to analyze the user's emotions from the captured data. For example, if the user is smiling, it will recognize the emotion as "joy."

[0511] Input: Captured facial and voice data of the user

[0512] Output: Parsed user sentiment data

[0513] Step 7:

[0514] The server adjusts the simulated world experience based on the analyzed emotions: for example, if the user is feeling happy, it will set the weather in the simulated world to sunny and have the AI-generated characters greet the user with a smile.

[0515] Input: Parsed user sentiment data

[0516] Output: Adjusted pseudo-world configuration

[0517] Step 8:

[0518] The server then sends the adjusted virtual world to the device, allowing the user to experience the changes. The device then displays the adjusted virtual world to the user through a VR headset or monitor.

[0519] Input: Adjusted pseudo-world settings

[0520] Output: The adjusted virtual world sent to the device.

[0521] (Application example 1)

[0522] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0523] Conventional virtual store systems do not adjust the experience according to the user's emotions, making it difficult to increase user satisfaction. Furthermore, to provide an experience similar to that of a real-world store, it is necessary to dynamically change the environment and the responses of store staff according to the user's emotions, but such a system has not existed.

[0524] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in VR, means for generating people in the pseudo-world using AI and enabling conversations with the generative AI, means for realizing purchases in the pseudo-world through e-commerce, means for linking with real-world transportation and accommodations, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience in the pseudo-world in accordance with the emotions, and means for changing the environment in the virtual store and the behavior of the store clerk based on the adjustments. This makes it possible to provide a dynamic experience that corresponds to the user's emotions.

[0525] A "digital twin" is a virtual model that recreates a real-world physical place or object in a digital space.

[0526] A "pseudo-world" is a virtual space created using digital twin technology, and is a virtual environment that users can experience.

[0527] "VR" is an abbreviation for virtual reality, a technology that allows users to visually experience a virtual space.

[0528] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to act in a way that mimics human intelligence.

[0529] "Generative AI" is a system that uses artificial intelligence to generate people and objects in virtual space.

[0530] "EC" stands for electronic commerce, the act of purchasing goods and services over the Internet.

[0531] "Emotion recognition" is a technology that analyzes a user's facial expressions and behavior to identify their emotions.

[0532] A "virtual store" is a store that exists in a virtual space, a virtual commercial facility where users can browse and purchase products.

[0533] "Environmental adjustment" refers to dynamically changing environmental elements such as weather, music, and lighting in a virtual space according to the user's emotions.

[0534] "Store staff response" refers to the service and assistance provided to users by AI-generated humans in the virtual space.

[0535] The embodiments of the present invention will be described in detail below.

[0536] System Overview

[0537] The system of this invention uses digital twin technology to generate a simulated world that users can experience in VR. It also uses emotion recognition technology to analyze the user's emotions and dynamically adjust the experience in the simulated world based on those emotions.

[0538] Hardware and software used

[0539] Hardware:

[0540] Smartphone or head-mounted display (HMD)

[0541] Camera (smartphone camera or HMD camera)

[0542] software:

[0543] OpenCV (camera image acquisition and processing)

[0544] EmotionRecognizer (emotion recognition library)

[0545] VirtualWorld (Digital Twin Generation and Rendering Library)

[0546] System Operation

[0547] Digital Twin Generation

[0548] The server acquires data on the location the user wants to visit (such as cityscapes, buildings, and scenery) and generates a digital twin. This data can be obtained from the Internet. The generated digital twin is sent to the user's device and displayed in a VR environment.

[0549] emotion recognition

[0550] The user's device captures the user's facial expressions in real time using a camera and analyzes their emotions using EmotionRecognizer. The analyzed emotional data is then sent to the server.

[0551] Tailoring the experience

[0552] The server dynamically adjusts the environment (weather, music, lighting, etc.) in the virtual world and the behavior of store staff based on the received emotional data. For example, if the user is happy, the server will adjust the music in the store to be brighter and the staff to be more friendly.

[0553] Specific examples

[0554] Users access the virtual store using their smartphones and walk around the store, which recreates the streets of Paris. When the user smiles, the music in the store brightens and a sales associate greets them in a friendly manner, asking, "Hello! What are you looking for today?"

[0555] Prompt Sentence Examples

[0556] Capture data on the places users want to visit and generate a digital twin. Recognize users' emotions and tailor the virtual in-store experience accordingly. For example, if a user is happy, make the music brighter and the staff more friendly.

[0557] In this way, the system of the present invention can provide a dynamic experience that responds to the user's emotions.

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] The user selects a place they want to visit. Using a smartphone or a head-mounted display (HMD), the user selects the place they want to visit (e.g., a street view of Paris). The input is the information about the user's selected place, and the output is a request to obtain data about that place.

[0561] Step 2:

[0562] The server retrieves data on the selected location. The server retrieves data on the streetscape, buildings, scenery, etc. of the selected location via the Internet. The input is the information on the location selected by the user, and the output is the retrieved location data.

[0563] Step 3:

[0564] The server generates a digital twin. The server generates a digital twin based on the acquired location data. The input is the acquired location data, and the output is the generated digital twin.

[0565] Step 4:

[0566] The terminal displays the digital twin. The terminal receives the digital twin sent from the server and displays it in a VR environment. The input is the generated digital twin, and the output is the VR environment experienced by the user.

[0567] Step 5:

[0568] The device recognizes the user's emotions. The device uses a camera to capture the user's facial expressions in real time and analyzes the emotions using EmotionRecognizer. The input is the camera image, and the output is the analyzed emotional data.

[0569] Step 6:

[0570] The device sends emotional data to the server. The device sends analyzed emotional data to the server. The input is the analyzed emotional data, and the output is the emotional data sent to the server.

[0571] Step 7:

[0572] The server adjusts the experience based on the emotional data. Based on the received emotional data, the server dynamically adjusts the environment (weather, music, lighting, etc.) in the virtual world and the responses of the store staff. The input is emotional data, and the output is the adjusted environmental settings.

[0573] Step 8:

[0574] The device displays the calibrated environment. The device receives the calibrated environment settings sent from the server and displays them in the VR environment. The input is the calibrated environment settings, and the output is the calibrated VR environment experienced by the user.

[0575] Example 2

[0576] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0577] In conventional virtual reality systems, the virtual world experienced by the user is fixed, making it difficult to provide interactive responses based on the user's emotions. Furthermore, conversations within the virtual world are monotonous, making it impossible to respond appropriately to the user's emotions, making it impossible to provide a realistic experience. Furthermore, the link with the real world is insufficient, resulting in low user convenience.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0579] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating humans in the simulated world using artificial intelligence and enabling conversations with the generative AI, means for enabling purchases in the simulated world through e-commerce, means for linking with transportation and accommodations in the real world, and means for analyzing the user's emotions and adjusting the content of the conversation based on the analysis results, allowing the user to experience a realistic simulated world and receive appropriate responses according to their emotions.

[0580] A "digital twin" is a virtual model that recreates real-world physical objects and environments in a digital space.

[0581] A "pseudo-world" is a virtual environment that users can experience, generated using virtual reality technology.

[0582] "Virtual reality" is a technology that immerses users in a virtual environment generated using computer technology.

[0583] "Artificial intelligence" is a technology that allows computers to imitate human intelligence and have functions such as learning, reasoning, and recognition.

[0584] "Generative AI" is AI that has the ability to generate new data or content based on specific input.

[0585] "Electronic commerce" is a form of transaction in which goods and services are bought and sold over the Internet.

[0586] "Emotion analysis" is a technology that identifies and analyzes emotions from a user's tone of voice, facial expressions, etc.

[0587] "Conversation content adjustment" is the process of appropriately changing the content of the conversation provided by generative artificial intelligence based on the results of user emotion analysis.

[0588] "Transportation" refers to the means of transportation that a user uses to move around in the real world.

[0589] "Accommodation" means a facility where users can stay temporarily.

[0590] This invention provides a system in which a user experiences a simulated world using a virtual reality (VR) device and has an interactive conversation with a human generated by artificial intelligence (AI). Specific embodiments of this system are described below.

[0591] Hardware and Software Use

[0592] Hardware: VR devices such as Oculus Rift and HTC Vive are used, allowing users to experience a highly immersive experience.

[0593] Software: Game engines such as Unity and Unreal Engine are used to generate the VR environment, OpenAI's GPT-3 is used as the generative AI model, and emotion recognition software is used for emotion analysis.

[0594] Data processing and calculation

[0595] Using a VR device: When a user wears a VR device, the device generates the Paris cityscape using Unity or Unreal Engine, providing a 360-degree view, allowing the user to experience the cityscape in a realistic way.

[0596] Use of generative AI models: The server uses generative AI models such as OpenAI's GPT-3 to generate humans in the simulated world, allowing users to ask for directions and engage in everyday conversations.

[0597] Use of emotion engine: The server uses emotion recognition software to analyze the user's emotions in real time. Based on the analysis results, the generative AI model adjusts the conversation content. For example, if the user is sad, it will offer words of comfort, and if the user is angry, it will offer words of apology.

[0598] Specific examples

[0599] The user puts on the VR device: The user puts on the Oculus Rift on their head and holds the controllers in their hands.

[0600] The device launches the VR environment: The device launches Unity and displays a 360-degree view of the Paris cityscape.

[0601] The server generates a simulated world: The server uses GPT-3 to generate cafe staff and people passing by.

[0602] The user moves through the virtual world: The user operates the controller and walks down the Champs-Élysées.

[0603] A user converses with an AI-generated human: A user asks a cafe waiter, "What do you recommend on the menu?"

[0604] The server analyzes the user's emotions: The server analyzes the user's tone of voice and determines that the user is a little tired.

[0605] The server adjusts the conversation content: The server sends a prompt to the generative AI model, such as "You seem particularly tired today. I'd like to introduce you to a cafe where you can relax," and generates an appropriate conversation.

[0606] Prompt Sentence Examples

[0607] "When a user gets lost, generate a conversation where an AI-generated human can provide appropriate directions."

[0608] "When a user asks a cafe staff member for a recommended menu item, please generate a conversation in which the staff member gives an appropriate answer."

[0609] This system allows users to experience the real city of Paris from the comfort of their own home and receive appropriate responses based on their emotions.

[0610] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0611] Step 1:

[0612] The user puts on the VR device.

[0613] Input: User puts on a VR device (e.g., Oculus Rift).

[0614] Output: The user is ready to enter the VR environment.

[0615] Specific actions: The user wears a VR device on their head and holds controllers in their hands.

[0616] Step 2:

[0617] The device will launch the VR environment.

[0618] Input: User wearing the VR device.

[0619] Output: The VR environment is displayed.

[0620] What it does: The device launches Unity or Unreal Engine, displaying a 360-degree view of the Paris cityscape.

[0621] Step 3:

[0622] The server generates a pseudo-world.

[0623] Input: VR environment activation signal from device.

[0624] Output: A human in a simulated world is generated.

[0625] How it works: The server uses OpenAI's GPT-3 to generate cafe staff and passersby.

[0626] Step 4:

[0627] The user moves through the virtual world.

[0628] Input: User controller actions.

[0629] Output: The user's viewpoint moves.

[0630] Specific action: The user operates the controller and walks down the Champs-Élysées.

[0631] Step 5:

[0632] The user converses with an AI-generated human.

[0633] Input: User's voice input.

[0634] Output: AI-generated human response.

[0635] Specific behavior: When a user asks a cafe staff member, "What's your recommendation on the menu?", the AI-generated human responds, "Today's recommendation is croissants and cafe au lait."

[0636] Step 6:

[0637] The server analyzes the user's emotions.

[0638] Input: User's voice and facial expression data.

[0639] Output: The user's emotional state.

[0640] What it does: The server uses emotion recognition software to analyze the user's tone of voice and facial expressions and determines that the user is a little tired.

[0641] Step 7:

[0642] The server moderates the conversation.

[0643] Input: The user's emotional state.

[0644] Output: The adjusted conversation.

[0645] Specific operation: The server sends a prompt to the generative AI model, such as "You seem particularly tired today. I'd like to introduce you to a cafe where you can relax," and generates an appropriate conversation.

[0646] (Application example 2)

[0647] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0648] While modern consumers enjoy the convenience of online shopping, they also desire the experience of face-to-face customer service and seeing products in person at a physical store. However, traditional online shopping has the drawback of not providing users with sufficient information when selecting products, and not being able to receive appropriate support based on their emotions. Furthermore, it has been difficult to provide a shopping experience customized to the user's emotions.

[0649] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0650] In this invention, the server includes: means for generating a virtual world using a digital twin; means for experiencing the virtual world in VR; means for generating humans in the virtual world using AI and enabling conversations with the generative AI; means for enabling purchases in the virtual world through e-commerce; means for linking with real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine for analyzing a user's emotions and generating conversation content according to the emotions; and means for selecting products while conversing with an AI-generated sales clerk in a virtual store. This allows users to enjoy a realistic shopping experience in the virtual store and receive appropriate responses according to their emotions.

[0651] A "digital twin" is a virtual model that recreates real-world physical objects and environments in a digital space.

[0652] A "pseudo-world" is a virtual space created using a digital twin, which users can experience through VR devices.

[0653] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space through their eyes and ears.

[0654] An "AI-generated human" is a human character generated in a virtual space using artificial intelligence technology.

[0655] "Generative AI" is an artificial intelligence system that generates conversations and interactions with users.

[0656] "EC" is an abbreviation for electronic commerce, a system for buying and selling goods and services over the Internet.

[0657] The "emotion engine" is a system that analyzes the user's emotions and generates appropriate responses and conversation content based on those emotions.

[0658] A "virtual store" is a store built in a virtual space that users can visit through a VR device.

[0659] An "AI-generated store clerk" is a store clerk character generated using artificial intelligence technology to interact with users in a virtual store.

[0660] A system for implementing this invention is configured as follows: First, a user puts on a VR device and accesses a virtual store. The VR device can be a commercially available VR headset such as Oculus Rift. When the user enters the virtual store, a simulated world created using digital twin technology is displayed.

[0661] The server uses the digital twin to generate a virtual world that users can experience through a VR device. AI-generated humans (store clerks) are placed within the virtual world and can converse with the user. The generative AI uses OpenAI's API to generate conversations with the user.

[0662] When a user selects a product in a virtual store, the emotion engine analyzes the user's emotions and responds appropriately based on those emotions. The emotion engine analyzes the user's voice input and facial expression data, generating words of comfort if the user is sad and words of apology if the user is angry.

[0663] As a concrete example, consider a scenario where a user is looking for shoes in a virtual store. When the user asks, "Do you have this shoe in my size?", the emotion engine analyzes the user's emotions and generates an appropriate prompt. For example, if the user is sad, the engine generates a prompt such as, "When the user is sad, offer comforting words."

[0664] Example prompt sentence:

[0665] "When a user asks for their shoe size, we suggest the appropriate size."

[0666] The server sends this prompt to OpenAI's API, which generates a dialogue in which an AI-generated store assistant suggests an appropriate size, which is then displayed to the user through the VR device.

[0667] This system allows users to enjoy a realistic shopping experience in a virtual store and receive appropriate responses based on their emotions, providing a new shopping experience that combines the convenience of online shopping with the face-to-face customer service of a physical store.

[0668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0669] Step 1:

[0670] The user puts on the VR device and accesses the virtual store.

[0671] Input: User puts on VR device and requests access to virtual store

[0672] Output: Connection established to virtual store

[0673] Specific operation: The user puts on the VR device and launches the virtual store application. The server receives the user's access request and establishes a connection to the virtual store.

[0674] Step 2:

[0675] The server uses the digital twin to generate a simulated world and displays it to the user.

[0676] Input: Digital twin data of virtual store

[0677] Output: A virtual world displayed on the user's VR device

[0678] How it works: The server uses digital twin technology to generate a simulated world of a virtual store and transmits it to the user's VR device, through which the user experiences the simulated world.

[0679] Step 3:

[0680] When a user selects a product in a virtual store, the emotion engine analyzes the user's emotions.

[0681] Input: User voice input and facial expression data

[0682] Output: User sentiment analysis results

[0683] How it works: When a user asks a question or makes a comment about a product, the VR device sends voice input and facial expression data to the server, which then uses an emotion engine to analyze this data and identify the user's emotions.

[0684] Step 4:

[0685] The server generates prompt sentences according to the emotion and sends them to the generative AI model.

[0686] Input: User sentiment analysis results

[0687] Output: Generated prompt statement

[0688] Specific operation: The server generates an appropriate prompt based on the analysis results of the emotion engine. For example, if the user is sad, the server generates a prompt such as "When the user is sad, say some comforting words."

[0689] Step 5:

[0690] The generative AI model generates conversation content based on the prompt sentence and sends it to the server.

[0691] Input: Generated prompt text

[0692] Output: Generated conversation

[0693] Specific operation: The server sends the generated prompt sentence to the generative AI model, which then generates appropriate conversation content based on the prompt sentence and returns the result to the server.

[0694] Step 6:

[0695] The server sends the generated conversation content to the user's VR device and displays it.

[0696] Input: Generated conversation

[0697] Output: The conversation displayed on the user's VR device

[0698] Specific operation: The server receives the conversation content from the generative AI model and sends it to the user's VR device, where the user experiences a conversation with the AI-generated store clerk.

[0699] Step 7:

[0700] When a user purchases an item in a virtual store, the server processes the purchase through the EC system.

[0701] Input: User purchase request

[0702] Output: Purchase completion notification

[0703] Specific operation: When the user selects a product and decides to purchase it, the server processes the purchase through the EC system. Once the purchase is complete, the server sends a purchase completion notification to the user.

[0704] Example 3

[0705] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0706] There is a demand for a system that can efficiently realize purchasing and reservation activities in a virtual world linked to the real world. Another challenge is to improve the user experience by recommending products based on the user's emotions. Conventional systems do not take user emotions into account when recommending products, making it difficult to increase user satisfaction.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0708] In this invention, the server includes: means for generating a simulated world using a digital twin; means for experiencing the simulated world in virtual reality; means for generating characters in the simulated world using artificial intelligence and enabling conversations with the generative artificial intelligence; means for making purchases in the simulated world through e-commerce; means for linking with transportation and accommodation in the real world; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine that recognizes a user's emotions and recommends products based on those emotions; means for the emotion engine to transmit user emotion data to the server and for the server to select products based on those emotions; and means for transmitting the selected product information to the user's terminal and for the terminal to display the recommended products. This enables product recommendations based on the user's emotions, improving the user experience.

[0709] "Digital twin" is a technology that recreates real-world physical objects and environments in a virtual space.

[0710] A "pseudo-world" is a virtual space created using digital twin technology, an environment that users can experience through virtual reality.

[0711] "Virtual reality" is a technology that allows users to experience a virtual space generated using computer technology through their eyes and ears.

[0712] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0713] "Generative AI" is a system that uses artificial intelligence technology to generate people and objects in a virtual space, enabling dialogue and interaction with users.

[0714] "Electronic commerce" is a form of transaction in which goods and services are bought and sold via the Internet.

[0715] "Transportation" means the means of transportation used by a User to travel from one location to another.

[0716] "Accommodation facilities" are facilities where users can stay temporarily, including hotels and inns.

[0717] "Information provision fee" is part of the fee paid by users when they use the system, and is compensation for the information and services provided.

[0718] An "emotion engine" is a system that recognizes a user's emotions and recommends appropriate products and services based on those emotions.

[0719] "Emotion data" is information about emotions acquired from the user's facial expressions, voice, etc.

[0720] A "server" is a computer system that stores, processes, and manages data on a network.

[0721] A "terminal" is a device that a user uses to access and operate the system, including a PC or smartphone.

[0722] "Recommended products" are products selected and suggested by the system based on the user's emotions and preferences.

[0723] This invention is a system in which users can select and purchase products in a virtual world, and the products are delivered in the real world. It is also linked to real-world transportation and accommodation options, so that selections made in the virtual world are reflected in real-world reservations. Furthermore, it uses an emotion engine to recognize the user's emotions and recommend products based on those emotions.

[0724] Hardware and software used

[0725] Server: Database management system (e.g., MySQL), web server (e.g., Apache), emotion engine (e.g., IBM Watson®)

[0726] Device: User interface (e.g. web browser, mobile app)

[0727] Generative AI models: Natural language processing models (e.g., GPT-4)

[0728] Specific operation of the system

[0729] 1. The user selects a product in a virtual world

[0730] A user accesses a store in the virtual world using a terminal. When the user selects a product and clicks the purchase button, the terminal sends information about the selected product to the server.

[0731] 2. Sending and saving product information

[0732] The device sends the selected product information in JSON format to the server. The server saves the received product information in a database (e.g., MySQL) and returns a confirmation message to the device that the information has been saved.

[0733] 3. Execution of delivery arrangements

[0734] The server coordinates with the delivery system (e.g., FedEx API) to arrange delivery. The server receives delivery confirmation information from the delivery system and notifies the user's device of the delivery confirmation information.

[0735] 4. Choosing transportation and accommodation

[0736] When the user selects transportation and accommodation within the virtual world and clicks the reservation button, the device sends the selected information to the server.

[0737] 5. Sending and storing reservation information

[0738] The device sends the selected reservation information in JSON format to the server. The server saves the received reservation information in a database and makes the reservation by connecting to a reservation system (e.g., Booking.com API). The server then notifies the user's device of the reservation confirmation information.

[0739] 6. Emotion Recognition by Emotion Engine

[0740] When a user uses the device to explore the virtual world, the device uses a camera and microphone to capture the user's facial expressions and voice. The device sends the captured data to an emotion engine (e.g., IBM Watson), which analyzes the user's emotions in real time and sends the data to a server.

[0741] 7. Emotion-based product recommendations

[0742] The server selects products that correspond to the user's emotions based on the emotion data, and sends the selected product information to the device, which then displays the recommended products to the user.

[0743] Examples and prompts

[0744] Example 1: Product Purchase

[0745] The user selects a smartphone in a virtual electronics store and clicks the purchase button.

[0746] The server receives the information from the smartphone and works with the delivery system to arrange delivery in the real world.

[0747] Users will receive their smartphone in the real world a few days later.

[0748] Example 2: Booking accommodation

[0749] The user selects a hotel in the virtual world and clicks the reservation button.

[0750] The server receives the hotel information and works with the reservation system to make reservations in the real world.

[0751] The user receives a hotel booking confirmation in the real world.

[0752] Prompt Sentence Examples

[0753] "Please explain the steps to deliver the product selected by the user in the virtual world to the real world."

[0754] "Please explain how product recommendations based on user emotions work."

[0755] The above is an embodiment of the present invention. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0756] Step 1:

[0757] The user selects a product in the virtual world

[0758] Input: The user uses a device to access a store in the virtual world and select a product.

[0759] Specific behavior: The user selects a product and clicks the purchase button.

[0760] Output: The device sends the selected product information to the server in JSON format.

[0761] Step 2:

[0762] Sending and storing product information

[0763] Input: Product information in JSON format sent from the device.

[0764] Specific operation: The server analyzes the received product information and saves the "user ID, product ID, purchase date and time" in a database (e.g. MySQL).

[0765] Output: The server returns a confirmation message to the terminal that the save was successful.

[0766] Step 3:

[0767] Executing delivery arrangements

[0768] Input: Product information stored in the database.

[0769] Specific operation: The server connects to the delivery system (e.g., FedEx API) and sends the user's address, product ID, and delivery date and time. The delivery system returns the delivery ID and expected delivery date and time to the server.

[0770] Output: The server notifies the user's device of the "Delivery ID, scheduled delivery date and time."

[0771] Step 4:

[0772] Choosing transportation and accommodation

[0773] Input: The user uses the device to select transportation and accommodation within the simulated world.

[0774] Specific operation: The user clicks the reservation button, and the device sends the selected information in JSON format to the server.

[0775] Output: The terminal sends the selected reservation information to the server.

[0776] Step 5:

[0777] Sending and saving reservation information

[0778] Input: Reservation information sent from the terminal in JSON format.

[0779] Specific operation: The server analyzes the received reservation information and saves the "user ID, reservation ID, reservation date and time" in the database. The server connects to the reservation system (e.g. Booking.com API) and sends the "user name, reservation ID, departure date and time". The reservation system returns the "reservation confirmation number, departure date and time" to the server.

[0780] Output: The server notifies the user's terminal of the reservation confirmation number and departure date and time.

[0781] Step 6:

[0782] Emotion recognition by emotion engine

[0783] Input: Facial and voice data as the user uses the device to explore the simulated world.

[0784] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, and sends them to an emotion engine (e.g., IBM Watson). The emotion engine analyzes the user's emotions in real time and sends the data to a server.

[0785] Output: The emotion engine sends the analysis results to the server.

[0786] Step 7:

[0787] Emotion-based product recommendations

[0788] Input: Emotion data sent from the emotion engine.

[0789] Specific operation: The server selects products that correspond to the user's emotions based on the emotion data, and then sends the selected product information to the terminal.

[0790] Output: The device displays the recommended products to the user.

[0791] (Application example 3)

[0792] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0793] There is a need to optimize purchasing and reservation activities in virtual worlds linked to the real world based on user emotions. Conventional systems provide products and services without considering user emotions, resulting in a limited user experience and low satisfaction. Furthermore, there is no adequate system in place to reflect choices made in virtual worlds in deliveries and reservations in the real world.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in VR, means for generating a person in the pseudo-world using AI and enabling conversation with the generative AI, means for realizing purchases in the pseudo-world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions using an emotion engine and recommending products corresponding to the emotions, means for delivering products selected in the pseudo-world in the real world, and means for reserving transportation and accommodation selected in the pseudo-world in the real world. This makes it possible to provide optimal products and services according to the user's emotions, and realizes a system in which selections in the pseudo-world are reflected in deliveries and reservations in the real world.

[0795] A "digital twin" is a digital representation of a real-world physical object or environment.

[0796] A "pseudo-world" is a virtual environment created using a digital twin, which users can experience using virtual reality (VR) technology.

[0797] "VR" is an abbreviation for Virtual Reality, a technology that allows users to experience a three-dimensional space generated by a computer.

[0798] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to learn and reason by imitating human intelligence.

[0799] "Generative AI" is a technology that uses artificial intelligence to generate people and objects in a simulated world, enabling them to interact with users.

[0800] "EC" is an abbreviation for Electronic Commerce, a system for buying and selling goods and services over the Internet.

[0801] An "emotion engine" is a technology that recognizes a user's emotions and responds or makes recommendations based on those emotions.

[0802] "Means of transportation" refers to the means of transportation that users use to move around in the real world, including, for example, cars, trains, and airplanes.

[0803] "Accommodation" refers to facilities where users can stay in the real world, including hotels and inns.

[0804] "Information Fee" means the fee paid for the information provided by the User, and is treated as part of the service fee.

[0805] "Delivery" refers to the act of delivering the purchased product to the location specified by the user.

[0806] "Reservation" means the act of a user reserving a specific service or facility in advance.

[0807] A system for implementing this invention generates a simulated world using a digital twin and allows users to experience that simulated world using virtual reality (VR) technology. The system includes the following main components:

[0808] 1. Digital twin generation method:

[0809] The server uses 3D modeling software and sensor technology to create digital twins of real-world physical objects and environments in a digital space.

[0810] 2. VR experience method:

[0811] Users experience the generated virtual world using a head-mounted display (HMD) or smartphone, with specific hardware such as Oculus Rift or HTC Vive.

[0812] 3. AI generation means:

[0813] The server uses artificial intelligence (AI) to generate people and objects in the simulated world and enable user interaction using generative AI models.

[0814] 4. EC Implementation Methods:

[0815] The server provides an electronic commerce (EC) system that allows users to select and purchase products in the virtual world. Purchased products are delivered in the real world.

[0816] 5. Emotion Engine:

[0817] The server recognizes the user's emotions in real time and recommends products based on those emotions. Emotion recognition is performed using OpenCV and TENSORFLOW (registered trademark) with a camera and microphone.

[0818] 6. Links to the real world:

[0819] The travel and accommodation choices users make in the virtual world are reflected in their real-world reservations, including integration with booking systems via APIs.

[0820] 7. Information Fee Method:

[0821] The server includes a mechanism for paying a portion of the usage fee for the information provided by the user as an information provision fee.

[0822] As a concrete example, consider a scenario where a user is searching for a new smartphone in a virtual store: if the user is feeling happy, the system will recommend the latest smartphone model.

[0823] Example prompts to input to a generative AI model:

[0824] A user is searching for a new smartphone in a virtual store. Recommend the latest smartphone model because they are excited about it.

[0825] This system allows users to choose the most suitable products and services based on their emotions, and their choices in the virtual world are reflected in their delivery and reservation in the real world, greatly improving the user experience.

[0826] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0827] Step 1:

[0828] The server generates digital twins to recreate real-world physical objects and environments in a digital space. It receives real-world data (e.g., 3D scan data or sensor data) as input and generates the digital twins using 3D modeling software. The output is a 3D model of the simulated world.

[0829] Step 2:

[0830] The user experiences the generated virtual world using a head-mounted display (HMD) or a smartphone. As input, the system receives a 3D model of the virtual world sent from the server and displays it through a VR application. The output is the visual information of the virtual reality that the user experiences.

[0831] Step 3:

[0832] The server uses artificial intelligence (AI) to generate people and objects in the simulated world, enabling them to interact with the user. It receives user behavior data and situational data from the simulated world as input, and uses a generative AI model to generate appropriate people and objects. The output is interactive characters and objects in the simulated world.

[0833] Step 4:

[0834] A user uses an electronic commerce (EC) system to select and purchase products in the virtual world. The system receives information about the product selected by the user as input and performs the purchase procedure through the EC system. The output is purchase confirmation information and delivery arrangement information.

[0835] Step 5:

[0836] The server uses an emotion engine to recognize the user's emotions in real time and recommends products according to those emotions. As input, it receives data on the user's facial expressions and voice captured using a camera and microphone, and analyzes the emotions using an emotion recognition algorithm. The output is product recommendation information according to the user's emotions.

[0837] Step 6:

[0838] The transportation and accommodations selected by the user in the virtual world are reflected in the reservation in the real world. The system receives information on the transportation and accommodations selected by the user as input and connects to the reservation system via an API. The output is reservation confirmation information.

[0839] Step 7:

[0840] The server includes a mechanism for paying a portion of the usage fee for the information provided by the user as an information provision fee. It receives the information provided by the user as input and calculates the usage fee. The output is payment confirmation information for the information provision fee.

[0841] In this way, through the specific actions and data flow at each step, it becomes possible to provide optimal products and services according to the user's emotions, and a system is realized in which choices made in the virtual world are reflected in deliveries and reservations in the real world.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0843] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0845] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0846] [Second embodiment]

[0847] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0848] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0849] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0850] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0851] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0852] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0853] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0854] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0855] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0856] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0857] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0858] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0859] "Example 1"

[0860] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on places a user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, data on the city's streets, buildings, and scenery is acquired, and a simulated world of Paris is generated based on that data.

[0861] "Example 2"

[0862] The generated simulated world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their own home. The people in the simulated world are also generated by AI, and can converse with the generative AI. This allows the user to ask for directions or enjoy everyday conversations.

[0863] "Example 3"

[0864] Furthermore, purchasing within the virtual world is realized through e-commerce. When a user selects and purchases a product at a store within the virtual world, the product is delivered to the user in the real world. It is also linked to real-world transportation and accommodation options, so when a user selects transportation or accommodation within the virtual world, that option is reflected in their reservation in the real world.

[0865] The processing flow of each embodiment will be described below.

[0866] "Example 1"

[0867] Step 1: The user inputs the place they want to visit into the system. For example, if the user wants to visit Paris, they input the information Paris into the system.

[0868] Step 2: The system retrieves data for the relevant location based on the input information. This data includes information on streetscapes, buildings, landscapes, etc.

[0869] Step 3: A digital twin is created based on the acquired data. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[0870] "Example 2"

[0871] Step 1: The user puts on the VR device.

[0872] Step 2: Experience the simulated world created through the VR device. Users can experience walking through the streets of Paris from the comfort of their own home.

[0873] Step 3: Humans in the virtual world are generated by AI and can converse with the generated AI. Users can ask the AI-generated humans for directions or enjoy everyday conversations.

[0874] "Example 3"

[0875] Step 1: The user selects and purchases a product from a store in the virtual world.

[0876] Step 2: The purchased product is delivered to the user in the real world via the e-commerce system.

[0877] Step 3: The user selects transportation and accommodation in the virtual world, which is then reflected in the booking in the real world.

[0878] Example 1

[0879] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional digital twin technology requires a complex process for users to virtually experience the places they want to visit, and lacks a system that users can easily access. Furthermore, there is a lack of means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This makes it difficult for users to have a virtual experience in real time.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0882] In this invention, the server includes means for inputting a place a user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin, thereby enabling the user to easily virtually experience the places they want to visit.

[0883] A "user" is an individual or group who uses the system to specify the place they want to visit and virtually experience it.

[0884] "Input" refers to the act of a user specifying to the system the places they wish to visit, or the information therein.

[0885] "Geographical data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layout.

[0886] "Analysis" is the process of calculations and data processing used to process collected geographic data and convert it into a 3D model.

[0887] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographic data.

[0888] A "digital twin" is a 3D model that recreates a specific location in the real world in a virtual space.

[0889] "Sending" is the act of transferring the digital twin generated by the server to the user's device.

[0890] "Operation" refers to the act of a user interactively using a digital twin on a terminal.

[0891] This invention is a system that allows users to virtually experience places they would like to visit. The system involves a series of processes: a user inputs the place they would like to visit, collects geographical data about the place, analyzes it, converts it into a 3D model, generates a digital twin, and transmits it to the user's device.

[0892] Hardware and software used

[0893] server

[0894] The server receives user input, collects and analyzes geographic data, and generates the 3D model. Specifically, it uses the following software and services:

[0895] Google Maps API: Used to collect geographic data.

[0896] OpenStreetMap: Used to collect complementary geographic data.

[0897] Python: Run scripts to analyze collected geographic data and convert it into 3D models.

[0898] Unity: A game engine for generating 3D models based on the analyzed data.

[0899] Terminal

[0900] The terminal is a device that allows users to operate the digital twin and virtually experience the place they want to visit. Specifically, it uses the following software:

[0901] Unity application: displays the 3D model sent from the server and allows the user to manipulate it.

[0902] Data processing and calculation

[0903] Data collection

[0904] The server collects geographical data from Google Maps API and OpenStreetMap based on the place the user wants to visit. For example, if the user enters "Paris," the server retrieves data such as Paris's latitude and longitude, building shapes, and road layout.

[0905] Data analysis

[0906] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and converts the building shape into polygon data. This process generates the basic data for the digital twin.

[0907] 3D model generation

[0908] The server uses Unity to generate a 3D model based on the analyzed data. Specifically, it runs Unity C scripts to create 3D models of Paris's streets and buildings. This creates a digital twin that allows users to virtually visit Paris.

[0909] Sending data

[0910] The server then sends the generated 3D model to the user's device, where the user can launch a Unity application and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll down the Champs-Élysées.

[0911] Specific examples

[0912] If a user inputs "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographic data for Paris. Then, it runs a Python script to analyze the retrieved geographic data and convert it into a 3D model. Using Unity, it generates a 3D model of Paris's streets and buildings based on the analyzed data. Finally, it sends the generated 3D model to the user's device, where the user can launch the Unity application and virtually experience Paris.

[0913] Prompt Sentence Examples

[0914] "Create a digital twin of Paris. Use Google Maps API to get geographic data of Paris and generate a 3D model using Unity."

[0915] In this way, a system is constructed that allows users to virtually experience places they would like to visit.

[0916] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0917] Step 1:

[0918] The user enters the location they want to visit.

[0919] Specifically, a user accesses the system's web interface, enters "Paris" in the search box, and clicks the search button.

[0920] Input: A place you would like to visit (e.g. Paris)

[0921] Output: Places you want to visit (e.g. Paris)

[0922] Step 2:

[0923] The server receives user input and collects geographic data.

[0924] Specifically, the server sends a request to the Google Maps API to retrieve geographic data for Paris. For example, it sends a request to the URL https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[0925] Input: Information about the place you want to visit (e.g. Paris)

[0926] Output: Collected geographical data (e.g., Paris's latitude and longitude, building shapes, road layout)

[0927] Step 3:

[0928] The server analyzes the collected geographic data and converts it into a 3D model.

[0929] Specifically, the server runs a Python script to parse the retrieved geographic data. For example, it runs a script called parse_geodata.py to convert the latitude and longitude information into 3D coordinates.

[0930] Input: Collected geographical data (e.g., latitude and longitude information of Paris, building shapes, road layout)

[0931] Output: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[0932] Step 4:

[0933] The server generates a 3D model based on the analyzed data.

[0934] Specifically, the server runs a Unity C script to create a 3D model of the Paris cityscape and buildings. For example, it runs a script called GenerateParisModel.cs.

[0935] Input: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[0936] Output: Generated 3D model (e.g. 3D model of Paris cityscape and buildings)

[0937] Step 5:

[0938] The server sends the generated 3D model to the user's device.

[0939] Specifically, the server transfers the generated 3D model to the user's device.

[0940] Input: Generated 3D model (e.g., 3D model of Paris cityscape or buildings)

[0941] Output: The 3D model sent to the user's device

[0942] Step 6:

[0943] The user operates the digital twin on the device.

[0944] Specifically, users can launch the Unity application on their device and interact with the digital twin, for example, virtually visiting the Eiffel Tower or strolling down the Champs-Élysées.

[0945] Input: The 3D model sent to the user's device

[0946] Output: A digital twin operated by the user

[0947] (Application example 1)

[0948] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0949] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has difficulty recreating the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to get detailed information about the places you want to visit and experience their atmosphere in advance. This often leaves consumers feeling anxious because they cannot check the details of products and places before purchasing. Furthermore, the lack of ways to link with real-world transportation and accommodations can lead to a lack of consistency in travel plans.

[0950] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0951] In this invention, the server includes: a means for generating a simulated world using a digital twin; a means for experiencing the simulated world in VR; a means for generating a person in the simulated world using AI and enabling conversation with the generating AI; a means for making purchases in the simulated world via e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for acquiring data on places the user wants to visit and generating a digital twin based on the data; a means for providing a virtual shopping tour based on the digital twin; and a means for experiencing the virtual shopping tour via a smartphone or head-mounted display. This allows users to experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[0952] A "digital twin" is a virtual representation of a physical place or object in the real world as digital data.

[0953] A "pseudo-world" is a virtual environment or space created using a digital twin.

[0954] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[0955] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[0956] "Generative AI" is a technology that uses artificial intelligence to generate people and objects in virtual space.

[0957] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the Internet.

[0958] "Transportation" refers to the means of transportation or method by which a user physically travels.

[0959] "Accommodation" means a facility or place where users can stay temporarily.

[0960] "Information Fee" means the fee paid by the User for the information provided.

[0961] A "virtual shopping tour" is a tour that allows you to experience shopping in a virtual space created using a digital twin.

[0962] A "smartphone" is a type of mobile phone, a multi-function device that can connect to the Internet and use applications.

[0963] A "head-mounted display" is a display device worn on the head, and is used to experience virtual reality and augmented reality.

[0964] A system for implementing this invention generates a simulated world using a digital twin and allows users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[0965] System Configuration

[0966] The system mainly consists of the following components:

[0967] 1. Server: Generates the digital twin and provides data upon user request.

[0968] 2. Device: The smartphone or head-mounted display used by the user.

[0969] 3. User: Use the system to experience a virtual shopping tour.

[0970] Program processing

[0971] The server implements the system using the following means:

[0972] Digital twin generation method: Obtain data on the place the user wants to visit and generate a digital twin based on that data. Specifically, the location data is obtained through an API and a virtual space is generated using a 3D rendering library.

[0973] VR experience: The generated digital twin can be experienced in VR. Users can move freely within the virtual space using a smartphone or head-mounted display.

[0974] AI generation: AI generates humans in virtual space and enables conversations with users. Generative AI models are used to achieve natural conversations in real time.

[0975] EC implementation: E-commerce is used to enable purchasing within a virtual space. Users can select products in a virtual store and complete the purchase process.

[0976] Linking: Linking to real-world transportation and accommodations, allowing users to make reservations and arrangements within the virtual space.

[0977] Payment method for information provision fee: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[0978] Hardware and software used

[0979] Hardware: Smartphone, head-mounted display

[0980] Software: Python, 3D rendering library (e.g. some_3d_rendering_library), generative AI model, API

[0981] Specific examples

[0982] If a user wants to go shopping on the Champs-Élysées in Paris, the system works as follows:

[0983] 1. The user accesses the system using a smartphone or head-mounted display.

[0984] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[0985] 3. Users can experience the generated digital twin in VR and move freely within the virtual space.

[0986] 4. Talk to an AI-generated human in the virtual space and receive directions and product explanations.

[0987] 5. The user selects products in the virtual store and completes the purchase through e-commerce.

[0988] 6. Book real-world transportation and accommodations, if necessary.

[0989] Prompt Sentence Examples

[0990] Users can enter the name of a city they want to visit, for example, "Paris," which will generate a digital twin of the Champs-Élysées in Paris and allow them to start a virtual shopping tour.

[0991] In this way, users can experience the detailed information and atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[0992] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0993] Step 1:

[0994] The user accesses the system using a smartphone or head-mounted display. The user inputs the name of the place they want to visit. The inputted name of the place is sent to the server.

[0995] Step 2:

[0996] The server retrieves the corresponding location data based on the input location name. Specifically, it requests the location data through the API and parses the retrieved data. The input is the location name, and the output is the location details.

[0997] Step 3:

[0998] The server generates a digital twin based on the acquired data. It uses a 3D rendering library to recreate detailed location data as a virtual space. The input is detailed location data, and the output is a digital twin.

[0999] Step 4:

[1000] The server sends the generated digital twin to the user's device, where the user experiences the digital twin in VR via a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[1001] Step 5:

[1002] The server generates a human in the virtual space using a generative AI model. The generated AI can converse with the user. The input is the generative AI model and the virtual space data, and the output is an AI-generated human.

[1003] Step 6:

[1004] The user interacts with an AI-generated human in a virtual space, receiving directions and product explanations. The input is the user's question or request, and the output is a response from the AI-generated human.

[1005] Step 7:

[1006] The user selects a product in a virtual store and completes the purchase procedure. The server then executes the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[1007] Step 8:

[1008] Users book transportation and accommodation in the real world as needed. The server processes this reservation information and provides it to the user. The input is the user's reservation information, and the output is a reservation confirmation.

[1009] Step 9:

[1010] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is confirmation of fee collection.

[1011] In this way, users can experience detailed information and the atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[1012] Example 2

[1013] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1014] Conventional virtual reality systems have limited the simulated worlds that users experience, making it difficult to have natural conversations and interactions in real time. Furthermore, purchasing within the simulated world and links to the real world are insufficient, limiting the user experience. Furthermore, there is a lack of a way to effectively use the user's voice to have natural conversations with generative AI.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1016] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating people in the simulated world using artificial intelligence and enabling them to converse with the generative artificial intelligence, means for making purchases in the simulated world through e-commerce, means for linking to transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for converting a user's voice into text and sending it to the generative artificial intelligence as a prompt, and means for converting a response generated by the generative artificial intelligence into voice and returning it to the user. This allows users to enjoy natural conversations in real time, improving interaction within the simulated world and strengthening the link with the real world.

[1017] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1018] A "pseudo-world" is a virtual environment different from the real world that a user experiences through a virtual reality device.

[1019] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment.

[1020] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[1021] "Generative AI" is an AI model that generates natural responses based on user input.

[1022] "Electronic commerce" refers to the buying and selling of goods and services over the Internet.

[1023] A "prompt sentence" is a text-based instruction sentence sent to a generative artificial intelligence based on user input.

[1024] "Speech recognition software" is software for converting speech into text.

[1025] "Speech synthesis software" is software for converting text into speech.

[1026] The present invention provides a system for generating a simulated world that a user experiences through a virtual reality device, and for enabling natural conversation with a generative artificial intelligence. Specific embodiments of this system are described below.

[1027] First, the server generates a virtual world using the digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then provided to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[1028] When a user wears a virtual reality device, the device connects to a server and receives data about the simulated world. Based on the received data, the device renders the simulated world in real time on the device's display, allowing the user to experience the world as if they were actually walking through it.

[1029] The humans in the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the generative artificial intelligence as a prompt.

[1030] Generative AI (e.g., OpenAI's GPT-4) generates an appropriate response based on the prompt it receives. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI generates the response, "The Eiffel Tower is just down this road, turn right at the next intersection."

[1031] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech), which is then transmitted back to the user through the virtual reality device's speakers, allowing the user to hear the response from a human in the simulated world.

[1032] As a concrete example, imagine a user wearing a virtual reality device is walking through the streets of Paris. If the user is near the Eiffel Tower and asks a person in the virtual world, "Where is the Eiffel Tower?", the following prompt is sent to the generative AI:

[1033] Example prompt sentence:

[1034] User: "Where is the Eiffel Tower?"

[1035] The generative AI receives this prompt and generates an appropriate response, which the device converts into speech and returns to the user, allowing the user to enjoy natural conversation in real time.

[1036] This system improves user interaction within the virtual world and strengthens the link with the real world. For example, it enables purchases within the virtual world to be made through e-commerce, and links to real-world transportation and accommodation options. Furthermore, it includes a means to pay a portion of the usage fee as an information provision fee, further enhancing the user experience.

[1037] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1038] Step 1:

[1039] The user puts on the virtual reality device.

[1040] Input: The action of the user putting on the virtual reality device.

[1041] Output: The virtual reality device is activated and the user's vision and hearing are switched to the virtual reality device.

[1042] Specific operation: The user puts the virtual reality device on their head and turns it on, which activates the device and switches the user's vision and hearing to the information provided through the virtual reality device.

[1043] Step 2:

[1044] The device connects to the server and receives data from the virtual world.

[1045] Input: Virtual reality device activation signal.

[1046] Output: Data from the virtual world is sent from the server to the device.

[1047] How it works: The device connects to a server via the Internet, and the server sends data about the virtual world created with Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[1048] Step 3:

[1049] The device renders the simulated world and displays it to the user.

[1050] Input: Pseudo-world data received from the server.

[1051] Output: The simulated world is displayed on the display of the virtual reality device.

[1052] How it works: Based on the received data, the device renders the city of Paris in real time on the display of the virtual reality device, giving the user a visual experience that makes them feel as if they are actually walking through the city of Paris.

[1053] Step 4:

[1054] The user acts within the virtual world and talks to people within the virtual world.

[1055] Input: User's voice input.

[1056] Output: The user's voice is captured on the device.

[1057] Specific operation: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[1058] Step 5:

[1059] The device converts the user's speech into text and sends it to the generative AI model as a prompt.

[1060] Input: User's voice data.

[1061] Output: The prompt in text format.

[1062] How it works: The device captures the user's voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text as a prompt to the generative AI model.

[1063] Step 6:

[1064] The server generates a response using a generative AI model.

[1065] Input: A prompt in text format.

[1066] Output: The generated response text.

[1067] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt, such as "The Eiffel Tower is just down this road, turn right at the next intersection."

[1068] Step 7:

[1069] The terminal converts the generated response into speech and returns it to the user.

[1070] Input: The generated response text.

[1071] Output: The response in audio format.

[1072] How it works: The device converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then sent back to the user through the speaker of the virtual reality device. The user can then hear the response from the person in the virtual world.

[1073] (Application example 2)

[1074] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1075] Today's consumers want to enjoy shopping without visiting a physical store, but traditional online shopping has the drawback of not being able to actually touch and check the product. It's also difficult to obtain detailed product information through conversations with store staff. Furthermore, there are limited ways for users to experience the streets of foreign countries and enjoy the local atmosphere from the comfort of their own homes. To address these challenges, a system that offers a more realistic shopping experience and interactive communication with users is needed.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1077] In this invention, the server includes: a means for generating a virtual world using a digital twin; a means for experiencing the virtual world in VR; a means for generating a human in the virtual world using AI and enabling conversation with the generated AI; a means for enabling purchases in the virtual world through e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for tracking the user's location and initiating a conversation with an AI-generated character; a means for processing the user's input and generating a response from the AI; and a means for displaying the response. This allows users to enjoy a realistic shopping experience from the comfort of their own home, obtaining detailed information about products through interactive communication with store clerks. Users can also experience the cityscapes of foreign countries and enjoy the local atmosphere.

[1078] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1079] A "pseudo-world" is a virtual environment or space created using a digital twin.

[1080] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[1081] "AI" stands for artificial intelligence, a computer system that mimics human intelligence.

[1082] "Generative AI" is artificial intelligence that interacts with users and generates content.

[1083] "EC" is an abbreviation for electronic commerce, a system for buying and selling goods and services over the Internet.

[1084] "Real-world transportation" refers to the means of transportation used by users to travel in the real world.

[1085] "Accommodation" refers to a facility where users can stay.

[1086] "Information Fee" means the fee paid by a user for the information provided by the user.

[1087] "Tracking user location" means tracking the user's current location in real time.

[1088] An "AI-generated character" is a virtual person or character generated by artificial intelligence.

[1089] "Processing user input" means analyzing input data from the user and generating an appropriate response.

[1090] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[1091] "Displaying the response" means visually presenting the generated response to the user.

[1092] To implement the present invention, the following system configuration and program are required.

[1093] System Configuration

[1094] 1. Hardware:

[1095] VR devices (e.g. Oculus Rift, HTC Vive)

[1096] Smartphone (e.g. iPhone, Android)

[1097] Server (with high-performance computing capabilities)

[1098] 2. Software:

[1099] VR engine (e.g. Unity, Unreal Engine)

[1100] AI conversation engine (e.g. OpenAI GPT-3)

[1101] Program processing explanation

[1102] The server first uses the digital twin to generate a virtual world that recreates the streets of Paris. The user then wears a VR device and experiences this virtual world. The user's location is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[1103] User input is sent to the server via the VR device or smartphone, which then analyzes the input and generates an appropriate response using an AI conversation engine, which then displays the response in the user's field of view.

[1104] Specific examples

[1105] When a user walks into a virtual store and asks an AI-generated sales associate, "Do you have this dress in my size?", the server receives this input and uses its AI conversation engine to generate a response: "Yes, this dress is available in sizes S, M, and L." This response is displayed in the user's field of view.

[1106] Prompt Sentence Examples

[1107] User: Do you have this dress in your size?

[1108] AI: Yes, this dress is available in sizes S, M, and L.

[1109] In this way, users can enjoy a realistic shopping experience from the comfort of their own home, obtain detailed information about products through interactive communication with store staff, and even experience the local atmosphere of a foreign city.

[1110] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1111] Step 1:

[1112] The server uses a digital twin to generate a simulated world that recreates the streets of Paris.

[1113] Input: Paris streetscape data

[1114] Data processing: Using digital twin technology, the real-world streets of Paris are recreated in a virtual space.

[1115] Output: A simulated world of Parisian streets

[1116] Step 2:

[1117] The user wears a VR device and experiences a simulated world transmitted from the server.

[1118] Input: Pseudo-world data sent from the server

[1119] Data calculation: The VR device tracks the user's viewpoint and movements in real time and displays a simulated world.

[1120] Output: A simulated world displayed to the user.

[1121] Step 3:

[1122] The server tracks the user's location in real time.

[1123] Input: User's location

[1124] Data calculation: Calculates the user's position in real time and updates their position in the virtual world.

[1125] Output: Updated user location

[1126] Step 4:

[1127] When a user approaches an AI-generated character, the server initiates a conversation.

[1128] Input: User location, AI-generated character location

[1129] Data calculation: Calculates the distance between the user and the character, and starts conversation when they come within a certain distance.

[1130] Output: Trigger to start a conversation

[1131] Step 5:

[1132] Users send input to the server through their VR device or smartphone.

[1133] Input: User voice and text input

[1134] Data calculation: Sends input data to the server.

[1135] Output: User input data sent to the server

[1136] Step 6:

[1137] The server analyzes the user's input and generates an appropriate response using an AI conversation engine.

[1138] Input: User-entered data

[1139] Data Computing: Using an AI conversation engine (e.g., OpenAI GPT-3) to generate appropriate responses to user input.

[1140] Output: The generated response data

[1141] Step 7:

[1142] The server displays the generated response in the user's view.

[1143] Input: Generated response data

[1144] Data calculation: Send the response data to the VR device and display it in the user's field of view.

[1145] Output: The response displayed in the user's view

[1146] In this way, users can enjoy a realistic shopping experience from the comfort of their own home. They can also obtain detailed information about products through interactive communication with store staff. They can also experience the streets of foreign countries and enjoy the local atmosphere.

[1147] Example 3

[1148] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1149] There is a demand for a system that allows efficient purchasing and reservation procedures within a virtual world linked to the real world. In conventional systems, the process of reflecting actions in the virtual world in the real world is complicated, resulting in low user convenience. There is also a need for a system that makes the experience in the virtual world more realistic and allows users to freely choose the location and time.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in virtual reality, means for generating characters in the pseudo-world using artificial intelligence and enabling conversations with the generative artificial intelligence, means for making purchases in the pseudo-world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for arranging delivery in the real world when a user selects a product in the pseudo-world and confirms the purchase, and means for arranging reservations in the real world when a user selects transportation and accommodation in the pseudo-world. This allows the user's actions in the pseudo-world to be reflected in the real world quickly and efficiently.

[1151] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1152] A "pseudo-world" is a virtual reality environment created using a digital twin.

[1153] "Virtual reality" is a technology that allows users to experience an artificial environment generated using computer technology.

[1154] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[1155] "Generative AI" is a system that uses artificial intelligence technology to generate characters in a virtual world and enable them to converse with the user.

[1156] "Electronic commerce" is a form of transaction in which goods and services are bought and sold over the Internet.

[1157] "Transportation" is the means by which a user travels from one place to another.

[1158] "Accommodation" means a facility where users can stay temporarily.

[1159] "Information provision fee" is part of the fee paid by users when they use the system, and is paid in exchange for the provision of information.

[1160] "Shipping procedure" refers to the procedure for shipping the products purchased by the user in the real world.

[1161] "Booking Process" means the process by which a User makes a real-world reservation for their chosen mode of transportation and accommodation.

[1162] MODE FOR CARRYING OUT THE INVENTION

[1163] This invention is a system that reflects the actions of a user in a virtual world, such as purchasing products and selecting transportation and accommodations, in the real world. Specific embodiments of this system are described below.

[1164] Hardware and software used

[1165] Server: Database management system (e.g., MySQL), web server (e.g., Apache)

[1166] Device: PC or smartphone used by the user

[1167] Generative AI models: Natural language processing models (e.g., GPT-4)

[1168] Data processing and calculation

[1169] 1. The user selects a product in a virtual world

[1170] Users use their devices to visit stores in the virtual world.

[1171] When a user selects a product, the information is sent to the server.

[1172] The server stores the data of the selected products in a database.

[1173] 2. Completing the purchase process

[1174] Once the user confirms the purchase, the server processes the purchase through the e-commerce system (e.g., Shopify).

[1175] The purchase information is sent to a real-world delivery system, and the product is delivered to the user.

[1176] 3. Choosing transportation and accommodation

[1177] Users choose their means of transportation and accommodation within the virtual world.

[1178] The selection information is sent to the server, which makes the real-world reservation through a reservation system (e.g., Booking.com API).

[1179] Specific examples

[1180] Specific examples of product purchases

[1181] A user chooses a smartphone in a virtual electronics store.

[1182] When the purchase button is clicked, the server completes the purchase process through the e-commerce system and the smartphone is delivered to the user's address.

[1183] Specific examples of transportation options

[1184] The user chooses a taxi within the virtual world.

[1185] The server makes taxi reservations in the real world through a taxi reservation system.

[1186] Prompt Sentence Examples

[1187] Purchase prompts

[1188] A user wants to purchase a smartphone in a simulated world. Please explain the steps to complete the purchase and have it shipped to a real-world address.

[1189] Prompt for transportation selection

[1190] A user wants to book a taxi in a simulated world. Please explain the steps to go through the booking process and complete the taxi booking in the real world.

[1191] This system allows the user's actions in the virtual world to be reflected in the real world quickly and efficiently. The server receives the selection information from the user and performs the procedure in the real world through the appropriate system. This significantly improves user convenience. The flow of the specific processing in Example 3 will be explained using Figure 15.

[1192] Step 1:

[1193] The user visits a store in the virtual world using a terminal. The user selects a product and clicks the select button. The input is the product information selected by the user, and the output is that information being sent to the server. In concrete terms, the user selects a smartphone at an electronics store in the virtual world.

[1194] Step 2:

[1195] The server receives the product selection information sent by the user. The input is the product information sent by the user, and the output is that information is saved in a database. Specifically, the server receives the selection information from the "smartphone" and saves it in the database.

[1196] Step 3:

[1197] The user clicks the button to confirm the purchase on the purchase confirmation screen. The input is the user's purchase confirmation operation, and the output is that information being sent to the server. The specific action is when the user clicks the "Confirm Purchase" button on the purchase confirmation screen.

[1198] Step 4:

[1199] The server connects to the EC system and starts the purchase procedure. The input is the user's purchase confirmation information, and the output is the start of the purchase procedure to the EC system. Specifically, the server connects to the EC system and performs the purchase procedure on the smartphone.

[1200] Step 5:

[1201] The server sends the user's payment information to the e-commerce system to complete the purchase. The input is the user's payment information, and the output is a confirmation of the purchase completion.

[1202] Step 6:

[1203] The server sends purchase information to the delivery system. The input is purchase completion information, and the output is sending information to the delivery system. Specifically, the server sends the smartphone's delivery information to the delivery system.

[1204] Step 7:

[1205] The delivery system delivers the product to the user's address. The input is delivery information, and the output is that the product will be delivered to the user's address. The specific operation is that the delivery system delivers a smartphone to the user's address.

[1206] Step 8:

[1207] The user uses a terminal to select transportation and accommodation within the virtual world. The input is the user's selection information, and the output is that information is sent to the server. The specific operation is for the user to select a taxi within the virtual world.

[1208] Step 9:

[1209] The server receives the selection information sent by the user. The input is the selection information sent by the user, and the output is that the information is saved in the database. Specifically, the server receives the selection information for "taxi" and saves it in the database.

[1210] Step 10:

[1211] The server connects to the reservation system and starts the reservation procedure. The input is the user's selection information, and the output is the start of the reservation procedure to the reservation system. In concrete terms, the server connects to the reservation system and performs the taxi reservation procedure.

[1212] Step 11:

[1213] The server sends the reservation information to the reservation system and completes the reservation in the real world. The input is the reservation information and the output is confirmation of the reservation completion. In concrete terms, the server sends the reservation information to the reservation system and completes the taxi reservation in the real world.

[1214] (Application example 3)

[1215] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1216] There is a demand for a system that allows users to seamlessly perform purchasing and reservation actions in the virtual world in the same way as they would in the real world. In conventional systems, the experience in the virtual world is separated from the actions in the real world, and users have to perform separate manual operations to use the products and services they selected in the virtual world in the real world. This has led to issues such as a poor user experience and a loss of convenience.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a virtual world using a digital twin, means for experiencing the virtual world in VR, means for generating a person in the virtual world using AI and enabling conversation with the generative AI, means for realizing purchases in the virtual world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for selecting a product in the virtual world and completing a purchase in the real world by pressing a purchase button, and means for selecting transportation and accommodation in the virtual world and completing a reservation in the real world by pressing a reservation button. This enables purchases and reservations made by a user in the virtual world to be seamlessly reflected in the real world.

[1218] A "digital twin" is a digital representation of a real-world physical object or environment.

[1219] A "pseudo-world" is a virtual space constructed using virtual reality or augmented reality.

[1220] "VR" stands for virtual reality, a technology that allows users to immerse themselves in a virtual space.

[1221] "AI" stands for artificial intelligence, a computer system that mimics human intelligence.

[1222] "Generative AI" is artificial intelligence that generates human-like characters in virtual space and interacts with users.

[1223] "EC" stands for electronic commerce, a system for buying and selling goods and services over the Internet.

[1224] "Transportation" refers to real-world transportation, including, for example, airplanes, trains, and taxis.

[1225] "Accommodation facilities" are facilities such as hotels and inns where users can stay.

[1226] "Information provision fee" is part of the fee paid by the user when using the service, and is paid as compensation for the provision of information.

[1227] "A means of completing a purchase in the real world by selecting a product in the virtual world and pressing the purchase button" is a system in which a user selects a product in the virtual space and presses the purchase button, causing that product to be purchased in the real world.

[1228] "A means of completing a reservation in the real world by selecting a means of transportation and accommodation within the virtual world and pressing a reservation button" is a system in which a user selects a means of transportation and accommodation within the virtual space and presses a reservation button, completing the reservation in the real world.

[1229] A system for implementing this invention uses a digital twin to generate a simulated world that users can experience in virtual reality (VR). The system includes a means for realizing purchasing and reservation actions in the virtual world in the real world.

[1230] Hardware and software used

[1231] Hardware: Smartphone, Head-Mounted Display (HMD)

[1232] Software: Unity (game engine), Python (backend processing), Firebase (database), Stripe (electronic payment)

[1233] Data processing and calculation

[1234] Building a Virtual World

[1235] The server uses Unity to build a virtual world that users can move freely through, populating it with 3D models of goods, transportation, and accommodations.

[1236] Product selection and purchase

[1237] When the user selects an item, details about that item are displayed within Unity. When the user presses the purchase button, the Python backend accesses the Firebase database to record the purchase, and Stripe is used to process the electronic payment and complete the purchase.

[1238] Transportation and accommodation bookings

[1239] Similarly, once a user selects their transportation and accommodation, a Python backend accesses a Firebase database to record the reservation information, which is then reflected in a real-world reservation system.

[1240] Specific examples

[1241] Product purchase: The user selects a smartphone in a store in the virtual world, presses the purchase button, and the smartphone is delivered to them in the real world.

[1242] Booking accommodation: The user selects a hotel in the virtual world and presses the reservation button, completing the hotel reservation in the real world.

[1243] Prompt Sentence Examples

[1244] Product Purchasing: "Build a system that allows you to purchase and deliver products selected in the virtual world in the real world."

[1245] Accommodation Booking: "Build a system that allows you to book accommodation selected in the virtual world in the real world."

[1246] In this way, a system can be realized in which actions in the virtual world are seamlessly reflected in the real world.

[1247] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1248] Step 1:

[1249] The server uses Unity to build the virtual world. As input, it receives data on real-world physical objects and environments, generates a digital twin, and as output, provides a virtual world in which users can freely move.

[1250] Step 2:

[1251] The user wears a head-mounted display (HMD) and accesses the virtual world. As input, the system receives positional and motion information from the HMD, which reflects the user's viewpoint and movement within the virtual world in real time. As output, the system provides the user with an immersive virtual experience.

[1252] Step 3:

[1253] The user selects a product in the virtual world. The system receives the user's selection as input and displays detailed information about the selected product. The system provides detailed information about the product to the user as output.

[1254] Step 4:

[1255] The user presses the Buy button. As input, the Buy button press information is received and sent to the Python backend. As output, the purchase information is provided to the Python backend.

[1256] Step 5:

[1257] The server uses a Python backend to access the Firebase database and record the purchase information. As input, it receives the purchase information and stores it in the Firebase database. As output, it provides the database with the purchase information recorded.

[1258] Step 6:

[1259] The server processes electronic payments using Stripe. As input, it receives purchase information and the user's payment information, calls the Stripe API to complete the payment, and as output, provides confirmation that the payment has been completed.

[1260] Step 7:

[1261] The user selects a means of transportation or accommodation in the virtual world. The system receives the user's selection as input and displays detailed information about the selected means of transportation or accommodation. The system provides detailed information about the selected means of transportation or accommodation to the user as output.

[1262] Step 8:

[1263] The user presses the reservation button. As input, the system receives the reservation button press information and sends it to the Python backend. As output, the system provides the reservation information to the Python backend.

[1264] Step 9:

[1265] The server uses a Python backend to access the Firebase database and record the reservation information. As input, it receives the reservation information and stores it in the Firebase database. As output, it provides the database with the reservation information.

[1266] Step 10:

[1267] The server reflects the reservation information to the real-world reservation system. As input, it receives the reservation information and sends it to the real-world reservation system. As output, it provides confirmation that the reservation has been completed.

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] "Example 1"

[1270] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and adjusts the experience in the simulated world according to the emotions. Specifically, when the user feels joy, the system adjusts the experience in the simulated world to be more enjoyable. For example, the system may adjust the weather in the simulated world to be sunny, or make the AI-generated humans behave more friendly toward the user.

[1271] "Example 2"

[1272] The emotion engine also adjusts the content of the conversation with the AI-generated human depending on the user's emotions. For example, if the user is sad, the AI-generated human will offer words of comfort. If the user is angry, the AI-generated human will offer words of apology. This allows users to experience appropriate responses in the virtual world according to their emotions.

[1273] "Example 3"

[1274] Furthermore, the emotion engine recognizes the user's emotions and supports purchasing behavior within the virtual world according to those emotions. For example, when a user feels happy, the system will recommend products that will increase that joy. When a user feels sad, the system will recommend products that will ease that sadness. This allows users to purchase appropriate products within the virtual world according to their emotions.

[1275] The processing flow of each embodiment will be described below.

[1276] "Example 1"

[1277] Step 1: The user logs into the virtual world through a VR device.

[1278] Step 2: The emotion engine recognizes the user's emotions, based on their facial expressions, tone of voice, and choice of words.

[1279] Step 3: The system adjusts the simulated world experience based on the emotions it recognizes. For example, if the user feels happy, the system might make the weather in the simulated world sunny or make the AI-generated human behave more friendly toward the user.

[1280] "Example 2"

[1281] Step 1: The user initiates a conversation with an AI-generated human.

[1282] Step 2: The emotion engine recognizes the user's emotions and adjusts the conversation with the AI-generated human accordingly. For example, if the user is sad, the AI-generated human will offer words of comfort; if the user is angry, the AI-generated human will offer words of apology.

[1283] "Example 3"

[1284] Step 1: The user begins shopping in the virtual world.

[1285] Step 2: The emotion engine recognizes the user's emotions and adjusts the product recommendations accordingly. For example, if the user is feeling happy, the system will recommend products that will increase that joy. If the user is sad, the system will recommend products that will alleviate that sadness.

[1286] Example 1

[1287] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1288] Conventional virtual reality systems have the problem of being difficult to generate detailed digital twins of places users want to visit, and they do not adjust the experience according to the user's emotions, resulting in low user satisfaction. Furthermore, purchasing within the virtual world and linking it to the real world are insufficient, resulting in a lack of user convenience.

[1289] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1290] In this invention, the server includes: means for generating a virtual world using a digital twin; means for experiencing the virtual world in virtual reality; means for generating characters in the virtual world using artificial intelligence and enabling conversations with the generated artificial intelligence; means for enabling purchases in the virtual world through e-commerce; means for linking to real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means for recognizing a user's emotions and adjusting the virtual world experience in response to those emotions; and an emotion recognition engine for analyzing the user's emotions. This allows for the generation of a detailed digital twin of a place the user wants to visit and for the adjustment of the experience in response to the user's emotions. Furthermore, the link between the virtual world and the real world and the purchases in the virtual world is strengthened, improving user convenience.

[1291] A "digital twin" is a digital model that recreates a real-world physical place or object in a virtual space.

[1292] A "pseudo-world" is a virtual reality environment created using a digital twin.

[1293] "Virtual reality" is a technology that allows users to experience an artificial environment generated using computer technology.

[1294] "Artificial intelligence" is a technology that allows computers to act in a way that mimics human intelligence.

[1295] "Electronic commerce" refers to the buying and selling of goods and services over the Internet.

[1296] An "emotion recognition engine" is software that analyzes emotions from a user's facial expressions, voice, etc.

[1297] "Emotional experience tailoring" refers to modifying the virtual reality environment and interactions based on the user's emotional state.

[1298] "Information Fee" means the fee paid by a user for the information provided by the user.

[1299] "Transportation" refers to the means of transportation that a user uses to move around in the real world.

[1300] "Accommodation" refers to a facility where users can stay.

[1301] MODE FOR CARRYING OUT THE INVENTION

[1302] This invention is a system that uses digital twins to generate a simulated world that users can experience in virtual reality. It also has the ability to recognize the user's emotions and adjust the experience of the simulated world according to those emotions.

[1303] Digital Twin Generation

[1304] The server collects data on the location the user wants to visit. This data includes streetscapes, buildings, and scenery. Specifically, it uses the Google Maps API or OpenStreetMap API to obtain a 3D model of the specified location and surrounding landscape data. The server then uses this data to generate a digital twin using Unity or Unreal Engine.

[1305] Example: If a user requests to "visit the Eiffel Tower in Paris," the server uses the Google Maps API to collect a 3D model of the Eiffel Tower and surrounding landscape data, and then uses Unity to generate a digital twin of the Eiffel Tower.

[1306] Example prompt sentence:

[1307] A user wants to visit the Eiffel Tower in Paris. Collect a 3D model of the Eiffel Tower and surrounding landscape data, and generate a simulated world using Unity.

[1308] Virtual reality experience

[1309] The device then presents the generated digital twin to the user, who can then experience the virtual world through a VR headset or monitor. The device tracks the user's movements and gaze, updating the virtual world in real time.

[1310] AI-powered conversation

[1311] The server uses artificial intelligence to generate characters in the virtual world and enables them to converse with users. For example, if a user asks for directions within the virtual world, the AI ​​will provide an appropriate response.

[1312] Enabling e-commerce

[1313] The server realizes purchasing actions within the virtual world through e-commerce. Users can select products in a virtual store and complete the purchase process. Purchase data is securely managed by the server.

[1314] Emotion recognition and experience regulation

[1315] The device uses a camera and microphone to capture the user's facial expressions and voice. The server then analyzes the user's emotions from the captured data using Microsoft Azure's Emotion API or Google Cloud's Vision API. Based on the analyzed emotions, the server adjusts the simulated world experience. For example, if the user is feeling happy, the server will set the weather in the simulated world to sunny and have an AI-generated character greet the user with a smile.

[1316] Example: If the server detects that the user is having fun in the simulated world, it will set the weather to sunny and adjust the AI-generated characters to greet the user with a smile.

[1317] Example prompt sentence:

[1318] The user is recognized as having fun in the simulated world. Please set the weather in the simulated world to sunny and adjust the AI-generated characters to greet the user with a smile.

[1319] The system allows users to experience a detailed digital twin of the place they want to visit, providing a personalized experience tailored to their emotions, and enhancing user convenience through enhanced purchasing behavior within the virtual world and links to the real world.

[1320] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1321] Step 1:

[1322] The user inputs the location they want to visit into the terminal, for example, "Eiffel Tower in Paris." The input data is sent to the server.

[1323] Input: Places the user wants to visit

[1324] Output: Places to visit data sent to the server

[1325] Step 2:

[1326] The server collects data for the specified location via the Internet, specifically using the Google Maps API and OpenStreetMap API to obtain a 3D model of the Eiffel Tower and surrounding landscape data.

[1327] Input: Data about places the user wants to visit

[1328] Output: 3D models and landscape data of collected locations

[1329] Step 3:

[1330] The server generates a digital twin using Unity or Unreal Engine based on the collected data. Specifically, it imports a 3D model of the Eiffel Tower into Unity and places the surrounding landscape data.

[1331] Input: 3D model and landscape data of collected locations

[1332] Output: Digital twin generated in Unity or Unreal Engine

[1333] Step 4:

[1334] The server then sends the generated digital twin to the device, allowing the user to experience the virtual world. The device then displays the virtual world to the user through a VR headset or monitor.

[1335] Input: Digital twin generated in Unity or Unreal Engine

[1336] Output: Digital twin sent to the device

[1337] Step 5:

[1338] The device uses a camera and microphone to capture the user's facial expressions and voice. Specifically, the camera takes a picture of the user's face and the microphone records the user's voice.

[1339] Input: User's facial expressions and voice

[1340] Output: Captured facial and voice data of the user

[1341] Step 6:

[1342] The server uses Microsoft Azure's Emotion API or Google Cloud's Vision API to analyze the user's emotions from the captured data. For example, if the user is smiling, it will recognize the emotion as "joy."

[1343] Input: Captured facial and voice data of the user

[1344] Output: Parsed user sentiment data

[1345] Step 7:

[1346] The server adjusts the simulated world experience based on the analyzed emotions: for example, if the user is feeling happy, it will set the weather in the simulated world to sunny and have the AI-generated characters greet the user with a smile.

[1347] Input: Parsed user sentiment data

[1348] Output: Adjusted pseudo-world configuration

[1349] Step 8:

[1350] The server then sends the adjusted virtual world to the device, allowing the user to experience the changes. The device then displays the adjusted virtual world to the user through a VR headset or monitor.

[1351] Input: Adjusted pseudo-world settings

[1352] Output: The adjusted virtual world sent to the device.

[1353] (Application example 1)

[1354] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1355] Conventional virtual store systems do not adjust the experience according to the user's emotions, making it difficult to increase user satisfaction. Furthermore, to provide an experience similar to that of a real-world store, it is necessary to dynamically change the environment and the responses of store staff according to the user's emotions, but such a system has not existed.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in VR, means for generating people in the pseudo-world using AI and enabling conversations with the generative AI, means for realizing purchases in the pseudo-world through e-commerce, means for linking with real-world transportation and accommodations, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience in the pseudo-world in accordance with the emotions, and means for changing the environment in the virtual store and the behavior of the store clerk based on the adjustments. This makes it possible to provide a dynamic experience that corresponds to the user's emotions.

[1357] A "digital twin" is a virtual model that recreates a real-world physical place or object in a digital space.

[1358] A "pseudo-world" is a virtual space created using digital twin technology, and is a virtual environment that users can experience.

[1359] "VR" is an abbreviation for virtual reality, a technology that allows users to visually experience a virtual space.

[1360] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to act in a way that mimics human intelligence.

[1361] "Generative AI" is a system that uses artificial intelligence to generate people and objects in virtual space.

[1362] "EC" stands for electronic commerce, the act of purchasing goods and services over the Internet.

[1363] "Emotion recognition" is a technology that analyzes a user's facial expressions and behavior to identify their emotions.

[1364] A "virtual store" is a store that exists in a virtual space, a virtual commercial facility where users can browse and purchase products.

[1365] "Environmental adjustment" refers to dynamically changing environmental elements such as weather, music, and lighting in a virtual space according to the user's emotions.

[1366] "Store staff response" refers to the service and assistance provided to users by AI-generated humans in the virtual space.

[1367] The embodiments of the present invention will be described in detail below.

[1368] System Overview

[1369] The system of this invention uses digital twin technology to generate a simulated world that users can experience in VR. It also uses emotion recognition technology to analyze the user's emotions and dynamically adjust the experience in the simulated world based on those emotions.

[1370] Hardware and software used

[1371] Hardware:

[1372] Smartphone or head-mounted display (HMD)

[1373] Camera (smartphone camera or HMD camera)

[1374] software:

[1375] OpenCV (camera image acquisition and processing)

[1376] EmotionRecognizer (emotion recognition library)

[1377] VirtualWorld (Digital Twin Generation and Rendering Library)

[1378] System Operation

[1379] Digital Twin Generation

[1380] The server acquires data on the location the user wants to visit (such as cityscapes, buildings, and scenery) and generates a digital twin. This data can be obtained from the Internet. The generated digital twin is sent to the user's device and displayed in a VR environment.

[1381] emotion recognition

[1382] The user's device captures the user's facial expressions in real time using a camera and analyzes their emotions using EmotionRecognizer. The analyzed emotional data is then sent to the server.

[1383] Tailoring the experience

[1384] The server dynamically adjusts the environment (weather, music, lighting, etc.) in the virtual world and the behavior of store staff based on the received emotional data. For example, if the user is happy, the server will adjust the music in the store to be brighter and the staff to be more friendly.

[1385] Specific examples

[1386] Users access the virtual store using their smartphones and walk around the store, which recreates the streets of Paris. When the user smiles, the music in the store brightens and a sales associate greets them in a friendly manner, asking, "Hello! What are you looking for today?"

[1387] Prompt Sentence Examples

[1388] Capture data on the places users want to visit and generate a digital twin. Recognize users' emotions and tailor the virtual in-store experience accordingly. For example, if a user is happy, make the music brighter and the staff more friendly.

[1389] In this way, the system of the present invention can provide a dynamic experience that responds to the user's emotions.

[1390] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1391] Step 1:

[1392] The user selects a place they want to visit. Using a smartphone or a head-mounted display (HMD), the user selects the place they want to visit (e.g., a street view of Paris). The input is the information about the user's selected place, and the output is a request to obtain data about that place.

[1393] Step 2:

[1394] The server retrieves data on the selected location. The server retrieves data on the streetscape, buildings, scenery, etc. of the selected location via the Internet. The input is the information on the location selected by the user, and the output is the retrieved location data.

[1395] Step 3:

[1396] The server generates a digital twin. The server generates a digital twin based on the acquired location data. The input is the acquired location data, and the output is the generated digital twin.

[1397] Step 4:

[1398] The terminal displays the digital twin. The terminal receives the digital twin sent from the server and displays it in a VR environment. The input is the generated digital twin, and the output is the VR environment experienced by the user.

[1399] Step 5:

[1400] The device recognizes the user's emotions. The device uses a camera to capture the user's facial expressions in real time and analyzes the emotions using EmotionRecognizer. The input is the camera image, and the output is the analyzed emotional data.

[1401] Step 6:

[1402] The device sends emotional data to the server. The device sends analyzed emotional data to the server. The input is the analyzed emotional data, and the output is the emotional data sent to the server.

[1403] Step 7:

[1404] The server adjusts the experience based on the emotional data. Based on the received emotional data, the server dynamically adjusts the environment (weather, music, lighting, etc.) in the virtual world and the responses of the store staff. The input is emotional data, and the output is the adjusted environmental settings.

[1405] Step 8:

[1406] The device displays the calibrated environment. The device receives the calibrated environment settings sent from the server and displays them in the VR environment. The input is the calibrated environment settings, and the output is the calibrated VR environment experienced by the user.

[1407] Example 2

[1408] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1409] In conventional virtual reality systems, the virtual world experienced by the user is fixed, making it difficult to provide interactive responses based on the user's emotions. Furthermore, conversations within the virtual world are monotonous, making it impossible to respond appropriately to the user's emotions, making it impossible to provide a realistic experience. Furthermore, the link with the real world is insufficient, resulting in low user convenience.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1411] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating humans in the simulated world using artificial intelligence and enabling conversations with the generative AI, means for enabling purchases in the simulated world through e-commerce, means for linking with transportation and accommodations in the real world, and means for analyzing the user's emotions and adjusting the content of the conversation based on the analysis results, allowing the user to experience a realistic simulated world and receive appropriate responses according to their emotions.

[1412] A "digital twin" is a virtual model that recreates real-world physical objects and environments in a digital space.

[1413] A "pseudo-world" is a virtual environment that users can experience, generated using virtual reality technology.

[1414] "Virtual reality" is a technology that immerses users in a virtual environment generated using computer technology.

[1415] "Artificial intelligence" is a technology that allows computers to imitate human intelligence and have functions such as learning, reasoning, and recognition.

[1416] "Generative AI" is AI that has the ability to generate new data or content based on specific input.

[1417] "Electronic commerce" is a form of transaction in which goods and services are bought and sold over the Internet.

[1418] "Emotion analysis" is a technology that identifies and analyzes emotions from a user's tone of voice, facial expressions, etc.

[1419] "Conversation content adjustment" is the process of appropriately changing the content of the conversation provided by generative artificial intelligence based on the results of user emotion analysis.

[1420] "Transportation" refers to the means of transportation that a user uses to move around in the real world.

[1421] "Accommodation" means a facility where users can stay temporarily.

[1422] This invention provides a system in which a user experiences a simulated world using a virtual reality (VR) device and has an interactive conversation with a human generated by artificial intelligence (AI). Specific embodiments of this system are described below.

[1423] Hardware and Software Use

[1424] Hardware: VR devices such as Oculus Rift and HTC Vive are used, allowing users to experience a highly immersive experience.

[1425] Software: Game engines such as Unity and Unreal Engine are used to generate the VR environment, OpenAI's GPT-3 is used as the generative AI model, and emotion recognition software is used for emotion analysis.

[1426] Data processing and calculation

[1427] Using a VR device: When a user wears a VR device, the device generates the Paris cityscape using Unity or Unreal Engine, providing a 360-degree view, allowing the user to experience the cityscape in a realistic way.

[1428] Use of generative AI models: The server uses generative AI models such as OpenAI's GPT-3 to generate humans in the simulated world, allowing users to ask for directions and engage in everyday conversations.

[1429] Use of emotion engine: The server uses emotion recognition software to analyze the user's emotions in real time. Based on the analysis results, the generative AI model adjusts the conversation content. For example, if the user is sad, it will offer words of comfort, and if the user is angry, it will offer words of apology.

[1430] Specific examples

[1431] The user puts on the VR device: The user puts on the Oculus Rift on their head and holds the controllers in their hands.

[1432] The device launches the VR environment: The device launches Unity and displays a 360-degree view of the Paris cityscape.

[1433] The server generates a simulated world: The server uses GPT-3 to generate cafe staff and people passing by.

[1434] The user moves through the virtual world: The user operates the controller and walks down the Champs-Élysées.

[1435] A user converses with an AI-generated human: A user asks a cafe waiter, "What do you recommend on the menu?"

[1436] The server analyzes the user's emotions: The server analyzes the user's tone of voice and determines that the user is a little tired.

[1437] The server adjusts the conversation content: The server sends a prompt to the generative AI model, such as "You seem particularly tired today. I'd like to introduce you to a cafe where you can relax," and generates an appropriate conversation.

[1438] Prompt Sentence Examples

[1439] "When a user gets lost, generate a conversation where an AI-generated human can provide appropriate directions."

[1440] "When a user asks a cafe staff member for a recommended menu item, please generate a conversation in which the staff member gives an appropriate answer."

[1441] This system allows users to experience the real city of Paris from the comfort of their own home and receive appropriate responses based on their emotions.

[1442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1443] Step 1:

[1444] The user puts on the VR device.

[1445] Input: User puts on a VR device (e.g., Oculus Rift).

[1446] Output: The user is ready to enter the VR environment.

[1447] Specific actions: The user wears a VR device on their head and holds controllers in their hands.

[1448] Step 2:

[1449] The device will launch the VR environment.

[1450] Input: User wearing the VR device.

[1451] Output: The VR environment is displayed.

[1452] What it does: The device launches Unity or Unreal Engine, displaying a 360-degree view of the Paris cityscape.

[1453] Step 3:

[1454] The server generates a pseudo-world.

[1455] Input: VR environment activation signal from device.

[1456] Output: A human in a simulated world is generated.

[1457] How it works: The server uses OpenAI's GPT-3 to generate cafe staff and passersby.

[1458] Step 4:

[1459] The user moves through the virtual world.

[1460] Input: User controller actions.

[1461] Output: The user's viewpoint moves.

[1462] Specific action: The user operates the controller and walks down the Champs-Élysées.

[1463] Step 5:

[1464] The user converses with an AI-generated human.

[1465] Input: User's voice input.

[1466] Output: AI-generated human response.

[1467] Specific behavior: When a user asks a cafe staff member, "What's your recommendation on the menu?", the AI-generated human responds, "Today's recommendation is croissants and cafe au lait."

[1468] Step 6:

[1469] The server analyzes the user's emotions.

[1470] Input: User's voice and facial expression data.

[1471] Output: The user's emotional state.

[1472] What it does: The server uses emotion recognition software to analyze the user's tone of voice and facial expressions and determines that the user is a little tired.

[1473] Step 7:

[1474] The server moderates the conversation.

[1475] Input: The user's emotional state.

[1476] Output: The adjusted conversation.

[1477] Specific operation: The server sends a prompt to the generative AI model, such as "You seem particularly tired today. I'd like to introduce you to a cafe where you can relax," and generates an appropriate conversation.

[1478] (Application example 2)

[1479] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1480] While modern consumers enjoy the convenience of online shopping, they also desire the experience of face-to-face customer service and seeing products in person at a physical store. However, traditional online shopping has the drawback of not providing users with sufficient information when selecting products, and not being able to receive appropriate support based on their emotions. Furthermore, it has been difficult to provide a shopping experience customized to the user's emotions.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1482] In this invention, the server includes: means for generating a virtual world using a digital twin; means for experiencing the virtual world in VR; means for generating humans in the virtual world using AI and enabling conversations with the generative AI; means for enabling purchases in the virtual world through e-commerce; means for linking with real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine for analyzing a user's emotions and generating conversation content according to the emotions; and means for selecting products while conversing with an AI-generated sales clerk in a virtual store. This allows users to enjoy a realistic shopping experience in the virtual store and receive appropriate responses according to their emotions.

[1483] A "digital twin" is a virtual model that recreates real-world physical objects and environments in a digital space.

[1484] A "pseudo-world" is a virtual space created using a digital twin, which users can experience through VR devices.

[1485] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space through their eyes and ears.

[1486] An "AI-generated human" is a human character generated in a virtual space using artificial intelligence technology.

[1487] "Generative AI" is an artificial intelligence system that generates conversations and interactions with users.

[1488] "EC" is an abbreviation for electronic commerce, a system for buying and selling goods and services over the Internet.

[1489] The "emotion engine" is a system that analyzes the user's emotions and generates appropriate responses and conversation content based on those emotions.

[1490] A "virtual store" is a store built in a virtual space that users can visit through a VR device.

[1491] An "AI-generated store clerk" is a store clerk character generated using artificial intelligence technology to interact with users in a virtual store.

[1492] A system for implementing this invention is configured as follows: First, a user puts on a VR device and accesses a virtual store. The VR device can be a commercially available VR headset such as Oculus Rift. When the user enters the virtual store, a simulated world created using digital twin technology is displayed.

[1493] The server uses the digital twin to generate a virtual world that users can experience through a VR device. AI-generated humans (store clerks) are placed within the virtual world and can converse with the user. The generative AI uses OpenAI's API to generate conversations with the user.

[1494] When a user selects a product in a virtual store, the emotion engine analyzes the user's emotions and responds appropriately based on those emotions. The emotion engine analyzes the user's voice input and facial expression data, generating words of comfort if the user is sad and words of apology if the user is angry.

[1495] As a concrete example, consider a scenario where a user is looking for shoes in a virtual store. When the user asks, "Do you have this shoe in my size?", the emotion engine analyzes the user's emotions and generates an appropriate prompt. For example, if the user is sad, the engine generates a prompt such as, "When the user is sad, offer comforting words."

[1496] Example prompt sentence:

[1497] "When a user asks for their shoe size, we suggest the appropriate size."

[1498] The server sends this prompt to OpenAI's API, which generates a dialogue in which an AI-generated store assistant suggests an appropriate size, which is then displayed to the user through the VR device.

[1499] This system allows users to enjoy a realistic shopping experience in a virtual store and receive appropriate responses based on their emotions, providing a new shopping experience that combines the convenience of online shopping with the face-to-face customer service of a physical store.

[1500] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1501] Step 1:

[1502] The user puts on the VR device and accesses the virtual store.

[1503] Input: User puts on VR device and requests access to virtual store

[1504] Output: Connection established to virtual store

[1505] Specific operation: The user puts on the VR device and launches the virtual store application. The server receives the user's access request and establishes a connection to the virtual store.

[1506] Step 2:

[1507] The server uses the digital twin to generate a simulated world and displays it to the user.

[1508] Input: Digital twin data of virtual store

[1509] Output: A virtual world displayed on the user's VR device

[1510] How it works: The server uses digital twin technology to generate a simulated world of a virtual store and transmits it to the user's VR device, through which the user experiences the simulated world.

[1511] Step 3:

[1512] When a user selects a product in a virtual store, the emotion engine analyzes the user's emotions.

[1513] Input: User voice input and facial expression data

[1514] Output: User sentiment analysis results

[1515] How it works: When a user asks a question or makes a comment about a product, the VR device sends voice input and facial expression data to the server, which then uses an emotion engine to analyze this data and identify the user's emotions.

[1516] Step 4:

[1517] The server generates prompt sentences according to the emotion and sends them to the generative AI model.

[1518] Input: User sentiment analysis results

[1519] Output: Generated prompt statement

[1520] Specific operation: The server generates an appropriate prompt based on the analysis results of the emotion engine. For example, if the user is sad, the server generates a prompt such as "When the user is sad, say some comforting words."

[1521] Step 5:

[1522] The generative AI model generates conversation content based on the prompt sentence and sends it to the server.

[1523] Input: Generated prompt text

[1524] Output: Generated conversation

[1525] Specific operation: The server sends the generated prompt sentence to the generative AI model, which then generates appropriate conversation content based on the prompt sentence and returns the result to the server.

[1526] Step 6:

[1527] The server sends the generated conversation content to the user's VR device and displays it.

[1528] Input: Generated conversation

[1529] Output: The conversation displayed on the user's VR device

[1530] Specific operation: The server receives the conversation content from the generative AI model and sends it to the user's VR device, where the user experiences a conversation with the AI-generated store clerk.

[1531] Step 7:

[1532] When a user purchases an item in a virtual store, the server processes the purchase through the EC system.

[1533] Input: User purchase request

[1534] Output: Purchase completion notification

[1535] Specific operation: When the user selects a product and decides to purchase it, the server processes the purchase through the EC system. Once the purchase is complete, the server sends a purchase completion notification to the user.

[1536] Example 3

[1537] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1538] There is a demand for a system that can efficiently realize purchasing and reservation activities in a virtual world linked to the real world. Another challenge is to improve the user experience by recommending products based on the user's emotions. Conventional systems do not take user emotions into account when recommending products, making it difficult to increase user satisfaction.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1540] In this invention, the server includes: means for generating a simulated world using a digital twin; means for experiencing the simulated world in virtual reality; means for generating characters in the simulated world using artificial intelligence and enabling conversations with the generative artificial intelligence; means for making purchases in the simulated world through e-commerce; means for linking with transportation and accommodation in the real world; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine that recognizes a user's emotions and recommends products based on those emotions; means for the emotion engine to transmit user emotion data to the server and for the server to select products based on those emotions; and means for transmitting the selected product information to the user's terminal and for the terminal to display the recommended products. This enables product recommendations based on the user's emotions, improving the user experience.

[1541] "Digital twin" is a technology that recreates real-world physical objects and environments in a virtual space.

[1542] A "pseudo-world" is a virtual space created using digital twin technology, an environment that users can experience through virtual reality.

[1543] "Virtual reality" is a technology that allows users to experience a virtual space generated using computer technology through their eyes and ears.

[1544] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1545] "Generative AI" is a system that uses artificial intelligence technology to generate people and objects in a virtual space, enabling dialogue and interaction with users.

[1546] "Electronic commerce" is a form of transaction in which goods and services are bought and sold via the Internet.

[1547] "Transportation" means the means of transportation used by a User to travel from one location to another.

[1548] "Accommodation facilities" are facilities where users can stay temporarily, including hotels and inns.

[1549] "Information provision fee" is part of the fee paid by users when they use the system, and is compensation for the information and services provided.

[1550] An "emotion engine" is a system that recognizes a user's emotions and recommends appropriate products and services based on those emotions.

[1551] "Emotion data" is information about emotions acquired from the user's facial expressions, voice, etc.

[1552] A "server" is a computer system that stores, processes, and manages data on a network.

[1553] A "terminal" is a device that a user uses to access and operate the system, including a PC or smartphone.

[1554] "Recommended products" are products selected and suggested by the system based on the user's emotions and preferences.

[1555] This invention is a system in which users can select and purchase products in a virtual world, and the products are delivered in the real world. It is also linked to real-world transportation and accommodation options, so that selections made in the virtual world are reflected in real-world reservations. Furthermore, it uses an emotion engine to recognize the user's emotions and recommend products based on those emotions.

[1556] Hardware and software used

[1557] Servers: Database management systems (e.g., MySQL), web servers (e.g., Apache), emotion engines (e.g., IBM Watson)

[1558] Device: User interface (e.g. web browser, mobile app)

[1559] Generative AI models: Natural language processing models (e.g., GPT-4)

[1560] Specific operation of the system

[1561] 1. The user selects a product in a virtual world

[1562] A user accesses a store in the virtual world using a terminal. When the user selects a product and clicks the purchase button, the terminal sends information about the selected product to the server.

[1563] 2. Sending and saving product information

[1564] The device sends the selected product information in JSON format to the server. The server saves the received product information in a database (e.g., MySQL) and returns a confirmation message to the device that the information has been saved.

[1565] 3. Execution of delivery arrangements

[1566] The server coordinates with the delivery system (e.g., FedEx API) to arrange delivery. The server receives delivery confirmation information from the delivery system and notifies the user's device of the delivery confirmation information.

[1567] 4. Choosing transportation and accommodation

[1568] When the user selects transportation and accommodation within the virtual world and clicks the reservation button, the device sends the selected information to the server.

[1569] 5. Sending and storing reservation information

[1570] The device sends the selected reservation information in JSON format to the server. The server saves the received reservation information in a database and makes the reservation by connecting to a reservation system (e.g., Booking.com API). The server then notifies the user's device of the reservation confirmation information.

[1571] 6. Emotion Recognition by Emotion Engine

[1572] When a user uses the device to explore the virtual world, the device uses a camera and microphone to capture the user's facial expressions and voice. The device sends the captured data to an emotion engine (e.g., IBM Watson), which analyzes the user's emotions in real time and sends the data to a server.

[1573] 7. Emotion-based product recommendations

[1574] The server selects products that correspond to the user's emotions based on the emotion data, and sends the selected product information to the device, which then displays the recommended products to the user.

[1575] Examples and prompts

[1576] Example 1: Product Purchase

[1577] The user selects a smartphone in a virtual electronics store and clicks the purchase button.

[1578] The server receives the information from the smartphone and works with the delivery system to arrange delivery in the real world.

[1579] Users will receive their smartphone in the real world a few days later.

[1580] Example 2: Booking accommodation

[1581] The user selects a hotel in the virtual world and clicks the reservation button.

[1582] The server receives the hotel information and works with the reservation system to make reservations in the real world.

[1583] The user receives a hotel booking confirmation in the real world.

[1584] Prompt Sentence Examples

[1585] "Please explain the steps to deliver the product selected by the user in the virtual world to the real world."

[1586] "Please explain how product recommendations based on user emotions work."

[1587] The above is an embodiment of the present invention. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1588] Step 1:

[1589] The user selects a product in the virtual world

[1590] Input: The user uses a device to access a store in the virtual world and select a product.

[1591] Specific behavior: The user selects a product and clicks the purchase button.

[1592] Output: The device sends the selected product information to the server in JSON format.

[1593] Step 2:

[1594] Sending and storing product information

[1595] Input: Product information in JSON format sent from the device.

[1596] Specific operation: The server analyzes the received product information and saves the "user ID, product ID, purchase date and time" in a database (e.g. MySQL).

[1597] Output: The server returns a confirmation message to the terminal that the save was successful.

[1598] Step 3:

[1599] Executing delivery arrangements

[1600] Input: Product information stored in the database.

[1601] Specific operation: The server connects to the delivery system (e.g., FedEx API) and sends the user's address, product ID, and delivery date and time. The delivery system returns the delivery ID and expected delivery date and time to the server.

[1602] Output: The server notifies the user's device of the "Delivery ID, scheduled delivery date and time."

[1603] Step 4:

[1604] Choosing transportation and accommodation

[1605] Input: The user uses the device to select transportation and accommodation within the simulated world.

[1606] Specific operation: The user clicks the reservation button, and the device sends the selected information in JSON format to the server.

[1607] Output: The terminal sends the selected reservation information to the server.

[1608] Step 5:

[1609] Sending and saving reservation information

[1610] Input: Reservation information sent from the terminal in JSON format.

[1611] Specific operation: The server analyzes the received reservation information and saves the "user ID, reservation ID, reservation date and time" in the database. The server connects to the reservation system (e.g. Booking.com API) and sends the "user name, reservation ID, departure date and time". The reservation system returns the "reservation confirmation number, departure date and time" to the server.

[1612] Output: The server notifies the user's terminal of the reservation confirmation number and departure date and time.

[1613] Step 6:

[1614] Emotion recognition by emotion engine

[1615] Input: Facial and voice data as the user uses the device to explore the simulated world.

[1616] How it works: The device uses a camera and microphone to capture the user's facial expressions and voice, and sends them to an emotion engine (e.g., IBM Watson). The emotion engine analyzes the user's emotions in real time and sends the data to a server.

[1617] Output: The emotion engine sends the analysis results to the server.

[1618] Step 7:

[1619] Emotion-based product recommendations

[1620] Input: Emotion data sent from the emotion engine.

[1621] Specific operation: The server selects products that correspond to the user's emotions based on the emotion data, and then sends the selected product information to the terminal.

[1622] Output: The device displays the recommended products to the user.

[1623] (Application example 3)

[1624] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1625] There is a need to optimize purchasing and reservation activities in virtual worlds linked to the real world based on user emotions. Conventional systems provide products and services without considering user emotions, resulting in a limited user experience and low satisfaction. Furthermore, there is no adequate system in place to reflect choices made in virtual worlds in deliveries and reservations in the real world.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in VR, means for generating a person in the pseudo-world using AI and enabling conversation with the generative AI, means for realizing purchases in the pseudo-world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions using an emotion engine and recommending products corresponding to the emotions, means for delivering products selected in the pseudo-world in the real world, and means for reserving transportation and accommodation selected in the pseudo-world in the real world. This makes it possible to provide optimal products and services according to the user's emotions, and realizes a system in which selections in the pseudo-world are reflected in deliveries and reservations in the real world.

[1627] A "digital twin" is a digital representation of a real-world physical object or environment.

[1628] A "pseudo-world" is a virtual environment created using a digital twin, which users can experience using virtual reality (VR) technology.

[1629] "VR" is an abbreviation for Virtual Reality, a technology that allows users to experience a three-dimensional space generated by a computer.

[1630] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to learn and reason by imitating human intelligence.

[1631] "Generative AI" is a technology that uses artificial intelligence to generate people and objects in a simulated world, enabling them to interact with users.

[1632] "EC" is an abbreviation for Electronic Commerce, a system for buying and selling goods and services over the Internet.

[1633] An "emotion engine" is a technology that recognizes a user's emotions and responds or makes recommendations based on those emotions.

[1634] "Means of transportation" refers to the means of transportation that users use to move around in the real world, including, for example, cars, trains, and airplanes.

[1635] "Accommodation" refers to facilities where users can stay in the real world, including hotels and inns.

[1636] "Information Fee" means the fee paid for the information provided by the User, and is treated as part of the service fee.

[1637] "Delivery" refers to the act of delivering the purchased product to the location specified by the user.

[1638] "Reservation" means the act of a user reserving a specific service or facility in advance.

[1639] A system for implementing this invention generates a simulated world using a digital twin and allows users to experience that simulated world using virtual reality (VR) technology. The system includes the following main components:

[1640] 1. Digital twin generation method:

[1641] The server uses 3D modeling software and sensor technology to create digital twins of real-world physical objects and environments in a digital space.

[1642] 2. VR experience method:

[1643] Users experience the generated virtual world using a head-mounted display (HMD) or smartphone, with specific hardware such as Oculus Rift or HTC Vive.

[1644] 3. AI generation means:

[1645] The server uses artificial intelligence (AI) to generate people and objects in the simulated world and enable user interaction using generative AI models.

[1646] 4. EC Implementation Methods:

[1647] The server provides an electronic commerce (EC) system that allows users to select and purchase products in the virtual world. Purchased products are delivered in the real world.

[1648] 5. Emotion Engine:

[1649] The server recognizes the user's emotions in real time and recommends products based on those emotions. Emotion recognition is performed using OpenCV and TensorFlow with the help of cameras and microphones.

[1650] 6. Links to the real world:

[1651] The travel and accommodation choices users make in the virtual world are reflected in their real-world reservations, including integration with booking systems via APIs.

[1652] 7. Information Fee Method:

[1653] The server includes a mechanism for paying a portion of the usage fee for the information provided by the user as an information provision fee.

[1654] As a concrete example, consider a scenario where a user is searching for a new smartphone in a virtual store: if the user is feeling happy, the system will recommend the latest smartphone model.

[1655] Example prompts to input to a generative AI model:

[1656] A user is searching for a new smartphone in a virtual store. Recommend the latest smartphone model because they are excited about it.

[1657] This system allows users to choose the most suitable products and services based on their emotions, and their choices in the virtual world are reflected in their delivery and reservation in the real world, greatly improving the user experience.

[1658] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1659] Step 1:

[1660] The server generates digital twins to recreate real-world physical objects and environments in a digital space. It receives real-world data (e.g., 3D scan data or sensor data) as input and generates the digital twins using 3D modeling software. The output is a 3D model of the simulated world.

[1661] Step 2:

[1662] The user experiences the generated virtual world using a head-mounted display (HMD) or a smartphone. As input, the system receives a 3D model of the virtual world sent from the server and displays it through a VR application. The output is the visual information of the virtual reality that the user experiences.

[1663] Step 3:

[1664] The server uses artificial intelligence (AI) to generate people and objects in the simulated world, enabling them to interact with the user. It receives user behavior data and situational data from the simulated world as input, and uses a generative AI model to generate appropriate people and objects. The output is interactive characters and objects in the simulated world.

[1665] Step 4:

[1666] A user uses an electronic commerce (EC) system to select and purchase products in the virtual world. The system receives information about the product selected by the user as input and performs the purchase procedure through the EC system. The output is purchase confirmation information and delivery arrangement information.

[1667] Step 5:

[1668] The server uses an emotion engine to recognize the user's emotions in real time and recommends products according to those emotions. As input, it receives data on the user's facial expressions and voice captured using a camera and microphone, and analyzes the emotions using an emotion recognition algorithm. The output is product recommendation information according to the user's emotions.

[1669] Step 6:

[1670] The transportation and accommodations selected by the user in the virtual world are reflected in the reservation in the real world. The system receives information on the transportation and accommodations selected by the user as input and connects to the reservation system via an API. The output is reservation confirmation information.

[1671] Step 7:

[1672] The server includes a mechanism for paying a portion of the usage fee for the information provided by the user as an information provision fee. It receives the information provided by the user as input and calculates the usage fee. The output is payment confirmation information for the information provision fee.

[1673] In this way, through the specific actions and data flow at each step, it becomes possible to provide optimal products and services according to the user's emotions, and a system is realized in which choices made in the virtual world are reflected in deliveries and reservations in the real world.

[1674] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1675] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1676] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1677] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1678] [Third embodiment]

[1679] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1680] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1681] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1682] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1683] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1684] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1685] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1686] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1687] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1688] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1689] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1690] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1691] "Example 1"

[1692] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on places a user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, data on the city's streets, buildings, and scenery is acquired, and a simulated world of Paris is generated based on that data.

[1693] "Example 2"

[1694] The generated simulated world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their own home. The people in the simulated world are also generated by AI, and can converse with the generative AI. This allows the user to ask for directions or enjoy everyday conversations.

[1695] "Example 3"

[1696] Furthermore, purchasing within the virtual world is realized through e-commerce. When a user selects and purchases a product at a store within the virtual world, the product is delivered to the user in the real world. It is also linked to real-world transportation and accommodation options, so when a user selects transportation or accommodation within the virtual world, that option is reflected in their reservation in the real world.

[1697] The processing flow of each embodiment will be described below.

[1698] "Example 1"

[1699] Step 1: The user inputs the place they want to visit into the system. For example, if the user wants to visit Paris, they input the information Paris into the system.

[1700] Step 2: The system retrieves data for the relevant location based on the input information. This data includes information on streetscapes, buildings, landscapes, etc.

[1701] Step 3: A digital twin is created based on the acquired data. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[1702] "Example 2"

[1703] Step 1: The user puts on the VR device.

[1704] Step 2: Experience the simulated world created through the VR device. Users can experience walking through the streets of Paris from the comfort of their own home.

[1705] Step 3: Humans in the virtual world are generated by AI and can converse with the generated AI. Users can ask the AI-generated humans for directions or enjoy everyday conversations.

[1706] "Example 3"

[1707] Step 1: The user selects and purchases a product from a store in the virtual world.

[1708] Step 2: The purchased product is delivered to the user in the real world via the e-commerce system.

[1709] Step 3: The user selects transportation and accommodation in the virtual world, which is then reflected in the booking in the real world.

[1710] Example 1

[1711] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1712] Conventional digital twin technology requires a complex process for users to virtually experience the places they want to visit, and lacks a system that users can easily access. Furthermore, there is a lack of means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This makes it difficult for users to have a virtual experience in real time.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1714] In this invention, the server includes means for inputting a place a user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin, thereby enabling the user to easily virtually experience the places they want to visit.

[1715] A "user" is an individual or group who uses the system to specify the place they want to visit and virtually experience it.

[1716] "Input" refers to the act of a user specifying to the system the places they wish to visit, or the information therein.

[1717] "Geographical data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layout.

[1718] "Analysis" is the process of calculations and data processing used to process collected geographic data and convert it into a 3D model.

[1719] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographic data.

[1720] A "digital twin" is a 3D model that recreates a specific location in the real world in a virtual space.

[1721] "Sending" is the act of transferring the digital twin generated by the server to the user's device.

[1722] "Operation" refers to the act of a user interactively using a digital twin on a terminal.

[1723] This invention is a system that allows users to virtually experience places they would like to visit. The system involves a series of processes: a user inputs the place they would like to visit, collects geographical data about the place, analyzes it, converts it into a 3D model, generates a digital twin, and transmits it to the user's device.

[1724] Hardware and software used

[1725] server

[1726] The server receives user input, collects and analyzes geographic data, and generates the 3D model. Specifically, it uses the following software and services:

[1727] Google Maps API: Used to collect geographic data.

[1728] OpenStreetMap: Used to collect complementary geographic data.

[1729] Python: Run scripts to analyze collected geographic data and convert it into 3D models.

[1730] Unity: A game engine for generating 3D models based on the analyzed data.

[1731] Terminal

[1732] The terminal is a device that allows users to operate the digital twin and virtually experience the place they want to visit. Specifically, it uses the following software:

[1733] Unity application: displays the 3D model sent from the server and allows the user to manipulate it.

[1734] Data processing and calculation

[1735] Data collection

[1736] The server collects geographical data from Google Maps API and OpenStreetMap based on the place the user wants to visit. For example, if the user enters "Paris," the server retrieves data such as Paris's latitude and longitude, building shapes, and road layout.

[1737] Data analysis

[1738] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and converts the building shape into polygon data. This process generates the basic data for the digital twin.

[1739] 3D model generation

[1740] The server uses Unity to generate a 3D model based on the analyzed data. Specifically, it runs Unity C scripts to create 3D models of Paris's streets and buildings. This creates a digital twin that allows users to virtually visit Paris.

[1741] Sending data

[1742] The server then sends the generated 3D model to the user's device, where the user can launch a Unity application and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll down the Champs-Élysées.

[1743] Specific examples

[1744] If a user inputs "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographic data for Paris. Then, it runs a Python script to analyze the retrieved geographic data and convert it into a 3D model. Using Unity, it generates a 3D model of Paris's streets and buildings based on the analyzed data. Finally, it sends the generated 3D model to the user's device, where the user can launch the Unity application and virtually experience Paris.

[1745] Prompt Sentence Examples

[1746] "Create a digital twin of Paris. Use Google Maps API to get geographic data of Paris and generate a 3D model using Unity."

[1747] In this way, a system is constructed that allows users to virtually experience places they would like to visit.

[1748] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1749] Step 1:

[1750] The user enters the location they want to visit.

[1751] Specifically, a user accesses the system's web interface, enters "Paris" in the search box, and clicks the search button.

[1752] Input: A place you would like to visit (e.g. Paris)

[1753] Output: Places you want to visit (e.g. Paris)

[1754] Step 2:

[1755] The server receives user input and collects geographic data.

[1756] Specifically, the server sends a request to the Google Maps API to retrieve geographic data for Paris. For example, it sends a request to the URL https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[1757] Input: Information about the place you want to visit (e.g. Paris)

[1758] Output: Collected geographical data (e.g., Paris's latitude and longitude, building shapes, road layout)

[1759] Step 3:

[1760] The server analyzes the collected geographic data and converts it into a 3D model.

[1761] Specifically, the server runs a Python script to parse the retrieved geographic data. For example, it runs a script called parse_geodata.py to convert the latitude and longitude information into 3D coordinates.

[1762] Input: Collected geographical data (e.g., latitude and longitude information of Paris, building shapes, road layout)

[1763] Output: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[1764] Step 4:

[1765] The server generates a 3D model based on the analyzed data.

[1766] Specifically, the server runs a Unity C script to create a 3D model of the Paris cityscape and buildings. For example, it runs a script called GenerateParisModel.cs.

[1767] Input: Basic data of the analyzed 3D model (e.g. 3D coordinates, polygon data)

[1768] Output: Generated 3D model (e.g. 3D model of Paris cityscape and buildings)

[1769] Step 5:

[1770] The server sends the generated 3D model to the user's device.

[1771] Specifically, the server transfers the generated 3D model to the user's device.

[1772] Input: Generated 3D model (e.g., 3D model of Paris cityscape or buildings)

[1773] Output: The 3D model sent to the user's device

[1774] Step 6:

[1775] The user operates the digital twin on the device.

[1776] Specifically, users can launch the Unity application on their device and interact with the digital twin, for example, virtually visiting the Eiffel Tower or strolling down the Champs-Élysées.

[1777] Input: The 3D model sent to the user's device

[1778] Output: A digital twin operated by the user

[1779] (Application example 1)

[1780] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1781] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has difficulty recreating the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to get detailed information about the places you want to visit and experience their atmosphere in advance. This often leaves consumers feeling anxious because they cannot check the details of products and places before purchasing. Furthermore, the lack of ways to link with real-world transportation and accommodations can lead to a lack of consistency in travel plans.

[1782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1783] In this invention, the server includes: a means for generating a simulated world using a digital twin; a means for experiencing the simulated world in VR; a means for generating a person in the simulated world using AI and enabling conversation with the generating AI; a means for making purchases in the simulated world via e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for acquiring data on places the user wants to visit and generating a digital twin based on the data; a means for providing a virtual shopping tour based on the digital twin; and a means for experiencing the virtual shopping tour via a smartphone or head-mounted display. This allows users to experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[1784] A "digital twin" is a virtual representation of a physical place or object in the real world as digital data.

[1785] A "pseudo-world" is a virtual environment or space created using a digital twin.

[1786] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[1787] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and reason.

[1788] "Generative AI" is a technology that uses artificial intelligence to generate people and objects in virtual space.

[1789] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the Internet.

[1790] "Transportation" refers to the means of transportation or method by which a user physically travels.

[1791] "Accommodation" means a facility or place where users can stay temporarily.

[1792] "Information Fee" means the fee paid by the User for the information provided.

[1793] A "virtual shopping tour" is a tour that allows you to experience shopping in a virtual space created using a digital twin.

[1794] A "smartphone" is a type of mobile phone, a multi-function device that can connect to the Internet and use applications.

[1795] A "head-mounted display" is a display device worn on the head, and is used to experience virtual reality and augmented reality.

[1796] A system for implementing this invention generates a simulated world using a digital twin and allows users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[1797] System Configuration

[1798] The system mainly consists of the following components:

[1799] 1. Server: Generates the digital twin and provides data upon user request.

[1800] 2. Device: The smartphone or head-mounted display used by the user.

[1801] 3. User: Use the system to experience a virtual shopping tour.

[1802] Program processing

[1803] The server implements the system using the following means:

[1804] Digital twin generation method: Obtain data on the place the user wants to visit and generate a digital twin based on that data. Specifically, the location data is obtained through an API and a virtual space is generated using a 3D rendering library.

[1805] VR experience: The generated digital twin can be experienced in VR. Users can move freely within the virtual space using a smartphone or head-mounted display.

[1806] AI generation: AI generates humans in virtual space and enables conversations with users. Generative AI models are used to achieve natural conversations in real time.

[1807] EC implementation: E-commerce is used to enable purchasing within a virtual space. Users can select products in a virtual store and complete the purchase process.

[1808] Linking: Linking to real-world transportation and accommodations, allowing users to make reservations and arrangements within the virtual space.

[1809] Payment method for information provision fee: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[1810] Hardware and software used

[1811] Hardware: Smartphone, head-mounted display

[1812] Software: Python, 3D rendering library (e.g. some_3d_rendering_library), generative AI model, API

[1813] Specific examples

[1814] If a user wants to go shopping on the Champs-Élysées in Paris, the system works as follows:

[1815] 1. The user accesses the system using a smartphone or head-mounted display.

[1816] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[1817] 3. Users can experience the generated digital twin in VR and move freely within the virtual space.

[1818] 4. Talk to an AI-generated human in the virtual space and receive directions and product explanations.

[1819] 5. The user selects products in the virtual store and completes the purchase through e-commerce.

[1820] 6. Book real-world transportation and accommodations, if necessary.

[1821] Prompt Sentence Examples

[1822] Users can enter the name of a city they want to visit, for example, "Paris," which will generate a digital twin of the Champs-Élysées in Paris and allow them to start a virtual shopping tour.

[1823] In this way, users can experience the detailed information and atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[1824] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1825] Step 1:

[1826] The user accesses the system using a smartphone or head-mounted display. The user inputs the name of the place they want to visit. The inputted name of the place is sent to the server.

[1827] Step 2:

[1828] The server retrieves the corresponding location data based on the input location name. Specifically, it requests the location data through the API and parses the retrieved data. The input is the location name, and the output is the location details.

[1829] Step 3:

[1830] The server generates a digital twin based on the acquired data. It uses a 3D rendering library to recreate detailed location data as a virtual space. The input is detailed location data, and the output is a digital twin.

[1831] Step 4:

[1832] The server sends the generated digital twin to the user's device, where the user experiences the digital twin in VR via a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[1833] Step 5:

[1834] The server generates a human in the virtual space using a generative AI model. The generated AI can converse with the user. The input is the generative AI model and the virtual space data, and the output is an AI-generated human.

[1835] Step 6:

[1836] The user interacts with an AI-generated human in a virtual space, receiving directions and product explanations. The input is the user's question or request, and the output is a response from the AI-generated human.

[1837] Step 7:

[1838] The user selects a product in a virtual store and completes the purchase procedure. The server then executes the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[1839] Step 8:

[1840] Users book transportation and accommodation in the real world as needed. The server processes this reservation information and provides it to the user. The input is the user's reservation information, and the output is a reservation confirmation.

[1841] Step 9:

[1842] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is confirmation of fee collection.

[1843] In this way, users can experience detailed information and the atmosphere of the places they want to visit in advance, reducing anxiety when shopping online or planning a trip.

[1844] Example 2

[1845] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1846] Conventional virtual reality systems have limited the simulated worlds that users experience, making it difficult to have natural conversations and interactions in real time. Furthermore, purchasing within the simulated world and links to the real world are insufficient, limiting the user experience. Furthermore, there is a lack of a way to effectively use the user's voice to have natural conversations with generative AI.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1848] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating people in the simulated world using artificial intelligence and enabling them to converse with the generative artificial intelligence, means for making purchases in the simulated world through e-commerce, means for linking to transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for converting a user's voice into text and sending it to the generative artificial intelligence as a prompt, and means for converting a response generated by the generative artificial intelligence into voice and returning it to the user. This allows users to enjoy natural conversations in real time, improving interaction within the simulated world and strengthening the link with the real world.

[1849] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1850] A "pseudo-world" is a virtual environment different from the real world that a user experiences through a virtual reality device.

[1851] A "virtual reality device" is a device that allows a user to visually and aurally experience a virtual environment.

[1852] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[1853] "Generative AI" is an AI model that generates natural responses based on user input.

[1854] "Electronic commerce" refers to the buying and selling of goods and services over the Internet.

[1855] A "prompt sentence" is a text-based instruction sentence sent to a generative artificial intelligence based on user input.

[1856] "Speech recognition software" is software for converting speech into text.

[1857] "Speech synthesis software" is software for converting text into speech.

[1858] The present invention provides a system for generating a simulated world that a user experiences through a virtual reality device, and for enabling natural conversation with a generative artificial intelligence. Specific embodiments of this system are described below.

[1859] First, the server generates a virtual world using the digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then provided to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[1860] When a user wears a virtual reality device, the device connects to a server and receives data about the simulated world. Based on the received data, the device renders the simulated world in real time on the device's display, allowing the user to experience the world as if they were actually walking through it.

[1861] The humans in the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the generative artificial intelligence as a prompt.

[1862] Generative AI (e.g., OpenAI's GPT-4) generates an appropriate response based on the prompt it receives. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI generates the response, "The Eiffel Tower is just down this road, turn right at the next intersection."

[1863] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech), which is then transmitted back to the user through the virtual reality device's speakers, allowing the user to hear the response from a human in the simulated world.

[1864] As a concrete example, imagine a user wearing a virtual reality device is walking through the streets of Paris. If the user is near the Eiffel Tower and asks a person in the virtual world, "Where is the Eiffel Tower?", the following prompt is sent to the generative AI:

[1865] Example prompt sentence:

[1866] User: "Where is the Eiffel Tower?"

[1867] The generative AI receives this prompt and generates an appropriate response, which the device converts into speech and returns to the user, allowing the user to enjoy natural conversation in real time.

[1868] This system improves user interaction within the virtual world and strengthens the link with the real world. For example, it enables purchases within the virtual world to be made through e-commerce, and links to real-world transportation and accommodation options. Furthermore, it includes a means to pay a portion of the usage fee as an information provision fee, further enhancing the user experience.

[1869] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1870] Step 1:

[1871] The user puts on the virtual reality device.

[1872] Input: The action of the user putting on the virtual reality device.

[1873] Output: The virtual reality device is activated and the user's vision and hearing are switched to the virtual reality device.

[1874] Specific operation: The user puts the virtual reality device on their head and turns it on, which activates the device and switches the user's vision and hearing to the information provided through the virtual reality device.

[1875] Step 2:

[1876] The device connects to the server and receives data from the virtual world.

[1877] Input: Virtual reality device activation signal.

[1878] Output: Data from the virtual world is sent from the server to the device.

[1879] How it works: The device connects to a server via the Internet, and the server sends data about the virtual world created with Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[1880] Step 3:

[1881] The device renders the simulated world and displays it to the user.

[1882] Input: Pseudo-world data received from the server.

[1883] Output: The simulated world is displayed on the display of the virtual reality device.

[1884] How it works: Based on the received data, the device renders the city of Paris in real time on the display of the virtual reality device, giving the user a visual experience that makes them feel as if they are actually walking through the city of Paris.

[1885] Step 4:

[1886] The user acts within the virtual world and talks to people within the virtual world.

[1887] Input: User's voice input.

[1888] Output: The user's voice is captured on the device.

[1889] Specific operation: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[1890] Step 5:

[1891] The device converts the user's speech into text and sends it to the generative AI model as a prompt.

[1892] Input: User's voice data.

[1893] Output: The prompt in text format.

[1894] How it works: The device captures the user's voice with a microphone and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text as a prompt to the generative AI model.

[1895] Step 6:

[1896] The server generates a response using a generative AI model.

[1897] Input: A prompt in text format.

[1898] Output: The generated response text.

[1899] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt, such as "The Eiffel Tower is just down this road, turn right at the next intersection."

[1900] Step 7:

[1901] The terminal converts the generated response into speech and returns it to the user.

[1902] Input: The generated response text.

[1903] Output: The response in audio format.

[1904] How it works: The device converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then sent back to the user through the speaker of the virtual reality device. The user can then hear the response from the person in the virtual world.

[1905] (Application example 2)

[1906] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1907] Today's consumers want to enjoy shopping without visiting a physical store, but traditional online shopping has the drawback of not being able to actually touch and check the product. It's also difficult to obtain detailed product information through conversations with store staff. Furthermore, there are limited ways for users to experience the streets of foreign countries and enjoy the local atmosphere from the comfort of their own homes. To address these challenges, a system that offers a more realistic shopping experience and interactive communication with users is needed.

[1908] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1909] In this invention, the server includes: a means for generating a virtual world using a digital twin; a means for experiencing the virtual world in VR; a means for generating a human in the virtual world using AI and enabling conversation with the generated AI; a means for enabling purchases in the virtual world through e-commerce; a means for linking with real-world transportation and accommodation; a means for paying a portion of the usage fee as an information provision fee; a means for tracking the user's location and initiating a conversation with an AI-generated character; a means for processing the user's input and generating a response from the AI; and a means for displaying the response. This allows users to enjoy a realistic shopping experience from the comfort of their own home, obtaining detailed information about products through interactive communication with store clerks. Users can also experience the cityscapes of foreign countries and enjoy the local atmosphere.

[1910] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1911] A "pseudo-world" is a virtual environment or space created using a digital twin.

[1912] "VR" is an abbreviation for virtual reality, a technology that allows you to experience a virtual space generated using computer technology.

[1913] "AI" stands for artificial intelligence, a computer system that mimics human intelligence.

[1914] "Generative AI" is artificial intelligence that interacts with users and generates content.

[1915] "EC" is an abbreviation for electronic commerce, a system for buying and selling goods and services over the Internet.

[1916] "Real-world transportation" refers to the means of transportation used by users to travel in the real world.

[1917] "Accommodation" refers to a facility where users can stay.

[1918] "Information Fee" means the fee paid by a user for the information provided by the user.

[1919] "Tracking user location" means tracking the user's current location in real time.

[1920] An "AI-generated character" is a virtual person or character generated by artificial intelligence.

[1921] "Processing user input" means analyzing input data from the user and generating an appropriate response.

[1922] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[1923] "Displaying the response" means visually presenting the generated response to the user.

[1924] To implement the present invention, the following system configuration and program are required.

[1925] System Configuration

[1926] 1. Hardware:

[1927] VR devices (e.g. Oculus Rift, HTC Vive)

[1928] Smartphone (e.g. iPhone, Android)

[1929] Server (with high-performance computing capabilities)

[1930] 2. Software:

[1931] VR engine (e.g. Unity, Unreal Engine)

[1932] AI conversation engine (e.g. OpenAI GPT-3)

[1933] Program processing explanation

[1934] The server first uses the digital twin to generate a virtual world that recreates the streets of Paris. The user then wears a VR device and experiences this virtual world. The user's location is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[1935] User input is sent to the server via the VR device or smartphone, which then analyzes the input and generates an appropriate response using an AI conversation engine, which then displays the response in the user's field of view.

[1936] Specific examples

[1937] When a user walks into a virtual store and asks an AI-generated sales associate, "Do you have this dress in my size?", the server receives this input and uses its AI conversation engine to generate a response: "Yes, this dress is available in sizes S, M, and L." This response is displayed in the user's field of view.

[1938] Prompt Sentence Examples

[1939] User: Do you have this dress in your size?

[1940] AI: Yes, this dress is available in sizes S, M, and L.

[1941] In this way, users can enjoy a realistic shopping experience from the comfort of their own home, obtain detailed information about products through interactive communication with store staff, and even experience the local atmosphere of a foreign city.

[1942] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1943] Step 1:

[1944] The server uses a digital twin to generate a simulated world that recreates the streets of Paris.

[1945] Input: Paris streetscape data

[1946] Data processing: Using digital twin technology, the real-world streets of Paris are recreated in a virtual space.

[1947] Output: A simulated world of Parisian streets

[1948] Step 2:

[1949] The user wears a VR device and experiences a simulated world transmitted from the server.

[1950] Input: Pseudo-world data sent from the server

[1951] Data calculation: The VR device tracks the user's viewpoint and movements in real time and displays a simulated world.

[1952] Output: A simulated world displayed to the user.

[1953] Step 3:

[1954] The server tracks the user's location in real time.

[1955] Input: User's location

[1956] Data calculation: Calculates the user's position in real time and updates their position in the virtual world.

[1957] Output: Updated user location

[1958] Step 4:

[1959] When a user approaches an AI-generated character, the server initiates a conversation.

[1960] Input: User location, AI-generated character location

[1961] Data calculation: Calculates the distance between the user and the character, and starts conversation when they come within a certain distance.

[1962] Output: Trigger to start a conversation

[1963] Step 5:

[1964] Users send input to the server through their VR device or smartphone.

[1965] Input: User voice and text input

[1966] Data calculation: Sends input data to the server.

[1967] Output: User input data sent to the server

[1968] Step 6:

[1969] The server analyzes the user's input and generates an appropriate response using an AI conversation engine.

[1970] Input: User-entered data

[1971] Data Computing: Using an AI conversation engine (e.g., OpenAI GPT-3) to generate appropriate responses to user input.

[1972] Output: The generated response data

[1973] Step 7:

[1974] The server displays the generated response in the user's view.

[1975] Input: Generated response data

[1976] Data calculation: Send the response data to the VR device and display it in the user's field of view.

[1977] Output: The response displayed in the user's view

[1978] In this way, users can enjoy a realistic shopping experience from the comfort of their own home. They can also obtain detailed information about products through interactive communication with store staff. They can also experience the streets of foreign countries and enjoy the local atmosphere.

[1979] Example 3

[1980] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1981] There is a demand for a system that allows efficient purchasing and reservation procedures within a virtual world linked to the real world. In conventional systems, the process of reflecting actions in the virtual world in the real world is complicated, resulting in low user convenience. There is also a need for a system that makes the experience in the virtual world more realistic and allows users to freely choose the location and time.

[1982] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a pseudo-world using a digital twin, means for experiencing the pseudo-world in virtual reality, means for generating characters in the pseudo-world using artificial intelligence and enabling conversations with the generative artificial intelligence, means for making purchases in the pseudo-world through e-commerce, means for linking with transportation and accommodation in the real world, means for paying a portion of the usage fee as an information provision fee, means for arranging delivery in the real world when a user selects a product in the pseudo-world and confirms the purchase, and means for arranging reservations in the real world when a user selects transportation and accommodation in the pseudo-world. This allows the user's actions in the pseudo-world to be reflected in the real world quickly and efficiently.

[1983] A "digital twin" is a virtual model that digitally recreates a real-world physical object or environment.

[1984] A "pseudo-world" is a virtual reality environment created using a digital twin.

[1985] "Virtual reality" is a technology that allows users to experience an artificial environment generated using computer technology.

[1986] "Artificial intelligence" is a technology that allows computer systems to behave in a way that mimics human intelligence.

[1987] "Generative AI" is a system that uses artificial intelligence technology to generate characters in a virtual world and enable them to converse with the user.

[1988] "Electronic commerce" is a form of transaction in which goods and services are bought and sold over the Internet.

[1989] "Transportation" is the means by which a user travels from one place to another.

[1990] "Accommodation" means a facility where users can stay temporarily.

[1991] "Information provision fee" is part of the fee paid by users when they use the system, and is paid in exchange for the provision of information.

[1992] "Shipping procedure" refers to the procedure for shipping the products purchased by the user in the real world.

[1993] "Booking Process" means the process by which a User makes a real-world reservation for their chosen mode of transportation and accommodation.

[1994] MODE FOR CARRYING OUT THE INVENTION

[1995] This invention is a system that reflects the actions of a user in a virtual world, such as purchasing products and selecting transportation and accommodations, in the real world. Specific embodiments of this system are described below.

[1996] Hardware and software used

[1997] Server: Database management system (e.g., MySQL), web server (e.g., Apache)

[1998] Device: PC or smartphone used by the user

[1999] Generative AI models: Natural language processing models (e.g., GPT-4)

[2000] Data processing and calculation

[2001] 1. The user selects a product in a virtual world

[2002] Users use their devices to visit stores in the virtual world.

[2003] When a user selects a product, the information is sent to the server.

[2004] The server stores the data of the selected products in a database.

[2005] 2. Completing the purchase process

[2006] Once the user confirms the purchase, the server processes the purchase through the e-commerce system (e.g., Shopify).

[2007] The purchase information is sent to a real-world delivery system, and the product is delivered to the user.

[2008] 3. Choosing transportation and accommodation

[2009] Users choose their means of transportation and accommodation within the virtual world.

[2010] The selection information is sent to the server, which makes the real-world reservation through a reservation system (e.g., Booking.com API).

[2011] Specific examples

[2012] Specific examples of product purchases

[2013] A user chooses a smartphone in a virtual electronics store.

[2014] When the purchase button is clicked, the server completes the purchase process through the e-commerce system and the smartphone is delivered to the user's address.

[2015] Specific examples of transportation options

[2016] The user chooses a taxi within the virtual world.

[2017] The server makes taxi reservations in the real world through a taxi reservation system.

[2018] Prompt Sentence Examples

[2019] Purchase prompts

[2020] A user wants to purchase a smartphone in a simulated world. Please explain the steps to complete the purchase and have it shipped to a real-world address.

[2021] Prompt for transportation selection

[2022] A user wants to book a taxi in a simulated world. Please explain the steps to go through the booking process and complete the taxi booking in the real world.

[2023] This system allows the user's actions in the virtual world to be reflected in the real world quickly and efficiently. The server receives the selection information from the user and performs the procedure in the real world through the appropriate system. This significantly improves user convenience. The flow of the specific processing in Example 3 will be explained using Figure 15.

[2024] Step 1:

[2025] The user visits a store in the virtual world using a terminal. The user selects a product and clicks the select button. The input is the product information selected by the user, and the output is that information being sent to the server. In concrete terms, the user selects a smartphone at an electronics store in the virtual world.

[2026] Step 2:

[2027] The server receives the product selection information sent by the user. The input is the product information sent by the user, and the output is that information is saved in a database. Specifically, the server receives the selection information from the "smartphone" and saves it in the database.

[2028] Step 3:

[2029] The user clicks the button to confirm the purchase on the purchase confirmation screen. The input is the user's purchase confirmation operation, and the output is that information being sent to the server. The specific action is when the user clicks the "Confirm Purchase" button on the purchase confirmation screen.

[2030] Step 4:

[2031] The server connects to the EC system and starts the purchase procedure. The input is the user's purchase confirmation informat...

Claims

1. A means of generating a simulated world using a digital twin; A means for experiencing the pseudo world in virtual reality; A means for generating a person in the pseudo world using artificial intelligence and enabling a conversation with the generating artificial intelligence; A means for realizing purchase actions in the virtual world through electronic commerce; a means for linking to real-world transportation or accommodation; A means for paying a part of the usage fee as an information provision fee; A means for carrying out a delivery procedure in the real world when a user selects a product in the virtual world and confirms the purchase; and a means for carrying out reservation procedures in the real world when the user selects the means of transportation or the accommodation in the virtual world.

2. A means of generating a simulated world using a digital twin; A means for experiencing the pseudo world in virtual reality; A means for generating a person in the pseudo world using artificial intelligence and enabling a conversation with the generating artificial intelligence; A means for realizing purchase actions in the virtual world through electronic commerce; a means for linking to real-world transportation or accommodation; A means for paying a part of the usage fee as an information provision fee; A means for recognizing a user's emotion using an emotion engine and recommending products according to the emotion; a means for transmitting the emotion data of the user recognized by the emotion engine to a server, and the server then selects a product based on the emotion; and a means for transmitting the selected product information to a user's terminal, and the terminal displaying the recommended products.

3. A means of generating a simulated world using a digital twin; A means for experiencing the pseudo world in virtual reality; A means for generating humans in the pseudo-world using artificial intelligence and enabling conversation with the generating artificial intelligence; A means for realizing purchase actions in the virtual world through electronic commerce; a means for linking to real-world transportation or accommodation; A means for paying a part of the usage fee as an information provision fee; A means for recognizing a user's emotion using an emotion engine and recommending products according to the emotion; A means for delivering the product selected in the virtual world in the real world; and a means for reserving the means of transportation or the accommodation selected in the virtual world in the real world.

Images