system

Abstract

To provide a system capable of eliminating various problems for travels in an actual world.SOLUTION: The system includes: means for generating an artificial world using digital twins; means for allowing experience of the artificial world in virtual reality; means for allowing communication with generative AI by producing a person in the artificial world using AI; mean for allowing purchasing activity in the artificial world to be realized in e-commerce; means for linking purchasing activity in the artificial world with transportation means or accommodation in an actual world; means for paying a part of a usage fee as an information provision fee; means for allowing delivery in the actual world when a user selects a product in the artificial world and confirms purchasing it; and means for carrying out reservation procedures in the actual world when the user selects the transportation means or the accommodation in the artificial world.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Travel in the real world has various problems such as time, cost, and physical constraints. Also, language barriers and lack of information at the travel destination may sometimes be factors hindering the travel experience.

Means for Solving the Problems

[0005] This invention provides a new travel experience that transcends the limitations of time, cost, and physical space by generating a simulated world using a digital twin and experiencing that simulated world in VR. Furthermore, it overcomes language barriers and information deficiencies by generating humans within the simulated world using AI and enabling conversations with generative AI. Additionally, it integrates with the real world by enabling e-commerce for purchases within the simulated world and linking it with real-world transportation and accommodation. This allows users to obtain a new travel experience that goes beyond the limitations of the real world. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12]This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 of Embodiment 2. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 3 of Example 3. [Figure 16] This is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] This is a sequence diagram showing the processing flow of the data processing system in Example 1 of the Form 1 when an emotion engine is combined. [Figure 18] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] This is a sequence diagram showing the processing flow of the data processing system in Example 2 of the Form 2 when an emotion engine is combined. [Figure 20] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] This is a sequence diagram showing the processing flow of the data processing system in Example 3 of the Form 3 when an emotion engine is combined. [Figure 22] This is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (TENSOR PROCESSING UNIT (registered trademark)), etc.

[0010] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0011] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0012] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0014] [First Embodiment]

[0015] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0016] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on a place the user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, the system acquires data on the streets, buildings, and scenery of Paris and generates a simulated world of Paris based on that data.

[0029] "Example of form 2"

[0030] The generated virtual world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their home. Furthermore, the people in the virtual world are generated by AI, and it is possible to converse with the generative AI. This allows the user to ask for directions and enjoy everyday conversations.

[0031] "Example of form 3"

[0032] Furthermore, purchases within the virtual world are implemented via e-commerce. When a user selects and purchases an item from a store within the virtual world, that item is delivered to the user in the real world. It is also linked to real-world transportation and accommodation; when a user selects transportation or accommodation within the virtual world, it is reflected in their real-world reservations.

[0033] The following describes the processing flow for each example of the form.

[0034] "Example of form 1"

[0035] Step 1: The user enters the place they want to visit into the system. For example, if the user wants to visit Paris, they would enter "Paris" into the system.

[0036] Step 2: The system retrieves data for the relevant location based on the entered information. This data includes information about the streetscape, buildings, and scenery.

[0037] Step 3: Based on the acquired data, a digital twin is generated. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[0038] "Example of form 2"

[0039] Step 1: The user puts on the VR device.

[0040] Step 2: Experience the generated virtual world through the VR device you're wearing. Users can experience walking through the streets of Paris from the comfort of their own homes.

[0041] Step 3: Humans in the simulated world are generated by AI, and it is possible to converse with the generative AI. Users can ask AI-generated humans for directions or enjoy everyday conversations.

[0042] "Example of form 3"

[0043] Step 1: The user selects and purchases products in a store within a simulated world.

[0044] Step 2: Purchased items are delivered to the user in the real world via the e-commerce system.

[0045] Step 3: When a user selects transportation and accommodation within the virtual world, these selections are reflected in their real-world reservations.

[0046] (Example 1)

[0047] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0048] Conventional digital twin technologies present challenges such as a complex process for users to virtually experience places they wish to visit, and a lack of easily accessible systems. Furthermore, there are insufficient means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This results in difficulties for users to engage in real-time virtual experiences.

[0049] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0050] In this invention, the server includes means for inputting a place the user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin. This makes it possible for the user to easily virtually experience a place they want to visit.

[0051] A "user" is an individual or group that uses the system to specify a place they want to visit and virtually experience it.

[0052] "Input" refers to the act or information that a user provides to the system regarding a place they wish to visit.

[0053] "Geographic data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layouts.

[0054] "Analysis" refers to the process of calculation and data processing to process collected geographic data and convert it into a 3D model.

[0055] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographical data.

[0056] A "digital twin" is a 3D model that recreates a specific location in the real world within a virtual space.

[0057] "Transmission" refers to the act of transferring the digital twin generated by the server to the user's terminal.

[0058] "Operation" refers to the act of a user interactively using a digital twin on a device.

[0059] This invention is a system for users to virtually experience places they wish to visit. The system includes a series of processes: the user inputs the place they wish to visit, geographic data about that place is collected and analyzed, converted into a 3D model, a digital twin is generated, and transmitted to the user's terminal.

[0060] Hardware and software to be used

[0061] server

[0062] The server is responsible for receiving user input, collecting and analyzing geographic data, and generating 3D models. Specifically, it uses the following software and services:

[0063] Google® Maps API: Used to collect geographic data.

[0064] OpenStreetMap: Used to collect supplementary geographic data.

[0065] Python: Run a script to analyze the collected geographic data and convert it into a 3D model.

[0066] Unity: A game engine for generating 3D models based on analyzed data.

[0067] terminal

[0068] The terminal is a device that allows users to interact with a digital twin and virtually experience places they wish to visit. Specifically, it uses the following software:

[0069] Unity application: Displays 3D models sent from the server and allows the user to interact with them.

[0070] Data processing and data calculation

[0071] Data collection

[0072] The server collects geographic data from the Google Maps API and OpenStreetMap based on the location the user enters. For example, if the user enters "Paris," the server retrieves data such as the latitude and longitude of Paris, the shape of buildings, and the layout of roads.

[0073] Data Analysis

[0074] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and building shapes into polygon data. This process generates the basic data for the digital twin.

[0075] 3D model generation

[0076] The server uses Unity to generate 3D models based on the analyzed data. Specifically, it executes Unity's C scripts to create 3D models of Parisian streets and buildings. This completes a digital twin that allows users to virtually visit Paris.

[0077] Sending data

[0078] The server sends the generated 3D model to the user's device. The user can then launch the Unity application on their device and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[0079] Specific example

[0080] When a user enters "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographical data for Paris. Next, a Python script is executed to analyze the retrieved geographical data and convert it into a 3D model. Unity is used to generate a 3D model of Paris's streets and buildings based on the analyzed data. Finally, the generated 3D model is sent to the user's device, and the user can launch the Unity application on their device to virtually experience Paris.

[0081] Example of a prompt

[0082] "Please create a digital twin of Paris. Use the Google Maps API to obtain geographical data for Paris and generate a 3D model using Unity."

[0083] In this way, a system is built that allows users to virtually experience places they want to visit.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] The user enters the place they want to visit.

[0087] Specifically, the user accesses the system's web interface, enters "Paris" into the search box, and clicks the search button.

[0088] Input: The location the user wants to visit (e.g., Paris)

[0089] Output: Information about the place you want to visit (e.g., Paris)

[0090] Step 2:

[0091] The server receives user input and collects geographical data.

[0092] Specifically, the server sends a request to the Google Maps API to retrieve geographical data for Paris. For example, it sends a request to a URL like https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[0093] Input: Information about the place you want to visit (e.g., Paris)

[0094] Output: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[0095] Step 3:

[0096] The server analyzes the geographic data it collects and converts it into a 3D model.

[0097] Specifically, the server executes a Python script to analyze the acquired geographic data. For example, it runs a script called parse_geodata.py to convert latitude and longitude information into 3D coordinates.

[0098] Input: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[0099] Output: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[0100] Step 4:

[0101] The server generates a 3D model based on the analyzed data.

[0102] Specifically, the server executes Unity C scripts to create 3D models of Parisian streets and buildings. For example, it executes a script called GenerateParisModel.cs.

[0103] Input: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[0104] Output: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[0105] Step 5:

[0106] The server generates a 3D model and sends it to the user's device.

[0107] Specifically, the server transfers the generated 3D model to the user's terminal.

[0108] Input: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[0109] Output: 3D model sent to the user's terminal

[0110] Step 6:

[0111] Users interact with the digital twin on their devices.

[0112] In terms of specific actions, the user launches a Unity application on their device and interacts with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[0113] Input: 3D model sent to the user's device

[0114] Output: User-operated digital twin

[0115] (Application Example 1)

[0116] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0117] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping struggles to replicate the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to experience detailed information and the atmosphere of places they want to visit beforehand. This often leads to anxiety for consumers who cannot verify product or location details before purchasing. Additionally, the lack of ways to link online shopping with real-world transportation and accommodation can result in a lack of consistency in travel planning.

[0118] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0119] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for acquiring data on places the user wants to visit and generating a digital twin based on that data, means for providing a virtual shopping tour based on the digital twin, and means for experiencing the virtual shopping tour through a smartphone or head-mounted display. This makes it possible for users to experience detailed information and the atmosphere of places they want to visit in advance, thereby reducing anxiety when online shopping or planning trips.

[0120] A "digital twin" is a virtual representation of physical locations and objects in the real world, created as digital data.

[0121] A "pseudo-world" refers to a virtual environment or space created using a digital twin.

[0122] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[0123] "AI" is an abbreviation for artificial intelligence, which is a technology in which computers imitate human intelligence to learn and reason.

[0124] "Generative AI" is a technology that uses artificial intelligence to generate humans and objects in a virtual space.

[0125] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the internet.

[0126] "Means of transportation" refers to the means or methods of transport that a user uses to move physically.

[0127] "Accommodation facilities" refer to facilities or places where users can stay temporarily.

[0128] "Information provision fee" refers to the fee that a user pays for the information provided.

[0129] A "virtual shopping tour" is a tour that allows users to experience shopping in a virtual space created using a digital twin.

[0130] A "smartphone" is a type of mobile phone, a multi-functional device capable of internet access and application use.

[0131] A "head-mounted display" is a display device worn on the head and used to experience virtual reality and augmented reality.

[0132] The system for carrying out this invention generates a simulated world using a digital twin, allowing users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[0133] System Configuration

[0134] The system mainly consists of the following components:

[0135] 1. Server: Generates a digital twin and provides data in response to user requests.

[0136] 2. Device: A smartphone or head-mounted display used by the user.

[0137] 3. User: Experience a virtual shopping tour using the system.

[0138] Program processing

[0139] The server implements the system using the following methods.

[0140] Digital Twin Generation Method: Data on a location the user wishes to visit is acquired, and a digital twin is generated based on that data. Specifically, location data is acquired via an API, and a virtual space is generated using a 3D rendering library.

[0141] VR Experience Method: The generated digital twin can be experienced in VR. Users can use smartphones or head-mounted displays to freely move around and experience the virtual space.

[0142] AI generation method: An AI generates human characters in a virtual space, enabling conversations with the user. The generated AI model is used to achieve natural, real-time conversations.

[0143] E-commerce implementation method: Purchases within a virtual space are realized through electronic commerce. Users can select products in a virtual store and complete the purchase process.

[0144] Linking mechanism: Links to real-world transportation and accommodation. Users can make reservations and arrangements within the virtual space.

[0145] Information provision fee payment method: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[0146] Hardware and software to be used

[0147] Hardware: Smartphones, head-mounted displays

[0148] Software: Python, 3D rendering library (e.g., some_3d_rendering_library), generative AI model, API

[0149] Specific example

[0150] If a user wants to shop on the Champs-Élysées in Paris, the system will operate according to the following steps.

[0151] 1. Users access the system using a smartphone or head-mounted display.

[0152] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[0153] 3. Users experience the generated digital twin in VR and move freely within the virtual space.

[0154] 4. Interact with AI-generated humans in a virtual space and receive directions and product descriptions.

[0155] 5. Users select products in a virtual store and complete the purchase process through e-commerce.

[0156] 6. Book real-world transportation and accommodation as needed.

[0157] Example of a prompt

[0158] Please enter the name of the city you would like to visit. For example, if you enter "Paris," a digital twin of the Champs-Élysées in Paris will be generated, allowing you to start a virtual shopping tour.

[0159] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[0160] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0161] Step 1:

[0162] The user accesses the system using a smartphone or head-mounted display. The user enters the name of the place they want to visit. The entered place name is sent to the server.

[0163] Step 2:

[0164] The server retrieves data for a location based on the entered location name. Specifically, it requests location data via an API and parses the retrieved data. The input is the location name, and the output is detailed location data.

[0165] Step 3:

[0166] The server generates a digital twin based on the acquired data. Using a 3D rendering library, it recreates detailed location data as a virtual space. The input is detailed location data, and the output is the digital twin.

[0167] Step 4:

[0168] The server sends the generated digital twin to the user's device. The user then experiences the digital twin in VR through a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[0169] Step 5:

[0170] The server generates human characters in a virtual space using a generative AI model. The generated AI enables conversation with the user. The input is the generative AI model and data from the virtual space, and the output is the AI-generated human character.

[0171] Step 6:

[0172] Users converse with AI-generated humans in a virtual space, receiving directions and product descriptions. Input consists of user questions and requests, while output is the AI-generated human's response.

[0173] Step 7:

[0174] Users select products in a virtual store and proceed with the purchase. The server facilitates the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[0175] Step 8:

[0176] Users book real-world transportation and accommodations as needed. The server processes this booking information and provides it to the user. The input is the user's booking information, and the output is a booking confirmation.

[0177] Step 9:

[0178] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is a confirmation of fee collection.

[0179] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[0180] (Example 2)

[0181] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0182] Traditional virtual reality systems have limitations, such as the restricted virtual world experienced by users, making natural real-time conversation and interaction difficult. Furthermore, the lack of sufficient linkage between the virtual world and the real world, including inadequate purchasing capabilities, limits the user experience. Additionally, there is a lack of effective means to utilize user voice to facilitate natural conversations with generative artificial intelligence.

[0183] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0184] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating humans in the simulated world with artificial intelligence and enabling conversation with the generative AI, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for converting the user's voice into text and sending it to the generative AI as a prompt, and means for converting the response generated by the generative AI into voice and returning it to the user. As a result, the user can enjoy natural conversations in real time, interaction within the simulated world is improved, and the link with the real world is strengthened.

[0185] A "digital twin" is a virtual model that digitally reproduces physical objects and environments in the real world.

[0186] A "virtual world" is a virtual environment that users experience through a virtual reality device, and which is different from the real world.

[0187] A "virtual reality device" is a device that allows users to experience a virtual environment visually and aurally.

[0188] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[0189] "Generative artificial intelligence" refers to artificial intelligence models that generate natural-sounding responses based on user input.

[0190] "E-commerce" refers to transactions involving the purchase and sale of goods and services via the internet.

[0191] A "prompt message" is a text-based instruction sent to a generative artificial intelligence system based on user input.

[0192] "Speech recognition software" is software used to convert speech into text.

[0193] "Speech synthesis software" is software used to convert text into speech.

[0194] This invention is a system that generates a simulated world experienced by the user through a virtual reality device and enables natural conversation with generative artificial intelligence. Specific embodiments of this system are described below.

[0195] First, the server generates a virtual world using a digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then delivered to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[0196] When a user puts on a virtual reality device, the device connects to a server and receives data from the virtual world. Based on the received data, the device renders the virtual world on the virtual reality device's display in real time. This allows the user to experience as if they were actually walking around in that place.

[0197] Humans within the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the audio with a microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is then sent to the generative artificial intelligence as a prompt.

[0198] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates an appropriate response based on the received prompt. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI will generate the response, "The Eiffel Tower is located straight down this road, and you'll turn right at the next intersection."

[0199] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then returned to the user through the virtual reality device's speakers. This allows the user to hear responses from people within the simulated world.

[0200] As a concrete example, suppose a user is walking around Paris wearing a virtual reality device. If the user asks a person in the virtual world, "Where is the Eiffel Tower?" near the Eiffel Tower, the following prompt will be sent to the generative artificial intelligence:

[0201] Example of a prompt:

[0202] User: "Where is the Eiffel Tower?"

[0203] The generative artificial intelligence receives this prompt and generates an appropriate response. The terminal converts this response into speech and returns it to the user. This allows the user to enjoy a natural conversation in real time.

[0204] This system enhances user interaction within the virtual world and strengthens its link to the real world. For example, it enables e-commerce transactions within the virtual world and links to real-world transportation and accommodation. Furthermore, it includes a method for paying a portion of the usage fee as an information provision fee, further enriching the user experience.

[0205] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0206] Step 1:

[0207] The user puts on a virtual reality device.

[0208] Input: The action of a user putting on a virtual reality device.

[0209] Output: The virtual reality device is activated, and the user's sight and hearing are switched to the virtual reality device.

[0210] Specific operation: The user puts the virtual reality device on their head and turns on the device. This activates the device, and the user's sight and hearing switch to information provided through the virtual reality device.

[0211] Step 2:

[0212] The device connects to the server and receives data from the simulated world.

[0213] Input: Activation signal for a virtual reality device.

[0214] Output: Data from the simulated world is sent from the server to the terminal.

[0215] Specific operation: The device connects to the server via the internet, and the server sends data of a simulated world generated by Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[0216] Step 3:

[0217] The device renders a simulated world and displays it to the user.

[0218] Input: Data from a simulated world received from the server.

[0219] Output: A simulated world is displayed on the virtual reality device's screen.

[0220] Specific operation: Based on the received data, the device renders the streets of Paris in real time on the virtual reality device's display. Through the virtual reality device, the user experiences a visual sensation as if they were actually walking through the streets of Paris.

[0221] Step 4:

[0222] The user acts within a simulated world and speaks to people within that world.

[0223] Input: User voice input.

[0224] Output: The user's voice is captured on the device.

[0225] Specific actions: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[0226] Step 5:

[0227] The device converts the user's voice into text and sends it as a prompt to the generative AI model.

[0228] Input: User's voice data.

[0229] Output: A text-formatted prompt message.

[0230] Specific operation: The device captures the user's voice using the microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the AI ​​model as a prompt.

[0231] Step 6:

[0232] The server generates a response using a generated AI model.

[0233] Input: A text-based prompt message.

[0234] Output: The generated response text.

[0235] Specific operation: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt. For example, it might generate a response like, "Go straight down this road and turn right at the next intersection."

[0236] Step 7:

[0237] The device converts the generated response into speech and returns it to the user.

[0238] Input: The generated response text.

[0239] Output: A voice response.

[0240] Specific operation: The terminal converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is returned to the user through the virtual reality device's speaker. The user can hear the response from a person in the simulated world.

[0241] (Application Example 2)

[0242] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0243] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has the problem of not being able to actually touch and examine products. Furthermore, it's difficult to obtain detailed information about products through interaction with store staff when shopping online. In addition, there are limited ways for users to experience the streetscapes of foreign cities and enjoy the local atmosphere from the comfort of their homes. To solve these problems, a system is needed that provides a more realistic shopping experience and interactive communication with users.

[0244] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0245] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for tracking the user's location and initiating a conversation with an AI-generated character, means for processing user input and generating a response from the AI, and means for displaying the response. As a result, users can enjoy a realistic shopping experience from the comfort of their homes and obtain detailed information about products through interactive communication with store staff. They can also experience the streetscapes of foreign countries and enjoy the local atmosphere.

[0246] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[0247] A "simulated world" is a virtual environment or space created using a digital twin.

[0248] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[0249] "AI" is an abbreviation for artificial intelligence, which is a computer system that mimics human intelligence.

[0250] "Generative AI" refers to artificial intelligence used for interacting with users and generating content.

[0251] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[0252] "Real-world transportation" refers to the means of transport that users use to travel in the real world.

[0253] An "accommodation facility" is a facility where users stay overnight.

[0254] "Information provision fees" are the fees paid by users for the information they provide.

[0255] "Tracking user location" means tracking the user's current location in real time.

[0256] An "AI-generated character" is a virtual person or character created by artificial intelligence.

[0257] "Processing user input" means analyzing user input data and generating an appropriate response.

[0258] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[0259] "Displaying the response" means visually presenting the generated response to the user.

[0260] To implement this invention, the following system configuration and program are required.

[0261] System Configuration

[0262] 1. Hardware:

[0263] VR devices (e.g., Oculus Rift, HTC Vive)

[0264] Smartphones (e.g., iPhone (registered trademark), ANDROID (registered trademark))

[0265] Servers (those with high-performance computing capabilities)

[0266] 2. Software:

[0267] VR engines (e.g., Unity, Unreal Engine)

[0268] AI conversation engine (e.g., OpenAI GPT-3 (registered trademark))

[0269] Program Processing Description

[0270] The server first generates a simulated world that recreates the streets of Paris using a digital twin. Next, the user can experience this simulated world by wearing a VR device. The user's position is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[0271] User input is sent to the server via a VR device or smartphone. The server analyzes this input and generates an appropriate response using an AI conversation engine. The generated response is displayed in the user's field of view.

[0272] Specific Example

[0273] When a user enters a virtual store and asks an AI-generated store clerk, "Do you have this dress in my size?", the server receives this input and uses an AI conversation engine to generate a response: "Yes, this dress is available in sizes S, M, and L." This response is displayed within the user's field of vision.

[0274] Examples of Prompt Sentences

[0275] User: Do you have this dress in my size?

[0276] AI: Yes, this dress is available in sizes S, M, and L.

[0277] In this way, users can enjoy a realistic shopping experience from the comfort of their homes. Additionally, they can obtain detailed information about products through interactive communication with the store clerk. Furthermore, it is also possible to experience the streets of foreign countries and enjoy the local atmosphere.

[0278] The flow of specific processing in Application Example 2 will be described using FIG. 14.

[0279] Step 1:

[0280] The server generates a virtual world that reproduces the streets of Paris using a digital twin.

[0281] Input: Data of the streets of Paris

[0282] Data Processing: Using digital twin technology, reproduce the streets of Paris in the real world in a virtual space.

[0283] Output: A virtual world that reproduces the streets of Paris

[0284] Step 2:

[0285] Users wear VR devices and experience a simulated world transmitted from a server.

[0286] Input: Data from a simulated world sent from the server.

[0287] Data processing: VR devices track the user's viewpoint and movements in real time to display a simulated world.

[0288] Output: A simulated world displayed in the user's field of view.

[0289] Step 3:

[0290] The server tracks the user's location in real time.

[0291] Input: User's location information

[0292] Data calculation: Calculates the user's position in real time and updates their position within the simulated world.

[0293] Output: Updated user location information

[0294] Step 4:

[0295] When a user approaches an AI-generated character, the server initiates a conversation.

[0296] Input: User's location information, AI-generated character's location information

[0297] Data calculation: Calculates the distance between the user and the character, and initiates a conversation when they are within a certain distance.

[0298] Output: Trigger for starting a conversation

[0299] Step 5:

[0300] Users send input to the server via VR devices or smartphones.

[0301] Input: User voice input or text input

[0302] Data operation: Send the input data to the server.

[0303] Output: The user's input data sent to the server

[0304] Step 6:

[0305] The server analyzes the user's input and generates an appropriate response using the AI conversation engine.

[0306] Input: The user's input data

[0307] Data operation: Use an AI conversation engine (e.g., OpenAI GPT-3) to generate an appropriate response to the user's input.

[0308] Output: The generated response data

[0309] Step 7:

[0310] The server displays the generated response in the user's field of vision.

[0311] Input: The generated response data

[0312] Data operation: Send the response data to the VR device and display it in the user's field of vision.

[0313] Output: The response displayed in the user's field of vision

[0314] In this way, the user can enjoy a real shopping experience while staying at home. Also, through interactive communication with the store staff, the user can obtain detailed information about the products. Furthermore, it is also possible to experience the streets of foreign countries and enjoy the local atmosphere.

[0315] (Example 3)

[0316] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0317] There is a need for a system that can efficiently handle purchases and reservation procedures within a simulated world linked to the real world. Conventional systems have cumbersome processes for reflecting actions in the simulated world back into the real world, resulting in low user convenience. Furthermore, there is a need for a system that makes the experience within the simulated world more realistic and allows users to freely choose locations and times.

[0318] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating people in the simulated world using artificial intelligence and enabling conversation with generative artificial intelligence, means for realizing purchases in the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for performing delivery procedures in the real world when the user selects a product in the simulated world and confirms the purchase, and means for performing reservation procedures in the real world when the user selects transportation or accommodation in the simulated world. As a result, the user's actions in the simulated world are reflected quickly and efficiently in the real world.

[0319] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[0320] A "simulated world" is a virtual reality environment created using a digital twin.

[0321] "Virtual reality" is a technology that allows users to experience an artificial environment created using computer technology.

[0322] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[0323] "Generative artificial intelligence" is a system that uses artificial intelligence technology to generate characters in a simulated world and enables conversations with users.

[0324] "E-commerce" refers to a form of transaction in which goods and services are bought and sold via the internet.

[0325] "Means of transportation" refers to the means by which a user moves from one place to another.

[0326] An "accommodation facility" is a facility where users can stay temporarily.

[0327] "Information provision fees" are a portion of the fees that users pay when using the system, and are paid as compensation for the provision of information.

[0328] "Delivery procedures" refer to the process of delivering the products purchased by the user to the real world.

[0329] "Reservation process" refers to the procedure for users to book their chosen mode of transportation and accommodation in the real world.

[0330] Modes for carrying out the invention

[0331] This invention is a system that reflects user actions in a simulated world, such as purchasing goods or choosing transportation and accommodation, into the real world. Specific embodiments of this system are described below.

[0332] Hardware and software to be used

[0333] Servers: Database management systems (e.g., MySQL®), Web servers (e.g., Apache®)

[0334] Device: The PC or smartphone used by the user.

[0335] Generative AI models: Natural language processing models (e.g., GPT-4)

[0336] Data processing and data calculation

[0337] 1. Users select products within a simulated world.

[0338] Users visit stores within a virtual world using their devices.

[0339] When a user selects a product, that information is sent to the server.

[0340] The server saves the data of the selected product to the database.

[0341] 2. Execute the purchase procedure

[0342] Once the user confirms their purchase, the server processes the purchase through the e-commerce system (e.g., Shopify).

[0343] Purchase information is sent to a real-world delivery system, and the product is delivered to the user.

[0344] 3. Choosing transportation and accommodation

[0345] Users choose their mode of transportation and accommodation within the simulated world.

[0346] The selection information is sent to the server, which then makes the reservation in the real world through the reservation system (e.g., Booking.com API).

[0347] Specific example

[0348] Specific examples of product purchases

[0349] Users choose a smartphone in an electronics store within a simulated world.

[0350] When the user clicks the purchase button, the server processes the purchase through the e-commerce system, and the smartphone is delivered to the user's address.

[0351] Examples of transportation options

[0352] The user chooses a taxi within a simulated world.

[0353] The server makes real-world taxi reservations through a taxi reservation system.

[0354] Example of a prompt

[0355] Prompt message regarding product purchase

[0356] A user wants to purchase a smartphone within a virtual world. Please explain the steps involved in completing the purchase and having it delivered to their real-world address.

[0357] Prompt message regarding the selection of mode of transport

[0358] A user wants to book a taxi within a virtual world. Please explain the steps involved in completing the booking process and finalizing the taxi reservation in the real world.

[0359] This system allows users to have their actions in the simulated world reflected quickly and efficiently in the real world. The server receives selection information from the user and performs procedures in the real world through the appropriate system. This significantly improves user convenience. The flow of a specific process in Example 3 will be explained using Figure 15.

[0360] Step 1:

[0361] The user visits a store in a virtual world using their device. The user selects an item and clicks a select button. The input is the user's selected item information, and the output is the transmission of that information to the server. Specifically, the user selects a smartphone in an electronics store in the virtual world.

[0362] Step 2:

[0363] The server receives product selection information sent by the user. The input is the product information sent by the user, and the output is that this information is saved in the database. Specifically, the server receives the selection information for "smartphone" and saves it in the database.

[0364] Step 3:

[0365] The user clicks the "Confirm Purchase" button on the purchase confirmation screen. The input is the user's purchase confirmation action, and the output is the transmission of that information to the server. Specifically, the user clicks the "Confirm Purchase" button on the purchase confirmation screen.

[0366] Step 4:

[0367] The server connects to the e-commerce system and initiates the purchase process. The input is the user's confirmed purchase information, and the output is the start of the purchase process to the e-commerce system. Specifically, the server connects to the e-commerce system and performs the purchase process on the smartphone.

[0368] Step 5:

[0369] The server sends the user's payment information to the e-commerce system and completes the purchase. The input is the user's payment information, and the output is a confirmation of the completed purchase. Specifically, the server sends the user's payment information to the e-commerce system and completes the purchase.

[0370] Step 6:

[0371] The server sends purchase information to the delivery system. The input is purchase completion information, and the output is the transmission of information to the delivery system. Specifically, the server sends smartphone delivery information to the delivery system.

[0372] Step 7:

[0373] The delivery system delivers the product to the user's address. The input is delivery information, and the output is the product being delivered to the user's address. Specifically, the delivery system delivers a smartphone to the user's address.

[0374] Step 8:

[0375] The user uses a device to select transportation and accommodation within a simulated world. The input is the user's selection information, and the output is the transmission of that information to the server. As a concrete example, the user selects a taxi within the simulated world.

[0376] Step 9:

[0377] The server receives selection information sent by the user. The input is the selection information sent by the user, and the output is that this information is saved in the database. Specifically, the server receives the selection information for "taxi" and saves it in the database.

[0378] Step 10:

[0379] The server connects to the reservation system and initiates the reservation process. The input is the user's selection information, and the output is the start of the reservation process in the reservation system. Specifically, the server connects to the reservation system and performs the taxi reservation process.

[0380] Step 11:

[0381] The server sends the reservation information to the reservation system, completing the reservation in the real world. The input is the reservation information, and the output is a confirmation of the reservation completion. Specifically, the server sends the reservation information to the reservation system, completing the taxi reservation in the real world.

[0382] (Application Example 3)

[0383] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0384] There is a need for a system that allows users to seamlessly perform purchases and reservations in the virtual world, just as they would in the real world. In conventional systems, the virtual world experience and real-world actions are disconnected, requiring separate manual actions for users to utilize products or services selected in the virtual world in the real world. This resulted in a diminished user experience and reduced convenience.

[0385] In Application Example 3, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a virtual world using a digital twin, means for experiencing the virtual world in VR, means for generating humans in the virtual world using AI and enabling conversation with the generative AI, means for realizing purchases in the virtual world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for selecting a product in the virtual world and completing a purchase in the real world by pressing a purchase button, and means for selecting transportation and accommodation in the virtual world and completing a reservation in the real world by pressing a reservation button. This makes it possible for purchases and reservations made by the user in the virtual world to be seamlessly reflected in the real world.

[0386] A "digital twin" is a digital reproduction of physical objects and environments from the real world.

[0387] A "pseudo-world" is a virtual space constructed using virtual reality or augmented reality.

[0388] "VR" is an abbreviation for virtual reality, a technology that allows users to immerse themselves in and experience a virtual space.

[0389] "AI" is an abbreviation for artificial intelligence, which is a computer system that mimics human intelligence.

[0390] "Generative AI" refers to artificial intelligence that generates human-like characters in a virtual space and interacts with users.

[0391] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[0392] "Means of transportation" refers to means of getting around in the real world, including airplanes, trains, and taxis.

[0393] "Accommodation facilities" refer to facilities such as hotels and inns where users stay overnight.

[0394] "Information provision fees" are a portion of the fees that users pay when using a service, and are paid as compensation for the provision of information.

[0395] "A method in which a product is selected in a virtual world and a purchase is completed in the real world by pressing a purchase button" refers to a system where a user selects a product in a virtual space and presses a purchase button, and that product is then purchased in the real world.

[0396] "A method in which users select transportation and accommodation within a virtual world and complete reservations in the real world by pressing a reservation button" refers to a system where users select transportation and accommodation within a virtual space and complete the reservation in the real world by pressing a reservation button.

[0397] The system for carrying out this invention generates a simulated world using a digital twin, allowing users to experience it in virtual reality (VR). The system includes means for realizing purchase and reservation activities in the real world that occur within the virtual world.

[0398] Hardware and software to use

[0399] Hardware: Smartphones, head-mounted displays (HMDs)

[0400] Software: Unity (game engine), Python (backend processing), Firebase (database), Stripe (electronic payment)

[0401] Data processing and data calculation

[0402] Building a virtual world

[0403] The server uses Unity to build a virtual world, allowing users to move around freely. Within the virtual world, 3D models of goods, transportation, and accommodations are placed.

[0404] Product selection and purchase

[0405] When a user selects a product, its details are displayed within Unity. When the purchase button is pressed, the Python backend accesses the Firebase database to record the purchase information. Stripe is used to process the electronic payment, completing the purchase.

[0406] Booking transportation and accommodation

[0407] When a user selects transportation and accommodation, the Python backend similarly accesses the Firebase database to record the booking information. This booking information is then reflected in the real-world booking system.

[0408] Specific example

[0409] Product Purchase: When a user selects a smartphone in a store within the virtual world and presses the purchase button, the smartphone is delivered to them in the real world.

[0410] Accommodation booking: When a user selects a hotel in the virtual world and presses the book button, the reservation for that hotel is completed in the real world.

[0411] Example of a prompt

[0412] Product Purchase: "Develop a system that allows users to select products within a virtual world, purchase them in the real world, and have them delivered."

[0413] Accommodation Booking: "Please create a system that allows users to book accommodations selected in a virtual world in the real world."

[0414] In this way, it is possible to realize a system in which actions within the virtual world are seamlessly reflected in the real world.

[0415] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[0416] Step 1:

[0417] The server uses Unity to build a virtual world. It receives data on real-world physical objects and environments as input, generating a digital twin. As output, it provides a virtual world that users can freely navigate.

[0418] Step 2:

[0419] The user wears a head-mounted display (HMD) to access the virtual world. Position and movement information from the HMD are received as input, and the viewpoint and movement within the virtual world are reflected in real time. The output is an immersive virtual experience for the user.

[0420] Step 3:

[0421] The user selects an item within the virtual world. The system receives the user's selection as input and displays detailed information about the selected item. The system provides the user with the item details as output.

[0422] Step 4:

[0423] The user presses the purchase button. The system receives the purchase button press information as input and sends it to the Python backend. The system provides the purchase information to the Python backend as output.

[0424] Step 5:

[0425] The server uses a Python backend to access the Firebase database and record purchase information. It accepts purchase information as input and saves it to the Firebase database. As output, it provides the database containing the recorded purchase information.

[0426] Step 6:

[0427] The server processes electronic payments using Stripe. It receives purchase information and user payment information as input, and calls the Stripe API to complete the payment. As output, it provides confirmation that the payment is complete.

[0428] Step 7:

[0429] The user selects transportation and accommodation within the virtual world. The system receives the user's selection as input and displays detailed information about the selected transportation and accommodation. The system provides the user with detailed information about the transportation and accommodation as output.

[0430] Step 8:

[0431] The user presses the reservation button. The system receives the button press information as input and sends it to the Python backend. The system provides the reservation information to the Python backend as output.

[0432] Step 9:

[0433] The server uses a Python backend to access the Firebase database and record reservation information. It accepts reservation information as input and saves it to the Firebase database. As output, it provides the database containing the recorded reservation information.

[0434] Step 10:

[0435] The server reflects reservation information in a real-world reservation system. It receives reservation information as input and sends it to the real-world reservation system. As output, it provides confirmation that the reservation has been completed.

[0436] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0437] "Example of form 1"

[0438] One embodiment of the present invention provides a system that combines an emotion engine. This system recognizes the user's emotions and adjusts the simulated world experience accordingly. Specifically, when the user is feeling happy, the system adjusts the experience within the simulated world to be more enjoyable. For example, it might change the weather in the simulated world to sunny or make AI-generated humans behave more friendly towards the user.

[0439] "Example of form 2"

[0440] Furthermore, the emotion engine adjusts the content of conversations with the AI-generated human based on the user's emotions. For example, when the user is sad, the AI-generated human offers words of comfort. When the user is angry, the AI-generated human offers an apology. This allows the user to experience appropriate responses to their emotions within the simulated world.

[0441] "Example of form 3"

[0442] Furthermore, the emotion engine recognizes the user's emotions and supports their purchasing behavior within the virtual world accordingly. For example, when a user is feeling happy, the system recommends products that amplify that happiness. Similarly, when a user is sad, the system recommends products that alleviate that sadness. This allows users to purchase appropriate products within the virtual world that match their emotions.

[0443] The following describes the processing flow for each example of the form.

[0444] "Example of form 1"

[0445] Step 1: The user logs into the virtual world through a VR device.

[0446] Step 2: The emotion engine recognizes the user's emotions. This emotion recognition is performed based on the user's facial expressions, tone of voice, word choices, and other factors.

[0447] Step 3: The system adjusts the simulated world experience based on the emotions it recognizes. For example, when the user is feeling happy, the system might change the weather in the simulated world to sunny or make the AI-generated human behave more friendly towards the user.

[0448] "Example of form 2"

[0449] Step 1: The user begins a conversation with an AI-generated human.

[0450] Step 2: The emotion engine recognizes the user's emotions and adjusts the content of the AI-generated human conversation accordingly. For example, if the user is sad, the AI-generated human will offer words of comfort. If the user is angry, the AI-generated human will offer an apology.

[0451] "Example of form 3"

[0452] Step 1: The user begins shopping within the simulated world.

[0453] Step 2: The emotion engine recognizes the user's emotions and adjusts the recommended products accordingly. For example, when a user is feeling happy, the system recommends products that amplify that happiness. Similarly, when a user is sad, the system recommends products that alleviate that sadness.

[0454] (Example 1)

[0455] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0456] Traditional virtual reality systems have problems such as difficulty in generating detailed digital twins of places users want to visit, and low user satisfaction due to the lack of experience adjustments based on user emotions. Furthermore, the insufficient linking of the virtual world to the real world and the ability to make purchases within the virtual world have resulted in a lack of user convenience.

[0457] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0458] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating characters within the simulated world using artificial intelligence and enabling conversation with the generated artificial intelligence, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience of the simulated world according to those emotions, and an emotion recognition engine for analyzing the user's emotions. This makes it possible to generate a detailed digital twin of a place the user wants to visit and adjust the experience according to the user's emotions. In addition, purchases within the simulated world and links to the real world are enhanced, improving user convenience.

[0459] A "digital twin" is a digital model that recreates physical locations and objects from the real world in a virtual space.

[0460] A "simulated world" is a virtual reality environment created using a digital twin.

[0461] "Virtual reality" is a technology that allows users to experience an artificial environment created using computer technology.

[0462] "Artificial intelligence" is a technology that allows computers to mimic human intelligence and act accordingly.

[0463] "E-commerce" refers to transactions involving the buying and selling of goods and services via the internet.

[0464] An "emotion recognition engine" is software that analyzes a user's emotions from their facial expressions, voice, and other data.

[0465] "Emotion-based experience adjustment" refers to modifying the virtual reality environment and interactions based on the user's emotional state.

[0466] "Information provision fees" are the fees paid by users for the information they provide.

[0467] "Means of transportation" refers to the means by which users travel in the real world.

[0468] An "accommodation facility" is a facility where users stay overnight.

[0469] Modes for carrying out the invention

[0470] This invention is a system that generates a simulated world using a digital twin, allowing users to experience that simulated world in virtual reality. Furthermore, it has a function to recognize the user's emotions and adjust the simulated world experience accordingly.

[0471] Digital Twin Generation

[0472] The server collects data on the location the user wants to visit. This data includes streetscapes, buildings, and scenery. Specifically, it uses the Google Maps API and OpenStreetMap API to obtain 3D models of the specified location and surrounding scenery data. The server then uses Unity or Unreal Engine to generate a digital twin based on this data.

[0473] Specific example: If a user requests to "visit the Eiffel Tower in Paris," the server uses the Google Maps API to collect a 3D model of the Eiffel Tower and surrounding landscape data, and then uses Unity to generate a digital twin of the Eiffel Tower.

[0474] Example of a prompt:

[0475] A user wants to visit the Eiffel Tower in Paris. Collect a 3D model of the Eiffel Tower and surrounding landscape data, and use Unity to generate a simulated world.

[0476] Experiences in virtual reality

[0477] The device provides the user with a generated digital twin. The user can experience the simulated world through a VR headset or monitor. The device tracks the user's movements and gaze, updating the simulated world in real time.

[0478] Conversation by artificial intelligence

[0479] The server uses artificial intelligence to generate characters within a simulated world, enabling conversations with the user. This utilizes natural language processing technology. For example, if a user asks for directions within the simulated world, the AI ​​will provide an appropriate response.

[0480] Realizing e-commerce

[0481] The server facilitates e-commerce transactions within a virtual world. Users can select products in a virtual store and complete the purchase process. Purchase data is securely managed by the server.

[0482] Emotional recognition and adjustment of experience

[0483] The device uses its camera and microphone to capture the user's facial expressions and voice. The server uses Microsoft® Azure® Emotion API and Google Cloud Vision API to analyze the user's emotions from the captured data. Based on the analyzed emotions, the server adjusts the simulated world experience. For example, if the user is feeling happy, the weather in the simulated world is set to sunny, and the AI-generated character greets the user with a smile.

[0484] Specific example: If the server detects that the user is enjoying themselves in the virtual world, it will set the weather to sunny and adjust the AI-generated character to greet the user with a smile.

[0485] Example of a prompt:

[0486] The system has recognized that the user is enjoying themselves within the virtual world. Please set the weather in the virtual world to sunny and adjust the AI-generated character so that it smiles and greets the user.

[0487] This system allows users to experience detailed digital twins of places they want to visit and enjoy emotionally personalized experiences. Furthermore, it enhances purchasing within the virtual world and strengthens the link to the real world, improving user convenience.

[0488] The flow of the specific processing in Example 1 will be explained using Figure 17.

[0489] Step 1:

[0490] The user enters the place they want to visit into the terminal. For example, they might enter "Eiffel Tower in Paris." The entered data is then sent to the server.

[0491] Input: Information about the place the user wants to visit

[0492] Output: Data of the desired location sent to the server

[0493] Step 2:

[0494] The server collects data from a specified location via the internet. Specifically, it uses the Google Maps API and OpenStreetMap API to obtain a 3D model of the Eiffel Tower and surrounding landscape data.

[0495] Input: Data of places the user wants to visit

[0496] Output: 3D models and landscape data of the collected locations.

[0497] Step 3:

[0498] The server generates a digital twin using Unity or Unreal Engine based on the collected data. Specifically, it imports a 3D model of the Eiffel Tower into Unity and places surrounding landscape data on it.

[0499] Input: 3D models and landscape data of the collected locations

[0500] Output: Digital twin generated in Unity or Unreal Engine

[0501] Step 4:

[0502] The server sends the generated digital twin to the terminal, allowing the user to experience the simulated world. The terminal displays the simulated world to the user through a VR headset or monitor.

[0503] Input: Digital twin generated by Unity or Unreal Engine

[0504] Output: Digital twin sent to the terminal

[0505] Step 5:

[0506] The device uses its camera and microphone to capture the user's facial expressions and voice. Specifically, it takes a picture of the user's face with the camera and records their voice with the microphone.

[0507] Input: User's facial expressions and voice

[0508] Output: Captured user facial expressions and voice data

[0509] Step 6:

[0510] The server uses Microsoft Azure's Emotion API and Google Cloud's Vision API to analyze the user's emotions from the captured data. For example, if the user is smiling, it recognizes this as "joy."

[0511] Input: Captured user facial expressions and voice data

[0512] Output: Analyzed user sentiment data

[0513] Step 7:

[0514] The server adjusts the simulated world experience based on the analyzed emotions. Specifically, if the user is feeling happy, it sets the weather in the simulated world to sunny and makes the AI-generated character greet the user with a smile.

[0515] Input: Analyzed user sentiment data

[0516] Output: Adjusted simulated world settings

[0517] Step 8:

[0518] The server sends a modified virtual world to the terminal, allowing the user to experience the changes. The terminal then displays the modified virtual world to the user through a VR headset or monitor.

[0519] Input: Settings for a modified simulated world

[0520] Output: The adjusted simulated world sent to the terminal.

[0521] (Application Example 1)

[0522] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0523] Traditional virtual store systems failed to adjust the user experience in response to their emotions, making it difficult to increase user satisfaction. Furthermore, providing an experience similar to that of a real-world store requires dynamically changing the environment and staff interactions in response to the user's emotions, but such a system did not exist.

[0524] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience of the simulated world according to those emotions, and means for changing the environment in the virtual store and the response of store clerks through such adjustments. This makes it possible to provide a dynamic experience that responds to the user's emotions.

[0525] A "digital twin" is a virtual model that recreates physical locations and objects from the real world in a digital space.

[0526] A "simulated world" is a virtual space created using digital twin technology, a virtual environment that users can experience.

[0527] "VR" is an abbreviation for virtual reality, and it is a technology that allows users to visually experience a virtual space.

[0528] "AI" is an abbreviation for artificial intelligence, which is a technology in which computers mimic human intelligence and act accordingly.

[0529] "Generative AI" refers to a system that uses artificial intelligence to generate humans and objects within a virtual space.

[0530] "EC" is an abbreviation for electronic commerce, which refers to the act of purchasing goods and services through the internet.

[0531] "Emotion recognition" is a technology that analyzes a user's facial expressions and behavior to identify their emotions.

[0532] A "virtual store" is a store that exists in a virtual space, a virtual commercial facility where users can browse and purchase products.

[0533] "Environmental adjustment" refers to dynamically changing environmental elements such as weather, music, and lighting within the virtual space in response to the user's emotions.

[0534] "Staff interaction" refers to the services and interactions provided to users by AI-generated human beings within a virtual space.

[0535] The embodiments for carrying out this invention will be described in detail below.

[0536] System Overview

[0537] The system of this invention generates a simulated world using digital twin technology and enables users to experience that simulated world in VR. Furthermore, it has the function of analyzing the user's emotions using emotion recognition technology and dynamically adjusting the experience within the simulated world according to those emotions.

[0538] Hardware and software to use

[0539] Hardware:

[0540] Smartphone or head-mounted display (HMD)

[0541] Camera (smartphone built-in camera or HMD camera)

[0542] software:

[0543] OpenCV (acquiring and processing camera images)

[0544] EmotionRecognizer (emotion recognition library)

[0545] VirtualWorld (Digital Twin Generation and Rendering Library)

[0546] System operation

[0547] Digital Twin Generation

[0548] The server retrieves data on the location the user wants to visit (cityscapes, buildings, scenery, etc.) and generates a digital twin. This data can be obtained from the internet. The generated digital twin is sent to the user's device and displayed in a VR environment.

[0549] emotion recognition

[0550] The user's device uses its camera to capture their facial expressions in real time, and EmotionRecognizer analyzes their emotions. The analyzed emotion data is then sent to a server.

[0551] Adjusting the experience

[0552] The server dynamically adjusts the environment within the simulated world (weather, music, lighting, etc.) and the staff's responses based on the received emotion data. For example, if the user is happy, the music in the store will be brightened and the staff will be set to be more friendly.

[0553] Specific example

[0554] Users access a virtual store using their smartphones and wander around inside a store that recreates the streets of Paris. When a user smiles, the music in the store brightens, and a store employee greets them in a friendly manner, saying, "Hello! What are you looking for today?"

[0555] Example of a prompt

[0556] Obtain data on places users want to visit and generate a digital twin. Recognize the user's emotions and adjust the virtual store experience accordingly. For example, if the user is happy, brighten the music in the store and make the staff more friendly.

[0557] In this way, the system of the present invention can provide a dynamic experience that responds to the user's emotions.

[0558] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[0559] Step 1:

[0560] The user selects a location they wish to visit. The user uses a smartphone or head-mounted display (HMD) to select the desired location (e.g., the streets of Paris). The input is information about the location selected by the user, and the output is a request to retrieve data for that location.

[0561] Step 2:

[0562] The server retrieves data for the selected location. The server obtains data such as streetscapes, buildings, and scenery from the selected location via the internet. The input is information about the location selected by the user, and the output is the retrieved location data.

[0563] Step 3:

[0564] The server generates a digital twin. The server generates a digital twin based on acquired location data. The input is the acquired location data, and the output is the generated digital twin.

[0565] Step 4:

[0566] The terminal displays the digital twin. The terminal receives the digital twin transmitted from the server and displays it in the VR environment. The input is the generated digital twin, and the output is the VR environment experienced by the user.

[0567] Step 5:

[0568] The device recognizes the user's emotions. The device uses a camera to capture the user's facial expressions in real time and analyzes the emotions using EmotionRecognizer. The input is the camera image, and the output is the analyzed emotion data.

[0569] Step 6:

[0570] The device sends emotion data to the server. The device sends the analyzed emotion data to the server. The input is the analyzed emotion data, and the output is the emotion data sent to the server.

[0571] Step 7:

[0572] The server adjusts the experience based on emotional data. Based on the received emotional data, the server dynamically adjusts the environment within the simulated world (weather, music, lighting, etc.) and the staff's responses. The input is emotional data, and the output is the adjusted environment settings.

[0573] Step 8:

[0574] The device displays the calibrated environment. The device receives the calibrated environment settings sent from the server and displays them in the VR environment. The input is the calibrated environment settings, and the output is the calibrated VR environment experienced by the user.

[0575] (Example 2)

[0576] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0577] Traditional virtual reality systems have faced challenges such as a fixed simulated world for users, making it difficult to provide interactive responses that respond to user emotions. Furthermore, conversations within the simulated world are often monotonous, making it difficult to respond appropriately to user emotions and thus failing to provide a realistic experience. Additionally, the insufficient link to the real world resulted in low user convenience.

[0578] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0579] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating humans within the simulated world using artificial intelligence and enabling conversation with the generative AI, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, and means for analyzing the user's emotions and adjusting the conversation content based on the analysis results. As a result, the user can experience a realistic simulated world and receive appropriate responses that match their emotions.

[0580] A "digital twin" is a virtual model that recreates physical objects and environments from the real world in a digital space.

[0581] A "simulated world" is a virtual environment created using virtual reality technology that users can experience.

[0582] "Virtual reality" is a technology that allows users to immerse themselves in a virtual environment created using computer technology.

[0583] "Artificial intelligence" is a technology in which computers mimic human intelligence and possess functions such as learning, reasoning, and recognition.

[0584] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new data or content based on specific inputs.

[0585] "E-commerce" refers to a form of transaction in which goods and services are bought and sold via the internet.

[0586] "Emotion analysis" is a technology that identifies and analyzes emotions from a user's voice tone, facial expressions, and other factors.

[0587] "Adjusting conversation content" is the process of appropriately modifying the content of conversations provided by generative artificial intelligence based on the results of user sentiment analysis.

[0588] "Means of transportation" refers to the means by which users travel in the real world.

[0589] An "accommodation facility" is a facility where users can stay temporarily.

[0590] This invention is a system in which a user experiences a simulated world using a virtual reality (VR) device and engages in interactive conversation with an artificial intelligence (AI)-generated human. Specific embodiments of this system are described below.

[0591] Hardware and software usage

[0592] Hardware: VR devices such as the Oculus Rift and HTC Vive will be used. This will allow users to experience a high level of immersion.

[0593] Software: Game engines such as Unity and Unreal Engine are used to generate the VR environment. OpenAI's GPT-3 is used for the generated AI model, and emotion recognition software is used for emotion analysis.

[0594] Data processing and data calculation

[0595] VR Device Usage: When a user puts on a VR device, the device uses Unity or Unreal Engine to generate a Parisian streetscape, providing a 360-degree view. This allows the user to experience a realistic cityscape.

[0596] Use of Generative AI Models: The server uses generative AI models such as OpenAI's GPT-3 to generate humans within the simulated world. This allows users to ask for directions and engage in everyday conversations.

[0597] Emotion Engine Usage: The server uses emotion recognition software to analyze the user's emotions in real time. Based on the analysis results, a generative AI model adjusts the conversation content. For example, it offers words of comfort when the user is sad and apologizes when the user is angry.

[0598] Specific example

[0599] The user puts on the VR device: The user puts on the Oculus Rift on their head and holds the controllers in their hands.

[0600] The device starts the VR environment: The device launches Unity, and the Parisian cityscape is displayed in 360 degrees.

[0601] The server generates a simulated world: The server uses GPT-3 to generate cafe staff and people walking by.

[0602] The user moves within the simulated world: The user operates a controller and walks down the Champs-Élysées.

[0603] A user converses with an AI-generated human: The user asks a cafe employee, "What do you recommend on the menu?"

[0604] The server analyzes the user's emotions: The server analyzes the tone of the user's voice and determines that the user is a little tired.

[0605] The server adjusts the conversation: The server sends a prompt to the generating AI model, such as, "You seem particularly tired today. I'd like to recommend a cafe where you can relax," and generates an appropriate conversation.

[0606] Example of a prompt

[0607] "When a user gets lost, please generate a conversation in which an AI-generated human provides appropriate directions."

[0608] "Generate a conversation where a user asks a cafe employee for a menu recommendation, and the employee provides an appropriate response."

[0609] This system allows users to experience the realistic streets of Paris from the comfort of their own homes and receive appropriate responses based on their emotions.

[0610] The flow of the specific processing in Example 2 will be explained using Figure 19.

[0611] Step 1:

[0612] The user puts on a VR device.

[0613] Input: The user puts on a VR device (e.g., Oculus Rift).

[0614] Output: The user is ready to enter the VR environment.

[0615] Specific actions: The user puts the VR device on their head and holds the controllers in their hands.

[0616] Step 2:

[0617] The device starts the VR environment.

[0618] Input: User wearing a VR device.

[0619] Output: The VR environment is displayed.

[0620] Specific action: The device launches Unity or Unreal Engine, and a 360-degree view of the Parisian cityscape is displayed.

[0621] Step 3:

[0622] The server generates a simulated world.

[0623] Input: VR environment activation signal from the device.

[0624] Output: Humans are generated within the simulated world.

[0625] Specific operation: The server uses OpenAI's GPT-3 to generate images of cafe staff and passersby.

[0626] Step 4:

[0627] The user moves around within the simulated world.

[0628] Input: User controller operation.

[0629] Output: The user's viewpoint shifts.

[0630] Specific action: The user operates the controller and walks down the Champs-Élysées.

[0631] Step 5:

[0632] Users converse with AI-generated human characters.

[0633] Input: User voice input.

[0634] Output: AI-generated human response.

[0635] Specific example: When a user asks a cafe employee, "What do you recommend?", an AI-generated human will reply, "Today's recommendations are croissants and cafe au lait."

[0636] Step 6:

[0637] The server analyzes the user's emotions.

[0638] Input: User voice data and facial expression data.

[0639] Output: User's emotional state.

[0640] Specific operation: The server uses emotion recognition software to analyze the user's voice tone and facial expressions, and determines that the user is slightly tired.

[0641] Step 7:

[0642] The server adjusts the conversation.

[0643] Input: User's emotional state.

[0644] Output: Edited conversation content.

[0645] Specific operation: The server sends a prompt message to the AI ​​model saying, "You seem particularly tired today. I'd like to recommend a cafe where you can relax," and then generates an appropriate conversation.

[0646] (Application Example 2)

[0647] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0648] Modern consumers enjoy the convenience of online shopping, but they also crave the face-to-face interaction and direct product viewing experience offered by physical stores. However, traditional online shopping has faced challenges, such as users not receiving sufficient information when choosing products and not being able to receive appropriate responses tailored to their emotions. Furthermore, providing a customized shopping experience that responds to users' emotions has been difficult.

[0649] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0650] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means including an emotion engine that analyzes the user's emotions and generates conversation content corresponding to those emotions, and means for selecting products while conversing with an AI-generated store clerk in a virtual store. As a result, the user can enjoy a realistic shopping experience in a virtual store and receive appropriate responses according to their emotions.

[0651] A "digital twin" is a virtual model that recreates physical objects and environments from the real world in a digital space.

[0652] A "simulated world" is a virtual space created using a digital twin, which users can experience through a VR device.

[0653] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space through sight and sound.

[0654] An "AI-generated human" is a human character created in a virtual space using artificial intelligence technology.

[0655] "Generative AI" refers to artificial intelligence systems designed to generate conversations and interactions with users.

[0656] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[0657] An "emotion engine" is a system that analyzes a user's emotions and generates appropriate responses and conversation content based on those emotions.

[0658] A "virtual store" is a store built in a virtual space that users can visit through VR devices.

[0659] An "AI-generated store clerk" is a store clerk character created using artificial intelligence technology to interact with users within a virtual store.

[0660] The system for implementing this invention is configured as follows: First, the user puts on a VR device and accesses a virtual store. A commercially available VR headset such as the Oculus Rift can be used as the VR device. When the user enters the virtual store, a simulated world generated using digital twin technology is displayed.

[0661] The server generates a simulated world using a digital twin, allowing users to experience it through VR devices. Within this simulated world, AI-generated humans (shop staff) are placed, enabling conversation with the user. The generative AI uses the OpenAI API to generate conversations with the user.

[0662] When a user selects products in a virtual store, an emotion engine analyzes the user's emotions and responds appropriately accordingly. The emotion engine analyzes the user's voice input and facial expression data, generating comforting words if the user is sad and apologies if the user is angry.

[0663] As a concrete example, consider a scenario where a user is looking for shoes in a virtual store. When the user asks, "Do you have this shoe in my size?", the emotion engine analyzes the user's emotions and generates an appropriate prompt. For example, if the user is sad, the prompt "Offer words of comfort when the user is sad" will be generated.

[0664] Example of a prompt:

[0665] "When a user asks for their shoe size, the system will suggest the appropriate size."

[0666] The server sends this prompt to the OpenAI API, which generates a conversation in which an AI-generated shop assistant suggests the appropriate size. The generated conversation is then displayed to the user through a VR device.

[0667] This system allows users to enjoy a realistic shopping experience within a virtual store and receive appropriate responses tailored to their emotions. This provides a new shopping experience that combines the convenience of online shopping with the face-to-face service of a physical store.

[0668] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[0669] Step 1:

[0670] Users wear VR devices and access virtual stores.

[0671] Input: User wearing a VR device and requesting access to a virtual store.

[0672] Output: Connection established to the virtual store

[0673] Specific operation: The user puts on a VR device and launches the virtual store application. The server receives the user's access request and establishes a connection to the virtual store.

[0674] Step 2:

[0675] The server uses a digital twin to generate a simulated world and displays it to the user.

[0676] Input: Digital twin data of the virtual store

[0677] Output: A simulated world displayed on the user's VR device.

[0678] Specific operation: The server uses digital twin technology to generate a simulated world of the virtual store and sends it to the user's VR device. The user then experiences the simulated world through the VR device.

[0679] Step 3:

[0680] When a user selects products in a virtual store, an emotion engine analyzes the user's emotions.

[0681] Input: User voice input and facial expression data

[0682] Output: User sentiment analysis results

[0683] Specific operation: When a user asks a question or makes a comment about a product, the VR device sends voice input and facial expression data to the server. The server uses an emotion engine to analyze this data and identify the user's emotions.

[0684] Step 4:

[0685] The server generates prompt messages that respond to emotions and sends them to the AI ​​model.

[0686] Input: User sentiment analysis results

[0687] Output: Generated prompt message

[0688] Specific operation: The server generates appropriate prompt messages based on the analysis results of the emotion engine. For example, if the user is sad, it will generate a prompt message that says, "When the user is sad, offer words of comfort."

[0689] Step 5:

[0690] The generative AI model generates conversation content based on the prompt text and sends it to the server.

[0691] Input: Generated prompt message

[0692] Output: Generated conversation content

[0693] Specific operation: The server sends the generated prompt text to the generation AI model. The generation AI model generates appropriate conversation content based on the prompt text and returns the result to the server.

[0694] Step 6:

[0695] The server sends the generated conversation content to the user's VR device and displays it.

[0696] Input: Generated conversation content

[0697] Output: Conversation content displayed on the user's VR device.

[0698] Specific operation: The server sends the conversation content received from the generated AI model to the user's VR device. The user then experiences a conversation with the AI-generated store clerk through the VR device.

[0699] Step 7:

[0700] When a user purchases an item within a virtual store, the server processes the purchase through the e-commerce system.

[0701] Input: User's purchase request

[0702] Output: Purchase completion notification

[0703] Specific operation: When a user selects a product and decides to purchase it, the server processes the purchase through the e-commerce system. Once the purchase is complete, the server sends a purchase completion notification to the user.

[0704] (Example 3)

[0705] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0706] There is a need for a system that can efficiently handle purchases and reservations within a simulated world linked to the real world. Furthermore, improving the user experience by recommending products that respond to user emotions is a challenge. Conventional systems have not considered user emotions in their product recommendations, making it difficult to increase user satisfaction.

[0707] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0708] In this invention, the server includes means for generating a simulated world using a digital twin; means for experiencing the simulated world in virtual reality; means for generating characters in the simulated world using artificial intelligence and enabling conversation with generative artificial intelligence; means for realizing purchases within the simulated world through e-commerce; means for linking with real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine that recognizes the user's emotions and recommends products according to those emotions; means for the emotion engine to transmit the user's emotion data to the server, for the server to select products based on the emotions; and means for transmitting the selected product information to the user's terminal, for the terminal to display the recommended products. This enables product recommendations that correspond to the user's emotions, thereby improving the user experience.

[0709] A "digital twin" is a technology that recreates physical objects and environments from the real world within a virtual space.

[0710] A "simulated world" is a virtual space created using digital twin technology, an environment that users can experience through virtual reality.

[0711] "Virtual reality" is a technology that allows users to experience a virtual space created using computer technology through their sight and hearing.

[0712] Artificial intelligence is a technology in which computer systems imitate human intelligence, learning, reasoning, and self-correcting.

[0713] "Generative artificial intelligence" refers to a system that uses artificial intelligence technology to generate people and objects in a virtual space, enabling dialogue and interaction with the user.

[0714] "E-commerce" refers to a form of transaction in which goods and services are purchased and sold via the internet.

[0715] "Transportation" refers to the means of transport used by a user to move from one place to another.

[0716] "Accommodation facilities" refer to facilities where users can stay temporarily, and include hotels, inns, and other similar establishments.

[0717] "Information provision fees" are a portion of the fees paid by users when using the system, and represent payment for the information and services provided.

[0718] An "emotion engine" is a system that recognizes a user's emotions and recommends appropriate products and services based on those emotions.

[0719] "Emotional data" refers to information about a user's emotions, obtained from their facial expressions, voice, and other sources.

[0720] A "server" is a computer system that stores, processes, and manages data on a network.

[0721] A "terminal" is a device that a user uses to access and operate a system, and includes personal computers and smartphones.

[0722] A "recommended product" is a product selected and suggested by the system based on the user's emotions and preferences.

[0723] This invention is a system in which users select and purchase products in a virtual world, and those products are then delivered to the real world. It also links to real-world transportation and accommodation, so that choices made in the virtual world are reflected in real-world reservations. Furthermore, it uses an emotion engine to recognize the user's emotions and recommend products that correspond to those emotions.

[0724] Hardware and software to be used

[0725] Servers: Database management systems (e.g., MySQL), web servers (e.g., Apache), emotion engines (e.g., IBM Watson®)

[0726] Device: User interface (e.g., web browser, mobile app)

[0727] Generative AI models: Natural language processing models (e.g., GPT-4)

[0728] Specific operation of the system

[0729] 1. Users select products within a simulated world.

[0730] The user accesses a store in a virtual world using their device. When the user selects an item and clicks the purchase button, the device sends information about the selected item to the server.

[0731] 2. Sending and saving product information

[0732] The terminal sends the information of the selected product to the server in JSON format. The server saves the received product information to a database (e.g., MySQL) and returns a confirmation message to the terminal when the saving is complete.

[0733] 3. Execution of delivery arrangements

[0734] The server interacts with the delivery system (e.g., FedEx API) to arrange delivery. The server receives delivery confirmation information from the delivery system and notifies the user's device of this information.

[0735] 4. Choosing transportation and accommodation

[0736] When a user selects transportation and accommodation within the virtual world and clicks the reservation button, the device sends the selected information to the server.

[0737] 5. Sending and saving reservation information

[0738] The device sends the selected reservation information to the server in JSON format. The server stores the received reservation information in its database and makes the reservation in conjunction with the reservation system (e.g., Booking.com API). The server then notifies the user's device of the reservation confirmation.

[0739] 6. Emotion recognition by an emotion engine

[0740] As the user explores the virtual world using the device, the device uses its camera and microphone to capture the user's facial expressions and voice. The device sends the captured data to an emotion engine (e.g., IBM Watson), which analyzes the user's emotions in real time and sends that data to a server.

[0741] 7. Emotion-based product recommendations

[0742] The server selects products that match the user's emotions based on emotional data. The server sends the selected product information to the terminal, and the terminal displays the recommended products to the user.

[0743] Specific examples and prompt statements

[0744] Specific example 1: Product purchase

[0745] The user selects a smartphone in an electronics store within a simulated world and clicks the purchase button.

[0746] The server receives information from smartphones and works with the delivery system to arrange deliveries in the real world.

[0747] The user will receive their smartphone in the real world a few days later.

[0748] Example 2: Hotel reservations

[0749] The user selects a hotel within a simulated world and clicks the reservation button.

[0750] The server receives hotel information and, in conjunction with the reservation system, makes reservations in the real world.

[0751] The user receives a hotel reservation confirmation in the real world.

[0752] Example of a prompt

[0753] "Please explain the procedure for delivering items selected by the user within a virtual world to the real world."

[0754] "Please explain the system for recommending products based on user emotions."

[0755] The above describes the embodiments for carrying out this invention. The flow of the specific processing in Example 3 will be explained with reference to Figure 21.

[0756] Step 1:

[0757] Users select products within a simulated world.

[0758] Input: The user uses a device to access a store in a virtual world and select an item.

[0759] Specific action: The user selects a product and clicks the purchase button.

[0760] Output: The terminal sends information about the selected product to the server in JSON format.

[0761] Step 2:

[0762] Sending and saving product information

[0763] Input: Product information in JSON format sent from the device.

[0764] Specific operation: The server analyzes the received product information and saves the "user ID, product ID, and purchase date and time" to a database (e.g., MySQL).

[0765] Output: The server returns a confirmation message to the terminal acknowledging that saving is complete.

[0766] Step 3:

[0767] Execute delivery arrangements

[0768] Input: Product information stored in the database.

[0769] Specific operation: The server interacts with the delivery system (e.g., FedEx API) and sends the user's address, product ID, and delivery date and time. The delivery system returns the delivery ID and estimated delivery date and time to the server.

[0770] Output: The server notifies the user's device of the "delivery ID and scheduled delivery date and time."

[0771] Step 4:

[0772] Selection of transportation and accommodation

[0773] Input: The user uses a device to select transportation and accommodation within the simulated world.

[0774] Specific operation: The user clicks the reservation button, and the device sends the selected information to the server in JSON format.

[0775] Output: The terminal sends the selected reservation information to the server.

[0776] Step 5:

[0777] Sending and saving reservation information

[0778] Input: Reservation information in JSON format sent from the terminal.

[0779] Specific operation: The server parses the received reservation information and stores the "User ID, Reservation ID, and Reservation Date and Time" in the database. The server interacts with the reservation system (e.g., Booking.com API) and sends the "User Name, Reservation ID, and Departure Date and Time". The reservation system returns the "Reservation Confirmation Number and Departure Date and Time" to the server.

[0780] Output: The server notifies the user's device of the "reservation confirmation number and departure date and time."

[0781] Step 6:

[0782] Emotion recognition by an emotion engine

[0783] Input: Facial expressions and audio data from when the user explores the simulated world using a device.

[0784] Specific operation: The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to an emotion engine (e.g., IBM Watson). The emotion engine analyzes the user's emotions in real time and sends the data to a server.

[0785] Output: The emotion engine sends the analysis results to the server.

[0786] Step 7:

[0787] Emotion-based product recommendations

[0788] Input: Emotional data sent from the emotion engine.

[0789] Specific operation: The server selects products that match the user's emotions based on emotion data. The server then sends the selected product information to the terminal.

[0790] Output: The terminal displays recommended products to the user.

[0791] (Application Example 3)

[0792] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0793] There is a need to optimize purchasing and reservation actions within a simulated world linked to the real world, in accordance with the user's emotions. Conventional systems provide goods and services without considering the user's emotions, resulting in a limited user experience and low satisfaction. Furthermore, systems that reflect choices made within the simulated world in real-world delivery and reservations are not yet sufficiently developed.

[0794] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions using an emotion engine and recommending products corresponding to those emotions, means for delivering the products selected in the simulated world to the real world, and means for booking transportation and accommodation selected in the simulated world to the real world. This makes it possible to provide optimal products and services according to the user's emotions, and realizes a system in which selections in the simulated world are reflected in delivery and reservations in the real world.

[0795] A "digital twin" is a digital reproduction of physical objects and environments from the real world.

[0796] A "simulated world" is a virtual environment created using a digital twin, which users can experience using virtual reality (VR) technology.

[0797] "VR" is an abbreviation for Virtual Reality, a technology that allows users to experience a computer-generated three-dimensional space.

[0798] "AI" is an abbreviation for Artificial Intelligence, which is a technology in which computers imitate human intelligence to learn and reason.

[0799] "Generative AI" is a technology that uses artificial intelligence to generate humans and objects within a simulated world, enabling interaction with the user.

[0800] "EC" is an abbreviation for Electronic Commerce, which is a system for buying and selling goods and services via the internet.

[0801] An "emotion engine" is a technology that recognizes a user's emotions and provides responses and recommendations accordingly.

[0802] "Means of transportation" refers to the means of transport that users use to move around in the real world, including, for example, cars, trains, and airplanes.

[0803] "Accommodation facilities" refer to facilities in the real world where users can stay overnight, and include, for example, hotels and inns.

[0804] "Information provision fees" are fees paid for information provided by users and are treated as part of the service usage fees.

[0805] "Delivery" refers to the act of delivering purchased goods to a location specified by the user.

[0806] "Reservation" refers to the act of a user securing a specific service or facility in advance.

[0807] The system for carrying out this invention generates a simulated world using a digital twin, and allows users to experience this simulated world using virtual reality (VR) technology. The system includes the following main components:

[0808] 1. Digital twin generation means:

[0809] The server generates a digital twin to recreate physical objects and environments from the real world in digital space. This uses 3D modeling software and sensor technology.

[0810] 2. VR experience methods:

[0811] Users will experience the generated virtual world using a head-mounted display (HMD) or smartphone. Specific hardware used will include the Oculus Rift and HTC Vive.

[0812] 3. AI generation means:

[0813] The server uses artificial intelligence (AI) to generate humans and objects within a simulated world, enabling interaction with the user. This is done using a generative AI model.

[0814] 4. Means of implementing e-commerce:

[0815] An e-commerce (EC) system is provided by a server, allowing users to select and purchase products within a virtual world. Purchased products are then delivered to the real world.

[0816] 5. Emotional Engine:

[0817] The server recognizes the user's emotions in real time and recommends products that correspond to those emotions. Emotion recognition uses OpenCV and TENSORFLOW®, which utilize cameras and microphones.

[0818] 6. Means of linking with the real world:

[0819] The modes of transportation and accommodations selected by the user within the virtual world will be reflected in their real-world reservations. This includes integration with the reservation system via an API.

[0820] 7. Information provision fee methods:

[0821] The server includes a mechanism for paying a portion of the usage fee as an information provision fee for the information provided by the user.

[0822] As a concrete example, consider a scenario where a user is searching for a new smartphone in a virtual store. If the user is happy, the system will recommend the latest smartphone model.

[0823] Examples of prompts to input into a generative AI model:

[0824] A user is searching for a new smartphone in a virtual store. Since the user is happy, please recommend the latest smartphone model.

[0825] This system enables the provision of optimal products and services tailored to the user's emotions, and choices made within the virtual world are reflected in real-world deliveries and reservations. This significantly improves the user experience.

[0826] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[0827] Step 1:

[0828] The server generates a digital twin to recreate physical objects and environments from the real world in digital space. It receives real-world data (e.g., 3D scan data or sensor data) as input and uses 3D modeling software to generate the digital twin. The output is a 3D model of the simulated world.

[0829] Step 2:

[0830] Users experience a generated virtual world using a head-mounted display (HMD) or smartphone. The input is a 3D model of the virtual world sent from a server, which is then displayed through a VR application. The output is the visual information of the virtual reality experienced by the user.

[0831] Step 3:

[0832] The server uses artificial intelligence (AI) to generate humans and objects within a simulated world, enabling interaction with the user. It receives user behavior data and simulated world situation data as input, and uses a generative AI model to generate appropriate humans and objects. The output is an interactive character or object within the simulated world.

[0833] Step 4:

[0834] Users utilize an e-commerce (EC) system to select and purchase products within a simulated world. The system receives information about the products selected by the user as input and processes the purchase through the EC system. Outputs include purchase confirmation information and shipping arrangement information.

[0835] Step 5:

[0836] The server uses an emotion engine to recognize the user's emotions in real time and recommends products that correspond to those emotions. It receives user facial and voice data acquired via camera and microphone as input, and analyzes those emotions using an emotion recognition algorithm. The output is product recommendations tailored to the user's emotions.

[0837] Step 6:

[0838] The mode of transportation and accommodation selected by the user within the virtual world are reflected in the real-world reservation. The system receives information on the user's selected mode of transportation and accommodation as input and connects to the reservation system via an API. The output is reservation confirmation information.

[0839] Step 7:

[0840] The server includes a mechanism for paying a portion of the usage fee as an information provision fee for the information provided by the user. It receives the user-provided information as input and calculates the usage fee. The output is payment confirmation information for the information provision fee.

[0841] In this way, through specific actions and data flows at each step, it becomes possible to provide optimal products and services that respond to the user's emotions, and a system is realized in which choices made in the simulated world are reflected in delivery and reservations in the real world.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0843] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0845] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0846] [Second Embodiment]

[0847] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0848] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0849] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0850] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0851] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0852] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0853] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0854] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0855] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0856] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0857] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0858] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0859] "Example of form 1"

[0860] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on a place the user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, the system acquires data on the streets, buildings, and scenery of Paris and generates a simulated world of Paris based on that data.

[0861] "Example of form 2"

[0862] The generated virtual world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their home. Furthermore, the people in the virtual world are generated by AI, and it is possible to converse with the generative AI. This allows the user to ask for directions and enjoy everyday conversations.

[0863] "Example of form 3"

[0864] Furthermore, purchases within the virtual world are implemented via e-commerce. When a user selects and purchases an item from a store within the virtual world, that item is delivered to the user in the real world. It is also linked to real-world transportation and accommodation; when a user selects transportation or accommodation within the virtual world, it is reflected in their real-world reservations.

[0865] The following describes the processing flow for each example of the form.

[0866] "Example of form 1"

[0867] Step 1: The user enters the place they want to visit into the system. For example, if the user wants to visit Paris, they would enter "Paris" into the system.

[0868] Step 2: The system retrieves data for the relevant location based on the entered information. This data includes information about the streetscape, buildings, and scenery.

[0869] Step 3: Based on the acquired data, a digital twin is generated. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[0870] "Example of form 2"

[0871] Step 1: The user puts on the VR device.

[0872] Step 2: Experience the generated virtual world through the VR device you're wearing. Users can experience walking through the streets of Paris from the comfort of their own homes.

[0873] Step 3: Humans in the simulated world are generated by AI, and it is possible to converse with the generative AI. Users can ask AI-generated humans for directions or enjoy everyday conversations.

[0874] "Example of form 3"

[0875] Step 1: The user selects and purchases products in a store within a simulated world.

[0876] Step 2: Purchased items are delivered to the user in the real world via the e-commerce system.

[0877] Step 3: When a user selects transportation and accommodation within the virtual world, these selections are reflected in their real-world reservations.

[0878] (Example 1)

[0879] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0880] Conventional digital twin technologies present challenges such as a complex process for users to virtually experience places they wish to visit, and a lack of easily accessible systems. Furthermore, there are insufficient means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This results in difficulties for users to engage in real-time virtual experiences.

[0881] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0882] In this invention, the server includes means for inputting a place the user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin. This makes it possible for the user to easily virtually experience a place they want to visit.

[0883] A "user" is an individual or group that uses the system to specify a place they want to visit and virtually experience it.

[0884] "Input" refers to the act or information that a user provides to the system regarding a place they wish to visit.

[0885] "Geographic data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layouts.

[0886] "Analysis" refers to the process of calculation and data processing to process collected geographic data and convert it into a 3D model.

[0887] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographical data.

[0888] A "digital twin" is a 3D model that recreates a specific location in the real world within a virtual space.

[0889] "Transmission" refers to the act of transferring the digital twin generated by the server to the user's terminal.

[0890] "Operation" refers to the act of a user interactively using a digital twin on a device.

[0891] This invention is a system for users to virtually experience places they wish to visit. The system includes a series of processes: the user inputs the place they wish to visit, geographic data about that place is collected and analyzed, converted into a 3D model, a digital twin is generated, and transmitted to the user's terminal.

[0892] Hardware and software to be used

[0893] server

[0894] The server is responsible for receiving user input, collecting and analyzing geographic data, and generating 3D models. Specifically, it uses the following software and services:

[0895] Google Maps API: Used to collect geographic data.

[0896] OpenStreetMap: Used to collect supplementary geographic data.

[0897] Python: Run a script to analyze the collected geographic data and convert it into a 3D model.

[0898] Unity: A game engine for generating 3D models based on analyzed data.

[0899] terminal

[0900] The terminal is a device that allows users to interact with a digital twin and virtually experience places they wish to visit. Specifically, it uses the following software:

[0901] Unity application: Displays 3D models sent from the server and allows the user to interact with them.

[0902] Data processing and data calculation

[0903] Data collection

[0904] The server collects geographic data from the Google Maps API and OpenStreetMap based on the location the user enters. For example, if the user enters "Paris," the server retrieves data such as the latitude and longitude of Paris, the shape of buildings, and the layout of roads.

[0905] Data Analysis

[0906] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and building shapes into polygon data. This process generates the basic data for the digital twin.

[0907] 3D model generation

[0908] The server uses Unity to generate 3D models based on the analyzed data. Specifically, it executes Unity's C scripts to create 3D models of Parisian streets and buildings. This completes a digital twin that allows users to virtually visit Paris.

[0909] Sending data

[0910] The server sends the generated 3D model to the user's device. The user can then launch the Unity application on their device and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[0911] Specific example

[0912] When a user enters "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographical data for Paris. Next, a Python script is executed to analyze the retrieved geographical data and convert it into a 3D model. Unity is used to generate a 3D model of Paris's streets and buildings based on the analyzed data. Finally, the generated 3D model is sent to the user's device, and the user can launch the Unity application on their device to virtually experience Paris.

[0913] Example of a prompt

[0914] "Please create a digital twin of Paris. Use the Google Maps API to obtain geographical data for Paris and generate a 3D model using Unity."

[0915] In this way, a system is built that allows users to virtually experience places they want to visit.

[0916] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0917] Step 1:

[0918] The user enters the place they want to visit.

[0919] Specifically, the user accesses the system's web interface, enters "Paris" into the search box, and clicks the search button.

[0920] Input: The location the user wants to visit (e.g., Paris)

[0921] Output: Information about the place you want to visit (e.g., Paris)

[0922] Step 2:

[0923] The server receives user input and collects geographical data.

[0924] Specifically, the server sends a request to the Google Maps API to retrieve geographical data for Paris. For example, it sends a request to a URL like https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[0925] Input: Information about the place you want to visit (e.g., Paris)

[0926] Output: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[0927] Step 3:

[0928] The server analyzes the geographic data it collects and converts it into a 3D model.

[0929] Specifically, the server executes a Python script to analyze the acquired geographic data. For example, it runs a script called parse_geodata.py to convert latitude and longitude information into 3D coordinates.

[0930] Input: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[0931] Output: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[0932] Step 4:

[0933] The server generates a 3D model based on the analyzed data.

[0934] Specifically, the server executes Unity C scripts to create 3D models of Parisian streets and buildings. For example, it executes a script called GenerateParisModel.cs.

[0935] Input: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[0936] Output: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[0937] Step 5:

[0938] The server generates a 3D model and sends it to the user's device.

[0939] Specifically, the server transfers the generated 3D model to the user's terminal.

[0940] Input: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[0941] Output: 3D model sent to the user's terminal

[0942] Step 6:

[0943] Users interact with the digital twin on their devices.

[0944] In terms of specific actions, the user launches a Unity application on their device and interacts with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[0945] Input: 3D model sent to the user's device

[0946] Output: User-operated digital twin

[0947] (Application Example 1)

[0948] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0949] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping struggles to replicate the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to experience detailed information and the atmosphere of places they want to visit beforehand. This often leads to anxiety for consumers who cannot verify product or location details before purchasing. Additionally, the lack of ways to link online shopping with real-world transportation and accommodation can result in a lack of consistency in travel planning.

[0950] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0951] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for acquiring data on places the user wants to visit and generating a digital twin based on that data, means for providing a virtual shopping tour based on the digital twin, and means for experiencing the virtual shopping tour through a smartphone or head-mounted display. This makes it possible for users to experience detailed information and the atmosphere of places they want to visit in advance, thereby reducing anxiety when online shopping or planning trips.

[0952] A "digital twin" is a virtual representation of physical locations and objects in the real world, created as digital data.

[0953] A "pseudo-world" refers to a virtual environment or space created using a digital twin.

[0954] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[0955] "AI" is an abbreviation for artificial intelligence, which is a technology in which computers imitate human intelligence to learn and reason.

[0956] "Generative AI" is a technology that uses artificial intelligence to generate humans and objects in a virtual space.

[0957] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the internet.

[0958] "Means of transportation" refers to the means or methods of transport that a user uses to move physically.

[0959] "Accommodation facilities" refer to facilities or places where users can stay temporarily.

[0960] "Information provision fee" refers to the fee that a user pays for the information provided.

[0961] A "virtual shopping tour" is a tour that allows users to experience shopping in a virtual space created using a digital twin.

[0962] A "smartphone" is a type of mobile phone, a multi-functional device capable of internet access and application use.

[0963] A "head-mounted display" is a display device worn on the head and used to experience virtual reality and augmented reality.

[0964] The system for carrying out this invention generates a simulated world using a digital twin, allowing users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[0965] System Configuration

[0966] The system mainly consists of the following components:

[0967] 1. Server: Generates a digital twin and provides data in response to user requests.

[0968] 2. Device: A smartphone or head-mounted display used by the user.

[0969] 3. User: Experience a virtual shopping tour using the system.

[0970] Program processing

[0971] The server implements the system using the following methods.

[0972] Digital Twin Generation Method: Data on a location the user wishes to visit is acquired, and a digital twin is generated based on that data. Specifically, location data is acquired via an API, and a virtual space is generated using a 3D rendering library.

[0973] VR Experience Method: The generated digital twin can be experienced in VR. Users can use smartphones or head-mounted displays to freely move around and experience the virtual space.

[0974] AI generation method: An AI generates human characters in a virtual space, enabling conversations with the user. The generated AI model is used to achieve natural, real-time conversations.

[0975] E-commerce implementation method: Purchases within a virtual space are realized through electronic commerce. Users can select products in a virtual store and complete the purchase process.

[0976] Linking mechanism: Links to real-world transportation and accommodation. Users can make reservations and arrangements within the virtual space.

[0977] Information provision fee payment method: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[0978] Hardware and software to be used

[0979] Hardware: Smartphones, head-mounted displays

[0980] Software: Python, 3D rendering library (e.g., some_3d_rendering_library), generative AI model, API

[0981] Specific example

[0982] If a user wants to shop on the Champs-Élysées in Paris, the system will operate according to the following steps.

[0983] 1. Users access the system using a smartphone or head-mounted display.

[0984] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[0985] 3. Users experience the generated digital twin in VR and move freely within the virtual space.

[0986] 4. Interact with AI-generated humans in a virtual space and receive directions and product descriptions.

[0987] 5. Users select products in a virtual store and complete the purchase process through e-commerce.

[0988] 6. Book real-world transportation and accommodation as needed.

[0989] Example of a prompt

[0990] Please enter the name of the city you would like to visit. For example, if you enter "Paris," a digital twin of the Champs-Élysées in Paris will be generated, allowing you to start a virtual shopping tour.

[0991] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[0992] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0993] Step 1:

[0994] The user accesses the system using a smartphone or head-mounted display. The user enters the name of the place they want to visit. The entered place name is sent to the server.

[0995] Step 2:

[0996] The server retrieves data for a location based on the entered location name. Specifically, it requests location data via an API and parses the retrieved data. The input is the location name, and the output is detailed location data.

[0997] Step 3:

[0998] The server generates a digital twin based on the acquired data. Using a 3D rendering library, it recreates detailed location data as a virtual space. The input is detailed location data, and the output is the digital twin.

[0999] Step 4:

[1000] The server sends the generated digital twin to the user's device. The user then experiences the digital twin in VR through a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[1001] Step 5:

[1002] The server generates human characters in a virtual space using a generative AI model. The generated AI enables conversation with the user. The input is the generative AI model and data from the virtual space, and the output is the AI-generated human character.

[1003] Step 6:

[1004] Users converse with AI-generated humans in a virtual space, receiving directions and product descriptions. Input consists of user questions and requests, while output is the AI-generated human's response.

[1005] Step 7:

[1006] Users select products in a virtual store and proceed with the purchase. The server facilitates the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[1007] Step 8:

[1008] Users book real-world transportation and accommodations as needed. The server processes this booking information and provides it to the user. The input is the user's booking information, and the output is a booking confirmation.

[1009] Step 9:

[1010] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is a confirmation of fee collection.

[1011] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[1012] (Example 2)

[1013] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1014] Traditional virtual reality systems have limitations, such as the restricted virtual world experienced by users, making natural real-time conversation and interaction difficult. Furthermore, the lack of sufficient linkage between the virtual world and the real world, including inadequate purchasing capabilities, limits the user experience. Additionally, there is a lack of effective means to utilize user voice to facilitate natural conversations with generative artificial intelligence.

[1015] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1016] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating humans in the simulated world with artificial intelligence and enabling conversation with the generative AI, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for converting the user's voice into text and sending it to the generative AI as a prompt, and means for converting the response generated by the generative AI into voice and returning it to the user. As a result, the user can enjoy natural conversations in real time, interaction within the simulated world is improved, and the link with the real world is strengthened.

[1017] A "digital twin" is a virtual model that digitally reproduces physical objects and environments in the real world.

[1018] A "virtual world" is a virtual environment that users experience through a virtual reality device, and which is different from the real world.

[1019] A "virtual reality device" is a device that allows users to experience a virtual environment visually and aurally.

[1020] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[1021] "Generative artificial intelligence" refers to artificial intelligence models that generate natural-sounding responses based on user input.

[1022] "E-commerce" refers to transactions involving the purchase and sale of goods and services via the internet.

[1023] A "prompt message" is a text-based instruction sent to a generative artificial intelligence system based on user input.

[1024] "Speech recognition software" is software used to convert speech into text.

[1025] "Speech synthesis software" is software used to convert text into speech.

[1026] This invention is a system that generates a simulated world experienced by the user through a virtual reality device and enables natural conversation with generative artificial intelligence. Specific embodiments of this system are described below.

[1027] First, the server generates a virtual world using a digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then delivered to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[1028] When a user puts on a virtual reality device, the device connects to a server and receives data from the virtual world. Based on the received data, the device renders the virtual world on the virtual reality device's display in real time. This allows the user to experience as if they were actually walking around in that place.

[1029] Humans within the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the audio with a microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is then sent to the generative artificial intelligence as a prompt.

[1030] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates an appropriate response based on the received prompt. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI will generate the response, "The Eiffel Tower is located straight down this road, and you'll turn right at the next intersection."

[1031] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then returned to the user through the virtual reality device's speakers. This allows the user to hear responses from people within the simulated world.

[1032] As a concrete example, suppose a user is walking around Paris wearing a virtual reality device. If the user asks a person in the virtual world, "Where is the Eiffel Tower?" near the Eiffel Tower, the following prompt will be sent to the generative artificial intelligence:

[1033] Example of a prompt:

[1034] User: "Where is the Eiffel Tower?"

[1035] The generative artificial intelligence receives this prompt and generates an appropriate response. The terminal converts this response into speech and returns it to the user. This allows the user to enjoy a natural conversation in real time.

[1036] This system enhances user interaction within the virtual world and strengthens its link to the real world. For example, it enables e-commerce transactions within the virtual world and links to real-world transportation and accommodation. Furthermore, it includes a method for paying a portion of the usage fee as an information provision fee, further enriching the user experience.

[1037] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1038] Step 1:

[1039] The user puts on a virtual reality device.

[1040] Input: The action of a user putting on a virtual reality device.

[1041] Output: The virtual reality device is activated, and the user's sight and hearing are switched to the virtual reality device.

[1042] Specific operation: The user puts the virtual reality device on their head and turns on the device. This activates the device, and the user's sight and hearing switch to information provided through the virtual reality device.

[1043] Step 2:

[1044] The device connects to the server and receives data from the simulated world.

[1045] Input: Activation signal for a virtual reality device.

[1046] Output: Data from the simulated world is sent from the server to the terminal.

[1047] Specific operation: The device connects to the server via the internet, and the server sends data of a simulated world generated by Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[1048] Step 3:

[1049] The device renders a simulated world and displays it to the user.

[1050] Input: Data from a simulated world received from the server.

[1051] Output: A simulated world is displayed on the virtual reality device's screen.

[1052] Specific operation: Based on the received data, the device renders the streets of Paris in real time on the virtual reality device's display. Through the virtual reality device, the user experiences a visual sensation as if they were actually walking through the streets of Paris.

[1053] Step 4:

[1054] The user acts within a simulated world and speaks to people within that world.

[1055] Input: User voice input.

[1056] Output: The user's voice is captured on the device.

[1057] Specific actions: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[1058] Step 5:

[1059] The device converts the user's voice into text and sends it as a prompt to the generative AI model.

[1060] Input: User's voice data.

[1061] Output: A text-formatted prompt message.

[1062] Specific operation: The device captures the user's voice using the microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the AI ​​model as a prompt.

[1063] Step 6:

[1064] The server generates a response using a generated AI model.

[1065] Input: A text-based prompt message.

[1066] Output: The generated response text.

[1067] Specific operation: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt. For example, it might generate a response like, "Go straight down this road and turn right at the next intersection."

[1068] Step 7:

[1069] The device converts the generated response into speech and returns it to the user.

[1070] Input: The generated response text.

[1071] Output: A voice response.

[1072] Specific operation: The terminal converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is returned to the user through the virtual reality device's speaker. The user can hear the response from a person in the simulated world.

[1073] (Application Example 2)

[1074] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1075] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has the problem of not being able to actually touch and examine products. Furthermore, it's difficult to obtain detailed information about products through interaction with store staff when shopping online. In addition, there are limited ways for users to experience the streetscapes of foreign cities and enjoy the local atmosphere from the comfort of their homes. To solve these problems, a system is needed that provides a more realistic shopping experience and interactive communication with users.

[1076] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1077] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for tracking the user's location and initiating a conversation with an AI-generated character, means for processing user input and generating a response from the AI, and means for displaying the response. As a result, users can enjoy a realistic shopping experience from the comfort of their homes and obtain detailed information about products through interactive communication with store staff. They can also experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1078] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[1079] A "simulated world" is a virtual environment or space created using a digital twin.

[1080] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[1081] "AI" is an abbreviation for artificial intelligence, which is a computer system that mimics human intelligence.

[1082] "Generative AI" refers to artificial intelligence used for interacting with users and generating content.

[1083] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[1084] "Real-world transportation" refers to the means of transport that users use to travel in the real world.

[1085] An "accommodation facility" is a facility where users stay overnight.

[1086] "Information provision fees" are the fees paid by users for the information they provide.

[1087] "Tracking user location" means tracking the user's current location in real time.

[1088] An "AI-generated character" is a virtual person or character created by artificial intelligence.

[1089] "Processing user input" means analyzing user input data and generating an appropriate response.

[1090] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[1091] "Displaying the response" means visually presenting the generated response to the user.

[1092] To implement this invention, the following system configuration and program are required.

[1093] System Configuration

[1094] 1. Hardware:

[1095] VR devices (e.g., Oculus Rift, HTC Vive)

[1096] Smartphones (e.g., iPhone, Android)

[1097] Servers (those with high-performance computing capabilities)

[1098] 2. Software:

[1099] VR engines (e.g., Unity, Unreal Engine)

[1100] AI conversation engine (e.g., OpenAI GPT-3)

[1101] Program Processing Description

[1102] The server first generates a simulated world that recreates the streets of Paris using a digital twin. Next, the user can experience this simulated world by wearing a VR device. The user's position is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[1103] User input is sent to the server via a VR device or smartphone. The server analyzes this input and generates an appropriate response using an AI conversation engine. The generated response is displayed in the user's field of view.

[1104] Specific example

[1105] When a user enters a virtual store and asks an AI-generated shop assistant, "Do you have this dress in my size?", the server receives this input and uses its AI conversation engine to generate a response such as, "Yes, this dress is available in sizes S, M, and L." This response is then displayed in the user's field of view.

[1106] Example of a prompt

[1107] User: Do you have this dress in my size?

[1108] AI: Yes, this dress is available in sizes S, M, and L.

[1109] In this way, users can enjoy a realistic shopping experience from the comfort of their homes. They can also obtain detailed information about products through interactive communication with store staff. Furthermore, they can experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1110] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1111] Step 1:

[1112] The server generates a simulated world that recreates the streets of Paris using a digital twin.

[1113] Input: Data on the streets of Paris

[1114] Data processing: Using digital twin technology, the streetscape of Paris in the real world is recreated in a virtual space.

[1115] Output: A simulated world that recreates the streets of Paris.

[1116] Step 2:

[1117] Users wear VR devices and experience a simulated world transmitted from a server.

[1118] Input: Data from a simulated world sent from the server.

[1119] Data processing: VR devices track the user's viewpoint and movements in real time to display a simulated world.

[1120] Output: A simulated world displayed in the user's field of view.

[1121] Step 3:

[1122] The server tracks the user's location in real time.

[1123] Input: User's location information

[1124] Data calculation: Calculates the user's position in real time and updates their position within the simulated world.

[1125] Output: Updated user location information

[1126] Step 4:

[1127] When a user approaches an AI-generated character, the server initiates a conversation.

[1128] Input: User's location information, AI-generated character's location information

[1129] Data calculation: Calculates the distance between the user and the character, and initiates a conversation when they are within a certain distance.

[1130] Output: Trigger for starting a conversation

[1131] Step 5:

[1132] Users send input to the server via VR devices or smartphones.

[1133] Input: User voice input or text input

[1134] Data processing: Send input data to the server.

[1135] Output: User input data sent to the server

[1136] Step 6:

[1137] The server analyzes user input and generates appropriate responses using an AI conversation engine.

[1138] Input: User input data

[1139] Data processing: Use an AI conversation engine (e.g., OpenAI GPT-3) to generate appropriate responses to user input.

[1140] Output: Generated response data

[1141] Step 7:

[1142] The server displays the generated response in the user's field of view.

[1143] Input: Generated response data

[1144] Data processing: Send response data to the VR device and display it in the user's field of view.

[1145] Output: Response displayed in the user's field of view

[1146] In this way, users can enjoy a realistic shopping experience from the comfort of their homes. They can also obtain detailed information about products through interactive communication with store staff. Furthermore, they can experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1147] (Example 3)

[1148] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1149] There is a need for a system that can efficiently handle purchases and reservation procedures within a simulated world linked to the real world. Conventional systems have cumbersome processes for reflecting actions in the simulated world back into the real world, resulting in low user convenience. Furthermore, there is a need for a system that makes the experience within the simulated world more realistic and allows users to freely choose locations and times.

[1150] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating people in the simulated world using artificial intelligence and enabling conversation with generative artificial intelligence, means for realizing purchases in the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for performing delivery procedures in the real world when the user selects a product in the simulated world and confirms the purchase, and means for performing reservation procedures in the real world when the user selects transportation or accommodation in the simulated world. As a result, the user's actions in the simulated world are reflected quickly and efficiently in the real world.

[1151] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[1152] A "simulated world" is a virtual reality environment created using a digital twin.

[1153] "Virtual reality" is a technology that allows users to experience an artificial environment created using computer technology.

[1154] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[1155] "Generative artificial intelligence" is a system that uses artificial intelligence technology to generate characters in a simulated world and enables conversations with users.

[1156] "E-commerce" refers to a form of transaction in which goods and services are bought and sold via the internet.

[1157] "Means of transportation" refers to the means by which a user moves from one place to another.

[1158] An "accommodation facility" is a facility where users can stay temporarily.

[1159] "Information provision fees" are a portion of the fees that users pay when using the system, and are paid as compensation for the provision of information.

[1160] "Delivery procedures" refer to the process of delivering the products purchased by the user to the real world.

[1161] "Reservation process" refers to the procedure for users to book their chosen mode of transportation and accommodation in the real world.

[1162] Modes for carrying out the invention

[1163] This invention is a system that reflects user actions in a simulated world, such as purchasing goods or choosing transportation and accommodation, into the real world. Specific embodiments of this system are described below.

[1164] Hardware and software to be used

[1165] Servers: Database management systems (e.g., MySQL), Web servers (e.g., Apache)

[1166] Device: The PC or smartphone used by the user.

[1167] Generative AI models: Natural language processing models (e.g., GPT-4)

[1168] Data processing and data calculation

[1169] 1. Users select products within a simulated world.

[1170] Users visit stores within a virtual world using their devices.

[1171] When a user selects a product, that information is sent to the server.

[1172] The server saves the data of the selected product to the database.

[1173] 2. Execute the purchase procedure

[1174] Once the user confirms their purchase, the server processes the purchase through the e-commerce system (e.g., Shopify).

[1175] Purchase information is sent to a real-world delivery system, and the product is delivered to the user.

[1176] 3. Choosing transportation and accommodation

[1177] Users choose their mode of transportation and accommodation within the simulated world.

[1178] The selection information is sent to the server, which then makes the reservation in the real world through the reservation system (e.g., Booking.com API).

[1179] Specific example

[1180] Specific examples of product purchases

[1181] Users choose a smartphone in an electronics store within a simulated world.

[1182] When the user clicks the purchase button, the server processes the purchase through the e-commerce system, and the smartphone is delivered to the user's address.

[1183] Examples of transportation options

[1184] The user chooses a taxi within a simulated world.

[1185] The server makes real-world taxi reservations through a taxi reservation system.

[1186] Example of a prompt

[1187] Prompt message regarding product purchase

[1188] A user wants to purchase a smartphone within a virtual world. Please explain the steps involved in completing the purchase and having it delivered to their real-world address.

[1189] Prompt message regarding the selection of mode of transport

[1190] A user wants to book a taxi within a virtual world. Please explain the steps involved in completing the booking process and finalizing the taxi reservation in the real world.

[1191] This system allows users to have their actions in the simulated world reflected quickly and efficiently in the real world. The server receives selection information from the user and performs procedures in the real world through the appropriate system. This significantly improves user convenience. The flow of a specific process in Example 3 will be explained using Figure 15.

[1192] Step 1:

[1193] The user visits a store in a virtual world using their device. The user selects an item and clicks a select button. The input is the user's selected item information, and the output is the transmission of that information to the server. Specifically, the user selects a smartphone in an electronics store in the virtual world.

[1194] Step 2:

[1195] The server receives product selection information sent by the user. The input is the product information sent by the user, and the output is that this information is saved in the database. Specifically, the server receives the selection information for "smartphone" and saves it in the database.

[1196] Step 3:

[1197] The user clicks the "Confirm Purchase" button on the purchase confirmation screen. The input is the user's purchase confirmation action, and the output is the transmission of that information to the server. Specifically, the user clicks the "Confirm Purchase" button on the purchase confirmation screen.

[1198] Step 4:

[1199] The server connects to the e-commerce system and initiates the purchase process. The input is the user's confirmed purchase information, and the output is the start of the purchase process to the e-commerce system. Specifically, the server connects to the e-commerce system and performs the purchase process on the smartphone.

[1200] Step 5:

[1201] The server sends the user's payment information to the e-commerce system and completes the purchase. The input is the user's payment information, and the output is a confirmation of the completed purchase. Specifically, the server sends the user's payment information to the e-commerce system and completes the purchase.

[1202] Step 6:

[1203] The server sends purchase information to the delivery system. The input is purchase completion information, and the output is the transmission of information to the delivery system. Specifically, the server sends smartphone delivery information to the delivery system.

[1204] Step 7:

[1205] The delivery system delivers the product to the user's address. The input is delivery information, and the output is the product being delivered to the user's address. Specifically, the delivery system delivers a smartphone to the user's address.

[1206] Step 8:

[1207] The user uses a device to select transportation and accommodation within a simulated world. The input is the user's selection information, and the output is the transmission of that information to the server. As a concrete example, the user selects a taxi within the simulated world.

[1208] Step 9:

[1209] The server receives selection information sent by the user. The input is the selection information sent by the user, and the output is that this information is saved in the database. Specifically, the server receives the selection information for "taxi" and saves it in the database.

[1210] Step 10:

[1211] The server connects to the reservation system and initiates the reservation process. The input is the user's selection information, and the output is the start of the reservation process in the reservation system. Specifically, the server connects to the reservation system and performs the taxi reservation process.

[1212] Step 11:

[1213] The server sends the reservation information to the reservation system, completing the reservation in the real world. The input is the reservation information, and the output is a confirmation of the reservation completion. Specifically, the server sends the reservation information to the reservation system, completing the taxi reservation in the real world.

[1214] (Application Example 3)

[1215] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1216] There is a need for a system that allows users to seamlessly perform purchases and reservations in the virtual world, just as they would in the real world. In conventional systems, the virtual world experience and real-world actions are disconnected, requiring separate manual actions for users to utilize products or services selected in the virtual world in the real world. This resulted in a diminished user experience and reduced convenience.

[1217] In Application Example 3, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a virtual world using a digital twin, means for experiencing the virtual world in VR, means for generating humans in the virtual world using AI and enabling conversation with the generative AI, means for realizing purchases in the virtual world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for selecting a product in the virtual world and completing a purchase in the real world by pressing a purchase button, and means for selecting transportation and accommodation in the virtual world and completing a reservation in the real world by pressing a reservation button. This makes it possible for purchases and reservations made by the user in the virtual world to be seamlessly reflected in the real world.

[1218] A "digital twin" is a digital reproduction of physical objects and environments from the real world.

[1219] A "pseudo-world" is a virtual space constructed using virtual reality or augmented reality.

[1220] "VR" is an abbreviation for virtual reality, a technology that allows users to immerse themselves in and experience a virtual space.

[1221] "AI" is an abbreviation for artificial intelligence, which is a computer system that mimics human intelligence.

[1222] "Generative AI" refers to artificial intelligence that generates human-like characters in a virtual space and interacts with users.

[1223] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[1224] "Means of transportation" refers to means of getting around in the real world, including airplanes, trains, and taxis.

[1225] "Accommodation facilities" refer to facilities such as hotels and inns where users stay overnight.

[1226] "Information provision fees" are a portion of the fees that users pay when using a service, and are paid as compensation for the provision of information.

[1227] "A method in which a product is selected in a virtual world and a purchase is completed in the real world by pressing a purchase button" refers to a system where a user selects a product in a virtual space and presses a purchase button, and that product is then purchased in the real world.

[1228] "A method in which users select transportation and accommodation within a virtual world and complete reservations in the real world by pressing a reservation button" refers to a system where users select transportation and accommodation within a virtual space and complete the reservation in the real world by pressing a reservation button.

[1229] The system for carrying out this invention generates a simulated world using a digital twin, allowing users to experience it in virtual reality (VR). The system includes means for realizing purchase and reservation activities in the real world that occur within the virtual world.

[1230] Hardware and software to use

[1231] Hardware: Smartphones, head-mounted displays (HMDs)

[1232] Software: Unity (game engine), Python (backend processing), Firebase (database), Stripe (electronic payment)

[1233] Data processing and data calculation

[1234] Building a virtual world

[1235] The server uses Unity to build a virtual world, allowing users to move around freely. Within the virtual world, 3D models of goods, transportation, and accommodations are placed.

[1236] Product selection and purchase

[1237] When a user selects a product, its details are displayed within Unity. When the purchase button is pressed, the Python backend accesses the Firebase database to record the purchase information. Stripe is used to process the electronic payment, completing the purchase.

[1238] Booking transportation and accommodation

[1239] When a user selects transportation and accommodation, the Python backend similarly accesses the Firebase database to record the booking information. This booking information is then reflected in the real-world booking system.

[1240] Specific example

[1241] Product Purchase: When a user selects a smartphone in a store within the virtual world and presses the purchase button, the smartphone is delivered to them in the real world.

[1242] Accommodation booking: When a user selects a hotel in the virtual world and presses the book button, the reservation for that hotel is completed in the real world.

[1243] Example of a prompt

[1244] Product Purchase: "Develop a system that allows users to select products within a virtual world, purchase them in the real world, and have them delivered."

[1245] Accommodation Booking: "Please create a system that allows users to book accommodations selected in a virtual world in the real world."

[1246] In this way, it is possible to realize a system in which actions within the virtual world are seamlessly reflected in the real world.

[1247] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1248] Step 1:

[1249] The server uses Unity to build a virtual world. It receives data on real-world physical objects and environments as input, generating a digital twin. As output, it provides a virtual world that users can freely navigate.

[1250] Step 2:

[1251] The user wears a head-mounted display (HMD) to access the virtual world. Position and movement information from the HMD are received as input, and the viewpoint and movement within the virtual world are reflected in real time. The output is an immersive virtual experience for the user.

[1252] Step 3:

[1253] The user selects an item within the virtual world. The system receives the user's selection as input and displays detailed information about the selected item. The system provides the user with the item details as output.

[1254] Step 4:

[1255] The user presses the purchase button. The system receives the purchase button press information as input and sends it to the Python backend. The system provides the purchase information to the Python backend as output.

[1256] Step 5:

[1257] The server uses a Python backend to access the Firebase database and record purchase information. It accepts purchase information as input and saves it to the Firebase database. As output, it provides the database containing the recorded purchase information.

[1258] Step 6:

[1259] The server processes electronic payments using Stripe. It receives purchase information and user payment information as input, and calls the Stripe API to complete the payment. As output, it provides confirmation that the payment is complete.

[1260] Step 7:

[1261] The user selects transportation and accommodation within the virtual world. The system receives the user's selection as input and displays detailed information about the selected transportation and accommodation. The system provides the user with detailed information about the transportation and accommodation as output.

[1262] Step 8:

[1263] The user presses the reservation button. The system receives the button press information as input and sends it to the Python backend. The system provides the reservation information to the Python backend as output.

[1264] Step 9:

[1265] The server uses a Python backend to access the Firebase database and record reservation information. It accepts reservation information as input and saves it to the Firebase database. As output, it provides the database containing the recorded reservation information.

[1266] Step 10:

[1267] The server reflects reservation information in a real-world reservation system. It receives reservation information as input and sends it to the real-world reservation system. As output, it provides confirmation that the reservation has been completed.

[1268] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1269] "Example of form 1"

[1270] One embodiment of the present invention provides a system that combines an emotion engine. This system recognizes the user's emotions and adjusts the simulated world experience accordingly. Specifically, when the user is feeling happy, the system adjusts the experience within the simulated world to be more enjoyable. For example, it might change the weather in the simulated world to sunny or make AI-generated humans behave more friendly towards the user.

[1271] "Example of form 2"

[1272] Furthermore, the emotion engine adjusts the content of conversations with the AI-generated human based on the user's emotions. For example, when the user is sad, the AI-generated human offers words of comfort. When the user is angry, the AI-generated human offers an apology. This allows the user to experience appropriate responses to their emotions within the simulated world.

[1273] "Example of form 3"

[1274] Furthermore, the emotion engine recognizes the user's emotions and supports their purchasing behavior within the virtual world accordingly. For example, when a user is feeling happy, the system recommends products that amplify that happiness. Similarly, when a user is sad, the system recommends products that alleviate that sadness. This allows users to purchase appropriate products within the virtual world that match their emotions.

[1275] The following describes the processing flow for each example of the form.

[1276] "Example of form 1"

[1277] Step 1: The user logs into the virtual world through a VR device.

[1278] Step 2: The emotion engine recognizes the user's emotions. This emotion recognition is performed based on the user's facial expressions, tone of voice, word choices, and other factors.

[1279] Step 3: The system adjusts the simulated world experience based on the emotions it recognizes. For example, when the user is feeling happy, the system might change the weather in the simulated world to sunny or make the AI-generated human behave more friendly towards the user.

[1280] "Example of form 2"

[1281] Step 1: The user begins a conversation with an AI-generated human.

[1282] Step 2: The emotion engine recognizes the user's emotions and adjusts the content of the AI-generated human conversation accordingly. For example, if the user is sad, the AI-generated human will offer words of comfort. If the user is angry, the AI-generated human will offer an apology.

[1283] "Example of form 3"

[1284] Step 1: The user begins shopping within the simulated world.

[1285] Step 2: The emotion engine recognizes the user's emotions and adjusts the recommended products accordingly. For example, when a user is feeling happy, the system recommends products that amplify that happiness. Similarly, when a user is sad, the system recommends products that alleviate that sadness.

[1286] (Example 1)

[1287] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1288] Traditional virtual reality systems have problems such as difficulty in generating detailed digital twins of places users want to visit, and low user satisfaction due to the lack of experience adjustments based on user emotions. Furthermore, the insufficient linking of the virtual world to the real world and the ability to make purchases within the virtual world have resulted in a lack of user convenience.

[1289] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1290] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating characters within the simulated world using artificial intelligence and enabling conversation with the generated artificial intelligence, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience of the simulated world according to those emotions, and an emotion recognition engine for analyzing the user's emotions. This makes it possible to generate a detailed digital twin of a place the user wants to visit and adjust the experience according to the user's emotions. In addition, purchases within the simulated world and links to the real world are enhanced, improving user convenience.

[1291] A "digital twin" is a digital model that recreates physical locations and objects from the real world in a virtual space.

[1292] A "simulated world" is a virtual reality environment created using a digital twin.

[1293] "Virtual reality" is a technology that allows users to experience an artificial environment created using computer technology.

[1294] "Artificial intelligence" is a technology that allows computers to mimic human intelligence and act accordingly.

[1295] "E-commerce" refers to transactions involving the buying and selling of goods and services via the internet.

[1296] An "emotion recognition engine" is software that analyzes a user's emotions from their facial expressions, voice, and other data.

[1297] "Emotion-based experience adjustment" refers to modifying the virtual reality environment and interactions based on the user's emotional state.

[1298] "Information provision fees" are the fees paid by users for the information they provide.

[1299] "Means of transportation" refers to the means by which users travel in the real world.

[1300] An "accommodation facility" is a facility where users stay overnight.

[1301] Modes for carrying out the invention

[1302] This invention is a system that generates a simulated world using a digital twin, allowing users to experience that simulated world in virtual reality. Furthermore, it has a function to recognize the user's emotions and adjust the simulated world experience accordingly.

[1303] Digital Twin Generation

[1304] The server collects data on the location the user wants to visit. This data includes streetscapes, buildings, and scenery. Specifically, it uses the Google Maps API and OpenStreetMap API to obtain 3D models of the specified location and surrounding scenery data. The server then uses Unity or Unreal Engine to generate a digital twin based on this data.

[1305] Specific example: If a user requests to "visit the Eiffel Tower in Paris," the server uses the Google Maps API to collect a 3D model of the Eiffel Tower and surrounding landscape data, and then uses Unity to generate a digital twin of the Eiffel Tower.

[1306] Example of a prompt:

[1307] A user wants to visit the Eiffel Tower in Paris. Collect a 3D model of the Eiffel Tower and surrounding landscape data, and use Unity to generate a simulated world.

[1308] Experiences in virtual reality

[1309] The device provides the user with a generated digital twin. The user can experience the simulated world through a VR headset or monitor. The device tracks the user's movements and gaze, updating the simulated world in real time.

[1310] Conversation by artificial intelligence

[1311] The server uses artificial intelligence to generate characters within a simulated world, enabling conversations with the user. This utilizes natural language processing technology. For example, if a user asks for directions within the simulated world, the AI ​​will provide an appropriate response.

[1312] Realizing e-commerce

[1313] The server facilitates e-commerce transactions within a virtual world. Users can select products in a virtual store and complete the purchase process. Purchase data is securely managed by the server.

[1314] Emotional recognition and adjustment of experience

[1315] The device uses its camera and microphone to capture the user's facial expressions and voice. The server uses Microsoft Azure's Emotion API or Google Cloud's Vision API to analyze the user's emotions from the captured data. Based on the analyzed emotions, the server adjusts the simulated world experience. For example, if the user is feeling happy, the server might set the weather in the simulated world to sunny and have an AI-generated character greet the user with a smile.

[1316] Specific example: If the server detects that the user is enjoying themselves in the virtual world, it will set the weather to sunny and adjust the AI-generated character to greet the user with a smile.

[1317] Example of a prompt:

[1318] The system has recognized that the user is enjoying themselves within the virtual world. Please set the weather in the virtual world to sunny and adjust the AI-generated character so that it smiles and greets the user.

[1319] This system allows users to experience detailed digital twins of places they want to visit and enjoy emotionally personalized experiences. Furthermore, it enhances purchasing within the virtual world and strengthens the link to the real world, improving user convenience.

[1320] The flow of the specific processing in Example 1 will be explained using Figure 17.

[1321] Step 1:

[1322] The user enters the place they want to visit into the terminal. For example, they might enter "Eiffel Tower in Paris." The entered data is then sent to the server.

[1323] Input: Information about the place the user wants to visit

[1324] Output: Data of the desired location sent to the server

[1325] Step 2:

[1326] The server collects data from a specified location via the internet. Specifically, it uses the Google Maps API and OpenStreetMap API to obtain a 3D model of the Eiffel Tower and surrounding landscape data.

[1327] Input: Data of places the user wants to visit

[1328] Output: 3D models and landscape data of the collected locations.

[1329] Step 3:

[1330] The server generates a digital twin using Unity or Unreal Engine based on the collected data. Specifically, it imports a 3D model of the Eiffel Tower into Unity and places surrounding landscape data on it.

[1331] Input: 3D models and landscape data of the collected locations

[1332] Output: Digital twin generated in Unity or Unreal Engine

[1333] Step 4:

[1334] The server sends the generated digital twin to the terminal, allowing the user to experience the simulated world. The terminal displays the simulated world to the user through a VR headset or monitor.

[1335] Input: Digital twin generated by Unity or Unreal Engine

[1336] Output: Digital twin sent to the terminal

[1337] Step 5:

[1338] The device uses its camera and microphone to capture the user's facial expressions and voice. Specifically, it takes a picture of the user's face with the camera and records their voice with the microphone.

[1339] Input: User's facial expressions and voice

[1340] Output: Captured user facial expressions and voice data

[1341] Step 6:

[1342] The server uses Microsoft Azure's Emotion API and Google Cloud's Vision API to analyze the user's emotions from the captured data. For example, if the user is smiling, it recognizes this as "joy."

[1343] Input: Captured user facial expressions and voice data

[1344] Output: Analyzed user sentiment data

[1345] Step 7:

[1346] The server adjusts the simulated world experience based on the analyzed emotions. Specifically, if the user is feeling happy, it sets the weather in the simulated world to sunny and makes the AI-generated character greet the user with a smile.

[1347] Input: Analyzed user sentiment data

[1348] Output: Adjusted simulated world settings

[1349] Step 8:

[1350] The server sends a modified virtual world to the terminal, allowing the user to experience the changes. The terminal then displays the modified virtual world to the user through a VR headset or monitor.

[1351] Input: Settings for a modified simulated world

[1352] Output: The adjusted simulated world sent to the terminal.

[1353] (Application Example 1)

[1354] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1355] Traditional virtual store systems failed to adjust the user experience in response to their emotions, making it difficult to increase user satisfaction. Furthermore, providing an experience similar to that of a real-world store requires dynamically changing the environment and staff interactions in response to the user's emotions, but such a system did not exist.

[1356] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions and adjusting the experience of the simulated world according to those emotions, and means for changing the environment in the virtual store and the response of store clerks through such adjustments. This makes it possible to provide a dynamic experience that responds to the user's emotions.

[1357] A "digital twin" is a virtual model that recreates physical locations and objects from the real world in a digital space.

[1358] A "simulated world" is a virtual space created using digital twin technology, a virtual environment that users can experience.

[1359] "VR" is an abbreviation for virtual reality, and it is a technology that allows users to visually experience a virtual space.

[1360] "AI" is an abbreviation for artificial intelligence, which is a technology in which computers mimic human intelligence and act accordingly.

[1361] "Generative AI" refers to a system that uses artificial intelligence to generate humans and objects within a virtual space.

[1362] "EC" is an abbreviation for electronic commerce, which refers to the act of purchasing goods and services through the internet.

[1363] "Emotion recognition" is a technology that analyzes a user's facial expressions and behavior to identify their emotions.

[1364] A "virtual store" is a store that exists in a virtual space, a virtual commercial facility where users can browse and purchase products.

[1365] "Environmental adjustment" refers to dynamically changing environmental elements such as weather, music, and lighting within the virtual space in response to the user's emotions.

[1366] "Staff interaction" refers to the services and interactions provided to users by AI-generated human beings within a virtual space.

[1367] The embodiments for carrying out this invention will be described in detail below.

[1368] System Overview

[1369] The system of this invention generates a simulated world using digital twin technology and enables users to experience that simulated world in VR. Furthermore, it has the function of analyzing the user's emotions using emotion recognition technology and dynamically adjusting the experience within the simulated world according to those emotions.

[1370] Hardware and software to use

[1371] Hardware:

[1372] Smartphone or head-mounted display (HMD)

[1373] Camera (smartphone built-in camera or HMD camera)

[1374] software:

[1375] OpenCV (acquiring and processing camera images)

[1376] EmotionRecognizer (emotion recognition library)

[1377] VirtualWorld (Digital Twin Generation and Rendering Library)

[1378] System operation

[1379] Digital Twin Generation

[1380] The server retrieves data on the location the user wants to visit (cityscapes, buildings, scenery, etc.) and generates a digital twin. This data can be obtained from the internet. The generated digital twin is sent to the user's device and displayed in a VR environment.

[1381] emotion recognition

[1382] The user's device uses its camera to capture their facial expressions in real time, and EmotionRecognizer analyzes their emotions. The analyzed emotion data is then sent to a server.

[1383] Adjusting the experience

[1384] The server dynamically adjusts the environment within the simulated world (weather, music, lighting, etc.) and the staff's responses based on the received emotion data. For example, if the user is happy, the music in the store will be brightened and the staff will be set to be more friendly.

[1385] Specific example

[1386] Users access a virtual store using their smartphones and wander around inside a store that recreates the streets of Paris. When a user smiles, the music in the store brightens, and a store employee greets them in a friendly manner, saying, "Hello! What are you looking for today?"

[1387] Example of a prompt

[1388] Obtain data on places users want to visit and generate a digital twin. Recognize the user's emotions and adjust the virtual store experience accordingly. For example, if the user is happy, brighten the music in the store and make the staff more friendly.

[1389] In this way, the system of the present invention can provide a dynamic experience that responds to the user's emotions.

[1390] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[1391] Step 1:

[1392] The user selects a location they wish to visit. The user uses a smartphone or head-mounted display (HMD) to select the desired location (e.g., the streets of Paris). The input is information about the location selected by the user, and the output is a request to retrieve data for that location.

[1393] Step 2:

[1394] The server retrieves data for the selected location. The server obtains data such as streetscapes, buildings, and scenery from the selected location via the internet. The input is information about the location selected by the user, and the output is the retrieved location data.

[1395] Step 3:

[1396] The server generates a digital twin. The server generates a digital twin based on acquired location data. The input is the acquired location data, and the output is the generated digital twin.

[1397] Step 4:

[1398] The terminal displays the digital twin. The terminal receives the digital twin transmitted from the server and displays it in the VR environment. The input is the generated digital twin, and the output is the VR environment experienced by the user.

[1399] Step 5:

[1400] The device recognizes the user's emotions. The device uses a camera to capture the user's facial expressions in real time and analyzes the emotions using EmotionRecognizer. The input is the camera image, and the output is the analyzed emotion data.

[1401] Step 6:

[1402] The device sends emotion data to the server. The device sends the analyzed emotion data to the server. The input is the analyzed emotion data, and the output is the emotion data sent to the server.

[1403] Step 7:

[1404] The server adjusts the experience based on emotional data. Based on the received emotional data, the server dynamically adjusts the environment within the simulated world (weather, music, lighting, etc.) and the staff's responses. The input is emotional data, and the output is the adjusted environment settings.

[1405] Step 8:

[1406] The device displays the calibrated environment. The device receives the calibrated environment settings sent from the server and displays them in the VR environment. The input is the calibrated environment settings, and the output is the calibrated VR environment experienced by the user.

[1407] (Example 2)

[1408] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1409] Traditional virtual reality systems have faced challenges such as a fixed simulated world for users, making it difficult to provide interactive responses that respond to user emotions. Furthermore, conversations within the simulated world are often monotonous, making it difficult to respond appropriately to user emotions and thus failing to provide a realistic experience. Additionally, the insufficient link to the real world resulted in low user convenience.

[1410] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1411] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating humans within the simulated world using artificial intelligence and enabling conversation with the generative AI, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, and means for analyzing the user's emotions and adjusting the conversation content based on the analysis results. As a result, the user can experience a realistic simulated world and receive appropriate responses that match their emotions.

[1412] A "digital twin" is a virtual model that recreates physical objects and environments from the real world in a digital space.

[1413] A "simulated world" is a virtual environment created using virtual reality technology that users can experience.

[1414] "Virtual reality" is a technology that allows users to immerse themselves in a virtual environment created using computer technology.

[1415] "Artificial intelligence" is a technology in which computers mimic human intelligence and possess functions such as learning, reasoning, and recognition.

[1416] "Generative artificial intelligence" refers to artificial intelligence that has the ability to generate new data or content based on specific inputs.

[1417] "E-commerce" refers to a form of transaction in which goods and services are bought and sold via the internet.

[1418] "Emotion analysis" is a technology that identifies and analyzes emotions from a user's voice tone, facial expressions, and other factors.

[1419] "Adjusting conversation content" is the process of appropriately modifying the content of conversations provided by generative artificial intelligence based on the results of user sentiment analysis.

[1420] "Means of transportation" refers to the means by which users travel in the real world.

[1421] An "accommodation facility" is a facility where users can stay temporarily.

[1422] This invention is a system in which a user experiences a simulated world using a virtual reality (VR) device and engages in interactive conversation with an artificial intelligence (AI)-generated human. Specific embodiments of this system are described below.

[1423] Hardware and software usage

[1424] Hardware: VR devices such as the Oculus Rift and HTC Vive will be used. This will allow users to experience a high level of immersion.

[1425] Software: Game engines such as Unity and Unreal Engine are used to generate the VR environment. OpenAI's GPT-3 is used for the generated AI model, and emotion recognition software is used for emotion analysis.

[1426] Data processing and data calculation

[1427] VR Device Usage: When a user puts on a VR device, the device uses Unity or Unreal Engine to generate a Parisian streetscape, providing a 360-degree view. This allows the user to experience a realistic cityscape.

[1428] Use of Generative AI Models: The server uses generative AI models such as OpenAI's GPT-3 to generate humans within the simulated world. This allows users to ask for directions and engage in everyday conversations.

[1429] Emotion Engine Usage: The server uses emotion recognition software to analyze the user's emotions in real time. Based on the analysis results, a generative AI model adjusts the conversation content. For example, it offers words of comfort when the user is sad and apologizes when the user is angry.

[1430] Specific example

[1431] The user puts on the VR device: The user puts on the Oculus Rift on their head and holds the controllers in their hands.

[1432] The device starts the VR environment: The device launches Unity, and the Parisian cityscape is displayed in 360 degrees.

[1433] The server generates a simulated world: The server uses GPT-3 to generate cafe staff and people walking by.

[1434] The user moves within the simulated world: The user operates a controller and walks down the Champs-Élysées.

[1435] A user converses with an AI-generated human: The user asks a cafe employee, "What do you recommend on the menu?"

[1436] The server analyzes the user's emotions: The server analyzes the tone of the user's voice and determines that the user is a little tired.

[1437] The server adjusts the conversation: The server sends a prompt to the generating AI model, such as, "You seem particularly tired today. I'd like to recommend a cafe where you can relax," and generates an appropriate conversation.

[1438] Example of a prompt

[1439] "When a user gets lost, please generate a conversation in which an AI-generated human provides appropriate directions."

[1440] "Generate a conversation where a user asks a cafe employee for a menu recommendation, and the employee provides an appropriate response."

[1441] This system allows users to experience the realistic streets of Paris from the comfort of their own homes and receive appropriate responses based on their emotions.

[1442] The flow of the specific processing in Example 2 will be explained using Figure 19.

[1443] Step 1:

[1444] The user puts on a VR device.

[1445] Input: The user puts on a VR device (e.g., Oculus Rift).

[1446] Output: The user is ready to enter the VR environment.

[1447] Specific actions: The user puts the VR device on their head and holds the controllers in their hands.

[1448] Step 2:

[1449] The device starts the VR environment.

[1450] Input: User wearing a VR device.

[1451] Output: The VR environment is displayed.

[1452] Specific action: The device launches Unity or Unreal Engine, and a 360-degree view of the Parisian cityscape is displayed.

[1453] Step 3:

[1454] The server generates a simulated world.

[1455] Input: VR environment activation signal from the device.

[1456] Output: Humans are generated within the simulated world.

[1457] Specific operation: The server uses OpenAI's GPT-3 to generate images of cafe staff and passersby.

[1458] Step 4:

[1459] The user moves around within the simulated world.

[1460] Input: User controller operation.

[1461] Output: The user's viewpoint shifts.

[1462] Specific action: The user operates the controller and walks down the Champs-Élysées.

[1463] Step 5:

[1464] Users converse with AI-generated human characters.

[1465] Input: User voice input.

[1466] Output: AI-generated human response.

[1467] Specific example: When a user asks a cafe employee, "What do you recommend?", an AI-generated human will reply, "Today's recommendations are croissants and cafe au lait."

[1468] Step 6:

[1469] The server analyzes the user's emotions.

[1470] Input: User voice data and facial expression data.

[1471] Output: User's emotional state.

[1472] Specific operation: The server uses emotion recognition software to analyze the user's voice tone and facial expressions, and determines that the user is slightly tired.

[1473] Step 7:

[1474] The server adjusts the conversation.

[1475] Input: User's emotional state.

[1476] Output: Edited conversation content.

[1477] Specific operation: The server sends a prompt message to the AI ​​model saying, "You seem particularly tired today. I'd like to recommend a cafe where you can relax," and then generates an appropriate conversation.

[1478] (Application Example 2)

[1479] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1480] Modern consumers enjoy the convenience of online shopping, but they also crave the face-to-face interaction and direct product viewing experience offered by physical stores. However, traditional online shopping has faced challenges, such as users not receiving sufficient information when choosing products and not being able to receive appropriate responses tailored to their emotions. Furthermore, providing a customized shopping experience that responds to users' emotions has been difficult.

[1481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1482] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means including an emotion engine that analyzes the user's emotions and generates conversation content corresponding to those emotions, and means for selecting products while conversing with an AI-generated store clerk in a virtual store. As a result, the user can enjoy a realistic shopping experience in a virtual store and receive appropriate responses according to their emotions.

[1483] A "digital twin" is a virtual model that recreates physical objects and environments from the real world in a digital space.

[1484] A "simulated world" is a virtual space created using a digital twin, which users can experience through a VR device.

[1485] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space through sight and sound.

[1486] An "AI-generated human" is a human character created in a virtual space using artificial intelligence technology.

[1487] "Generative AI" refers to artificial intelligence systems designed to generate conversations and interactions with users.

[1488] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[1489] An "emotion engine" is a system that analyzes a user's emotions and generates appropriate responses and conversation content based on those emotions.

[1490] A "virtual store" is a store built in a virtual space that users can visit through VR devices.

[1491] An "AI-generated store clerk" is a store clerk character created using artificial intelligence technology to interact with users within a virtual store.

[1492] The system for implementing this invention is configured as follows: First, the user puts on a VR device and accesses a virtual store. A commercially available VR headset such as the Oculus Rift can be used as the VR device. When the user enters the virtual store, a simulated world generated using digital twin technology is displayed.

[1493] The server generates a simulated world using a digital twin, allowing users to experience it through VR devices. Within this simulated world, AI-generated humans (shop staff) are placed, enabling conversation with the user. The generative AI uses the OpenAI API to generate conversations with the user.

[1494] When a user selects products in a virtual store, an emotion engine analyzes the user's emotions and responds appropriately accordingly. The emotion engine analyzes the user's voice input and facial expression data, generating comforting words if the user is sad and apologies if the user is angry.

[1495] As a concrete example, consider a scenario where a user is looking for shoes in a virtual store. When the user asks, "Do you have this shoe in my size?", the emotion engine analyzes the user's emotions and generates an appropriate prompt. For example, if the user is sad, the prompt "Offer words of comfort when the user is sad" will be generated.

[1496] Example of a prompt:

[1497] "When a user asks for their shoe size, the system will suggest the appropriate size."

[1498] The server sends this prompt to the OpenAI API, which generates a conversation in which an AI-generated shop assistant suggests the appropriate size. The generated conversation is then displayed to the user through a VR device.

[1499] This system allows users to enjoy a realistic shopping experience within a virtual store and receive appropriate responses tailored to their emotions. This provides a new shopping experience that combines the convenience of online shopping with the face-to-face service of a physical store.

[1500] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[1501] Step 1:

[1502] Users wear VR devices and access virtual stores.

[1503] Input: User wearing a VR device and requesting access to a virtual store.

[1504] Output: Connection established to the virtual store

[1505] Specific operation: The user puts on a VR device and launches the virtual store application. The server receives the user's access request and establishes a connection to the virtual store.

[1506] Step 2:

[1507] The server uses a digital twin to generate a simulated world and displays it to the user.

[1508] Input: Digital twin data of the virtual store

[1509] Output: A simulated world displayed on the user's VR device.

[1510] Specific operation: The server uses digital twin technology to generate a simulated world of the virtual store and sends it to the user's VR device. The user then experiences the simulated world through the VR device.

[1511] Step 3:

[1512] When a user selects products in a virtual store, an emotion engine analyzes the user's emotions.

[1513] Input: User voice input and facial expression data

[1514] Output: User sentiment analysis results

[1515] Specific operation: When a user asks a question or makes a comment about a product, the VR device sends voice input and facial expression data to the server. The server uses an emotion engine to analyze this data and identify the user's emotions.

[1516] Step 4:

[1517] The server generates prompt messages that respond to emotions and sends them to the AI ​​model.

[1518] Input: User sentiment analysis results

[1519] Output: Generated prompt message

[1520] Specific operation: The server generates appropriate prompt messages based on the analysis results of the emotion engine. For example, if the user is sad, it will generate a prompt message that says, "When the user is sad, offer words of comfort."

[1521] Step 5:

[1522] The generative AI model generates conversation content based on the prompt text and sends it to the server.

[1523] Input: Generated prompt message

[1524] Output: Generated conversation content

[1525] Specific operation: The server sends the generated prompt text to the generation AI model. The generation AI model generates appropriate conversation content based on the prompt text and returns the result to the server.

[1526] Step 6:

[1527] The server sends the generated conversation content to the user's VR device and displays it.

[1528] Input: Generated conversation content

[1529] Output: Conversation content displayed on the user's VR device.

[1530] Specific operation: The server sends the conversation content received from the generated AI model to the user's VR device. The user then experiences a conversation with the AI-generated store clerk through the VR device.

[1531] Step 7:

[1532] When a user purchases an item within a virtual store, the server processes the purchase through the e-commerce system.

[1533] Input: User's purchase request

[1534] Output: Purchase completion notification

[1535] Specific operation: When a user selects a product and decides to purchase it, the server processes the purchase through the e-commerce system. Once the purchase is complete, the server sends a purchase completion notification to the user.

[1536] (Example 3)

[1537] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1538] There is a need for a system that can efficiently handle purchases and reservations within a simulated world linked to the real world. Furthermore, improving the user experience by recommending products that respond to user emotions is a challenge. Conventional systems have not considered user emotions in their product recommendations, making it difficult to increase user satisfaction.

[1539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1540] In this invention, the server includes means for generating a simulated world using a digital twin; means for experiencing the simulated world in virtual reality; means for generating characters in the simulated world using artificial intelligence and enabling conversation with generative artificial intelligence; means for realizing purchases within the simulated world through e-commerce; means for linking with real-world transportation and accommodation; means for paying a portion of the usage fee as an information provision fee; means including an emotion engine that recognizes the user's emotions and recommends products according to those emotions; means for the emotion engine to transmit the user's emotion data to the server, for the server to select products based on the emotions; and means for transmitting the selected product information to the user's terminal, for the terminal to display the recommended products. This enables product recommendations that correspond to the user's emotions, thereby improving the user experience.

[1541] A "digital twin" is a technology that recreates physical objects and environments from the real world within a virtual space.

[1542] A "simulated world" is a virtual space created using digital twin technology, an environment that users can experience through virtual reality.

[1543] "Virtual reality" is a technology that allows users to experience a virtual space created using computer technology through their sight and hearing.

[1544] Artificial intelligence is a technology in which computer systems imitate human intelligence, learning, reasoning, and self-correcting.

[1545] "Generative artificial intelligence" refers to a system that uses artificial intelligence technology to generate people and objects in a virtual space, enabling dialogue and interaction with the user.

[1546] "E-commerce" refers to a form of transaction in which goods and services are purchased and sold via the internet.

[1547] "Transportation" refers to the means of transport used by a user to move from one place to another.

[1548] "Accommodation facilities" refer to facilities where users can stay temporarily, and include hotels, inns, and other similar establishments.

[1549] "Information provision fees" are a portion of the fees paid by users when using the system, and represent payment for the information and services provided.

[1550] An "emotion engine" is a system that recognizes a user's emotions and recommends appropriate products and services based on those emotions.

[1551] "Emotional data" refers to information about a user's emotions, obtained from their facial expressions, voice, and other sources.

[1552] A "server" is a computer system that stores, processes, and manages data on a network.

[1553] A "terminal" is a device that a user uses to access and operate a system, and includes personal computers and smartphones.

[1554] A "recommended product" is a product selected and suggested by the system based on the user's emotions and preferences.

[1555] This invention is a system in which users select and purchase products in a virtual world, and those products are then delivered to the real world. It also links to real-world transportation and accommodation, so that choices made in the virtual world are reflected in real-world reservations. Furthermore, it uses an emotion engine to recognize the user's emotions and recommend products that correspond to those emotions.

[1556] Hardware and software to be used

[1557] Servers: Database management systems (e.g., MySQL), web servers (e.g., Apache), sentiment engines (e.g., IBM Watson)

[1558] Device: User interface (e.g., web browser, mobile app)

[1559] Generative AI models: Natural language processing models (e.g., GPT-4)

[1560] Specific operation of the system

[1561] 1. Users select products within a simulated world.

[1562] The user accesses a store in a virtual world using their device. When the user selects an item and clicks the purchase button, the device sends information about the selected item to the server.

[1563] 2. Sending and saving product information

[1564] The terminal sends the information of the selected product to the server in JSON format. The server saves the received product information to a database (e.g., MySQL) and returns a confirmation message to the terminal when the saving is complete.

[1565] 3. Execution of delivery arrangements

[1566] The server interacts with the delivery system (e.g., FedEx API) to arrange delivery. The server receives delivery confirmation information from the delivery system and notifies the user's device of this information.

[1567] 4. Choosing transportation and accommodation

[1568] When a user selects transportation and accommodation within the virtual world and clicks the reservation button, the device sends the selected information to the server.

[1569] 5. Sending and saving reservation information

[1570] The device sends the selected reservation information to the server in JSON format. The server stores the received reservation information in its database and makes the reservation in conjunction with the reservation system (e.g., Booking.com API). The server then notifies the user's device of the reservation confirmation.

[1571] 6. Emotion recognition by an emotion engine

[1572] As the user explores the virtual world using the device, the device uses its camera and microphone to capture the user's facial expressions and voice. The device sends the captured data to an emotion engine (e.g., IBM Watson), which analyzes the user's emotions in real time and sends that data to a server.

[1573] 7. Emotion-based product recommendations

[1574] The server selects products that match the user's emotions based on emotional data. The server sends the selected product information to the terminal, and the terminal displays the recommended products to the user.

[1575] Specific examples and prompt statements

[1576] Specific example 1: Product purchase

[1577] The user selects a smartphone in an electronics store within a simulated world and clicks the purchase button.

[1578] The server receives information from smartphones and works with the delivery system to arrange deliveries in the real world.

[1579] The user will receive their smartphone in the real world a few days later.

[1580] Example 2: Hotel reservations

[1581] The user selects a hotel within a simulated world and clicks the reservation button.

[1582] The server receives hotel information and, in conjunction with the reservation system, makes reservations in the real world.

[1583] The user receives a hotel reservation confirmation in the real world.

[1584] Example of a prompt

[1585] "Please explain the procedure for delivering items selected by the user within a virtual world to the real world."

[1586] "Please explain the system for recommending products based on user emotions."

[1587] The above describes the embodiments for carrying out this invention. The flow of the specific processing in Example 3 will be explained with reference to Figure 21.

[1588] Step 1:

[1589] Users select products within a simulated world.

[1590] Input: The user uses a device to access a store in a virtual world and select an item.

[1591] Specific action: The user selects a product and clicks the purchase button.

[1592] Output: The terminal sends information about the selected product to the server in JSON format.

[1593] Step 2:

[1594] Sending and saving product information

[1595] Input: Product information in JSON format sent from the device.

[1596] Specific operation: The server analyzes the received product information and saves the "user ID, product ID, and purchase date and time" to a database (e.g., MySQL).

[1597] Output: The server returns a confirmation message to the terminal acknowledging that saving is complete.

[1598] Step 3:

[1599] Execute delivery arrangements

[1600] Input: Product information stored in the database.

[1601] Specific operation: The server interacts with the delivery system (e.g., FedEx API) and sends the user's address, product ID, and delivery date and time. The delivery system returns the delivery ID and estimated delivery date and time to the server.

[1602] Output: The server notifies the user's device of the "delivery ID and scheduled delivery date and time."

[1603] Step 4:

[1604] Selection of transportation and accommodation

[1605] Input: The user uses a device to select transportation and accommodation within the simulated world.

[1606] Specific operation: The user clicks the reservation button, and the device sends the selected information to the server in JSON format.

[1607] Output: The terminal sends the selected reservation information to the server.

[1608] Step 5:

[1609] Sending and saving reservation information

[1610] Input: Reservation information in JSON format sent from the terminal.

[1611] Specific operation: The server parses the received reservation information and stores the "User ID, Reservation ID, and Reservation Date and Time" in the database. The server interacts with the reservation system (e.g., Booking.com API) and sends the "User Name, Reservation ID, and Departure Date and Time". The reservation system returns the "Reservation Confirmation Number and Departure Date and Time" to the server.

[1612] Output: The server notifies the user's device of the "reservation confirmation number and departure date and time."

[1613] Step 6:

[1614] Emotion recognition by an emotion engine

[1615] Input: Facial expressions and audio data from when the user explores the simulated world using a device.

[1616] Specific operation: The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to an emotion engine (e.g., IBM Watson). The emotion engine analyzes the user's emotions in real time and sends the data to a server.

[1617] Output: The emotion engine sends the analysis results to the server.

[1618] Step 7:

[1619] Emotion-based product recommendations

[1620] Input: Emotional data sent from the emotion engine.

[1621] Specific operation: The server selects products that match the user's emotions based on emotion data. The server then sends the selected product information to the terminal.

[1622] Output: The terminal displays recommended products to the user.

[1623] (Application Example 3)

[1624] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1625] There is a need to optimize purchase and reservation actions within a simulated world linked to the real world, in accordance with the user's emotions. Conventional systems provide goods and services without considering the user's emotions, resulting in a limited user experience and low satisfaction. Furthermore, systems that reflect choices made within the simulated world in real-world delivery and reservations are not yet sufficiently developed.

[1626] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases in the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for recognizing the user's emotions using an emotion engine and recommending products corresponding to those emotions, means for delivering the products selected in the simulated world to the real world, and means for booking transportation and accommodation selected in the simulated world to the real world. This makes it possible to provide optimal products and services according to the user's emotions, and realizes a system in which selections in the simulated world are reflected in delivery and reservations in the real world.

[1627] A "digital twin" is a digital reproduction of physical objects and environments from the real world.

[1628] A "simulated world" is a virtual environment created using a digital twin, which users can experience using virtual reality (VR) technology.

[1629] "VR" is an abbreviation for Virtual Reality, a technology that allows users to experience a computer-generated three-dimensional space.

[1630] "AI" is an abbreviation for Artificial Intelligence, which is a technology in which computers imitate human intelligence to learn and reason.

[1631] "Generative AI" is a technology that uses artificial intelligence to generate humans and objects within a simulated world, enabling interaction with the user.

[1632] "EC" is an abbreviation for Electronic Commerce, which is a system for buying and selling goods and services via the internet.

[1633] An "emotion engine" is a technology that recognizes a user's emotions and provides responses and recommendations accordingly.

[1634] "Means of transportation" refers to the means of transport that users use to move around in the real world, including, for example, cars, trains, and airplanes.

[1635] "Accommodation facilities" refer to facilities in the real world where users can stay overnight, and include, for example, hotels and inns.

[1636] "Information provision fees" are fees paid for information provided by users and are treated as part of the service usage fees.

[1637] "Delivery" refers to the act of delivering purchased goods to a location specified by the user.

[1638] "Reservation" refers to the act of a user securing a specific service or facility in advance.

[1639] The system for carrying out this invention generates a simulated world using a digital twin, and allows users to experience this simulated world using virtual reality (VR) technology. The system includes the following main components:

[1640] 1. Digital twin generation means:

[1641] The server generates a digital twin to recreate physical objects and environments from the real world in digital space. This uses 3D modeling software and sensor technology.

[1642] 2. VR experience methods:

[1643] Users will experience the generated virtual world using a head-mounted display (HMD) or smartphone. Specific hardware used will include the Oculus Rift and HTC Vive.

[1644] 3. AI generation means:

[1645] The server uses artificial intelligence (AI) to generate humans and objects within a simulated world, enabling interaction with the user. This is done using a generative AI model.

[1646] 4. Means of implementing e-commerce:

[1647] An e-commerce (EC) system is provided by a server, allowing users to select and purchase products within a virtual world. Purchased products are then delivered to the real world.

[1648] 5. Emotional Engine:

[1649] The server recognizes the user's emotions in real time and recommends products that correspond to those emotions. OpenCV and TensorFlow, using cameras and microphones, are used for emotion recognition.

[1650] 6. Means of linking with the real world:

[1651] The modes of transportation and accommodations selected by the user within the virtual world will be reflected in their real-world reservations. This includes integration with the reservation system via an API.

[1652] 7. Information provision fee methods:

[1653] The server includes a mechanism for paying a portion of the usage fee as an information provision fee for the information provided by the user.

[1654] As a concrete example, consider a scenario where a user is searching for a new smartphone in a virtual store. If the user is happy, the system will recommend the latest smartphone model.

[1655] Examples of prompts to input into a generative AI model:

[1656] A user is searching for a new smartphone in a virtual store. Since the user is happy, please recommend the latest smartphone model.

[1657] This system enables the provision of optimal products and services tailored to the user's emotions, and choices made within the virtual world are reflected in real-world deliveries and reservations. This significantly improves the user experience.

[1658] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[1659] Step 1:

[1660] The server generates a digital twin to recreate physical objects and environments from the real world in digital space. It receives real-world data (e.g., 3D scan data or sensor data) as input and uses 3D modeling software to generate the digital twin. The output is a 3D model of the simulated world.

[1661] Step 2:

[1662] Users experience a generated virtual world using a head-mounted display (HMD) or smartphone. The input is a 3D model of the virtual world sent from a server, which is then displayed through a VR application. The output is the visual information of the virtual reality experienced by the user.

[1663] Step 3:

[1664] The server uses artificial intelligence (AI) to generate humans and objects within a simulated world, enabling interaction with the user. It receives user behavior data and simulated world situation data as input, and uses a generative AI model to generate appropriate humans and objects. The output is an interactive character or object within the simulated world.

[1665] Step 4:

[1666] Users utilize an e-commerce (EC) system to select and purchase products within a simulated world. The system receives information about the products selected by the user as input and processes the purchase through the EC system. Outputs include purchase confirmation information and shipping arrangement information.

[1667] Step 5:

[1668] The server uses an emotion engine to recognize the user's emotions in real time and recommends products that correspond to those emotions. It receives user facial and voice data acquired via camera and microphone as input, and analyzes those emotions using an emotion recognition algorithm. The output is product recommendations tailored to the user's emotions.

[1669] Step 6:

[1670] The mode of transportation and accommodation selected by the user within the virtual world are reflected in the real-world reservation. The system receives information on the user's selected mode of transportation and accommodation as input and connects to the reservation system via an API. The output is reservation confirmation information.

[1671] Step 7:

[1672] The server includes a mechanism for paying a portion of the usage fee as an information provision fee for the information provided by the user. It receives the user-provided information as input and calculates the usage fee. The output is payment confirmation information for the information provision fee.

[1673] In this way, through specific actions and data flows at each step, it becomes possible to provide optimal products and services that respond to the user's emotions, and a system is realized in which choices made in the simulated world are reflected in delivery and reservations in the real world.

[1674] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1675] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1676] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1677] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1678] [Third Embodiment]

[1679] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1680] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1681] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1682] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1683] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1684] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1685] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1686] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1687] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1688] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1689] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1690] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1691] "Example of form 1"

[1692] One embodiment of the present invention is a system that generates a simulated world using a digital twin. This system acquires data on a place the user wants to visit and generates a digital twin based on that data. For example, if a user wants to visit Paris, the system acquires data on the streets, buildings, and scenery of Paris and generates a simulated world of Paris based on that data.

[1693] "Example of form 2"

[1694] The generated virtual world is experienced by the user through a VR device. By wearing the VR device, the user can experience walking through the streets of Paris from the comfort of their home. Furthermore, the people in the virtual world are generated by AI, and it is possible to converse with the generative AI. This allows the user to ask for directions and enjoy everyday conversations.

[1695] "Example of form 3"

[1696] Furthermore, purchases within the virtual world are implemented via e-commerce. When a user selects and purchases an item from a store within the virtual world, that item is delivered to the user in the real world. It is also linked to real-world transportation and accommodation; when a user selects transportation or accommodation within the virtual world, it is reflected in their real-world reservations.

[1697] The following describes the processing flow for each example of the form.

[1698] "Example of form 1"

[1699] Step 1: The user enters the place they want to visit into the system. For example, if the user wants to visit Paris, they would enter "Paris" into the system.

[1700] Step 2: The system retrieves data for the relevant location based on the entered information. This data includes information about the streetscape, buildings, and scenery.

[1701] Step 3: Based on the acquired data, a digital twin is generated. This digital twin is a simulated world that recreates the streets, buildings, and scenery of Paris.

[1702] "Example of form 2"

[1703] Step 1: The user puts on the VR device.

[1704] Step 2: Experience the generated virtual world through the VR device you're wearing. Users can experience walking through the streets of Paris from the comfort of their own homes.

[1705] Step 3: Humans in the simulated world are generated by AI, and it is possible to converse with the generative AI. Users can ask AI-generated humans for directions or enjoy everyday conversations.

[1706] "Example of form 3"

[1707] Step 1: The user selects and purchases products in a store within a simulated world.

[1708] Step 2: Purchased items are delivered to the user in the real world via the e-commerce system.

[1709] Step 3: When a user selects transportation and accommodation within the virtual world, these selections are reflected in their real-world reservations.

[1710] (Example 1)

[1711] Next, we will describe Embodiment 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1712] Conventional digital twin technologies present challenges such as a complex process for users to virtually experience places they wish to visit, and a lack of easily accessible systems. Furthermore, there are insufficient means to efficiently convert collected geographic data into 3D models and transmit them to users' devices. This results in difficulties for users to engage in real-time virtual experiences.

[1713] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1714] In this invention, the server includes means for inputting a place the user wants to visit, means for collecting geographic data based on the input, means for analyzing the collected geographic data and converting it into a 3D model, means for generating a digital twin using the 3D model, means for transmitting the digital twin to the user's terminal, and means for the user to operate the digital twin. This makes it possible for the user to easily virtually experience a place they want to visit.

[1715] A "user" is an individual or group that uses the system to specify a place they want to visit and virtually experience it.

[1716] "Input" refers to the act or information that a user provides to the system regarding a place they wish to visit.

[1717] "Geographic data" refers to information about a specified location, such as latitude and longitude, building shapes, and road layouts.

[1718] "Analysis" refers to the process of calculation and data processing to process collected geographic data and convert it into a 3D model.

[1719] A "3D model" is an object in a three-dimensional virtual space that is generated based on analyzed geographical data.

[1720] A "digital twin" is a 3D model that recreates a specific location in the real world within a virtual space.

[1721] "Transmission" refers to the act of transferring the digital twin generated by the server to the user's terminal.

[1722] "Operation" refers to the act of a user interactively using a digital twin on a device.

[1723] This invention is a system for users to virtually experience places they wish to visit. The system includes a series of processes: the user inputs the place they wish to visit, geographic data about that place is collected and analyzed, converted into a 3D model, a digital twin is generated, and transmitted to the user's terminal.

[1724] Hardware and software to be used

[1725] server

[1726] The server is responsible for receiving user input, collecting and analyzing geographic data, and generating 3D models. Specifically, it uses the following software and services:

[1727] Google Maps API: Used to collect geographic data.

[1728] OpenStreetMap: Used to collect supplementary geographic data.

[1729] Python: Run a script to analyze the collected geographic data and convert it into a 3D model.

[1730] Unity: A game engine for generating 3D models based on analyzed data.

[1731] terminal

[1732] The terminal is a device that allows users to interact with a digital twin and virtually experience places they wish to visit. Specifically, it uses the following software:

[1733] Unity application: Displays 3D models sent from the server and allows the user to interact with them.

[1734] Data processing and data calculation

[1735] Data collection

[1736] The server collects geographic data from the Google Maps API and OpenStreetMap based on the location the user enters. For example, if the user enters "Paris," the server retrieves data such as the latitude and longitude of Paris, the shape of buildings, and the layout of roads.

[1737] Data Analysis

[1738] The server analyzes the collected geographic data and converts it into a 3D model. Specifically, it uses a Python script to convert latitude and longitude information into 3D coordinates and building shapes into polygon data. This process generates the basic data for the digital twin.

[1739] 3D model generation

[1740] The server uses Unity to generate 3D models based on the analyzed data. Specifically, it executes Unity's C scripts to create 3D models of Parisian streets and buildings. This completes a digital twin that allows users to virtually visit Paris.

[1741] Sending data

[1742] The server sends the generated 3D model to the user's device. The user can then launch the Unity application on their device and interact with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[1743] Specific example

[1744] When a user enters "I want to visit Paris" into the system, the server uses the Google Maps API to retrieve geographical data for Paris. Next, a Python script is executed to analyze the retrieved geographical data and convert it into a 3D model. Unity is used to generate a 3D model of Paris's streets and buildings based on the analyzed data. Finally, the generated 3D model is sent to the user's device, and the user can launch the Unity application on their device to virtually experience Paris.

[1745] Example of a prompt

[1746] "Please create a digital twin of Paris. Use the Google Maps API to obtain geographical data for Paris and generate a 3D model using Unity."

[1747] In this way, a system is built that allows users to virtually experience places they want to visit.

[1748] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1749] Step 1:

[1750] The user enters the place they want to visit.

[1751] Specifically, the user accesses the system's web interface, enters "Paris" into the search box, and clicks the search button.

[1752] Input: The location the user wants to visit (e.g., Paris)

[1753] Output: Information about the place you want to visit (e.g., Paris)

[1754] Step 2:

[1755] The server receives user input and collects geographical data.

[1756] Specifically, the server sends a request to the Google Maps API to retrieve geographical data for Paris. For example, it sends a request to a URL like https: / / maps.googleapis.com / maps / api / place / textsearch / json?query=Paris&key=YOUR_API_KEY.

[1757] Input: Information about the place you want to visit (e.g., Paris)

[1758] Output: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[1759] Step 3:

[1760] The server analyzes the geographic data it collects and converts it into a 3D model.

[1761] Specifically, the server executes a Python script to analyze the acquired geographic data. For example, it runs a script called parse_geodata.py to convert latitude and longitude information into 3D coordinates.

[1762] Input: Collected geographical data (e.g., latitude and longitude information for Paris, building shapes, road layout)

[1763] Output: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[1764] Step 4:

[1765] The server generates a 3D model based on the analyzed data.

[1766] Specifically, the server executes Unity C scripts to create 3D models of Parisian streets and buildings. For example, it executes a script called GenerateParisModel.cs.

[1767] Input: Basic data of the analyzed 3D model (e.g., 3D coordinates, polygon data)

[1768] Output: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[1769] Step 5:

[1770] The server generates a 3D model and sends it to the user's device.

[1771] Specifically, the server transfers the generated 3D model to the user's terminal.

[1772] Input: Generated 3D model (e.g., 3D model of Parisian streets and buildings)

[1773] Output: 3D model sent to the user's terminal

[1774] Step 6:

[1775] Users interact with the digital twin on their devices.

[1776] In terms of specific actions, the user launches a Unity application on their device and interacts with the digital twin. For example, the user can virtually visit the Eiffel Tower or stroll along the Champs-Élysées.

[1777] Input: 3D model sent to the user's device

[1778] Output: User-operated digital twin

[1779] (Application Example 1)

[1780] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1781] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping struggles to replicate the experience and atmosphere of a real store. Furthermore, when planning trips or sightseeing, there are limited ways to experience detailed information and the atmosphere of places they want to visit beforehand. This often leads to anxiety for consumers who cannot verify product or location details before purchasing. Additionally, the lack of ways to link online shopping with real-world transportation and accommodation can result in a lack of consistency in travel planning.

[1782] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1783] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for acquiring data on places the user wants to visit and generating a digital twin based on that data, means for providing a virtual shopping tour based on the digital twin, and means for experiencing the virtual shopping tour through a smartphone or head-mounted display. This makes it possible for users to experience detailed information and the atmosphere of places they want to visit in advance, thereby reducing anxiety when online shopping or planning trips.

[1784] A "digital twin" is a virtual representation of physical locations and objects in the real world, created as digital data.

[1785] A "pseudo-world" refers to a virtual environment or space created using a digital twin.

[1786] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[1787] "AI" is an abbreviation for artificial intelligence, which is a technology in which computers imitate human intelligence to learn and reason.

[1788] "Generative AI" is a technology that uses artificial intelligence to generate humans and objects in a virtual space.

[1789] "EC" is an abbreviation for electronic commerce, which refers to the buying and selling of goods and services via the internet.

[1790] "Means of transportation" refers to the means or methods of transport that a user uses to move physically.

[1791] "Accommodation facilities" refer to facilities or places where users can stay temporarily.

[1792] "Information provision fee" refers to the fee that a user pays for the information provided.

[1793] A "virtual shopping tour" is a tour that allows users to experience shopping in a virtual space created using a digital twin.

[1794] A "smartphone" is a type of mobile phone, a multi-functional device capable of internet access and application use.

[1795] A "head-mounted display" is a display device worn on the head and used to experience virtual reality and augmented reality.

[1796] The system for carrying out this invention generates a simulated world using a digital twin, allowing users to experience a virtual shopping tour. Specific embodiments of this system are described below.

[1797] System Configuration

[1798] The system mainly consists of the following components:

[1799] 1. Server: Generates a digital twin and provides data in response to user requests.

[1800] 2. Device: A smartphone or head-mounted display used by the user.

[1801] 3. User: Experience a virtual shopping tour using the system.

[1802] Program processing

[1803] The server implements the system using the following methods.

[1804] Digital Twin Generation Method: Data on a location the user wishes to visit is acquired, and a digital twin is generated based on that data. Specifically, location data is acquired via an API, and a virtual space is generated using a 3D rendering library.

[1805] VR Experience Method: The generated digital twin can be experienced in VR. Users can use smartphones or head-mounted displays to freely move around and experience the virtual space.

[1806] AI generation method: An AI generates human characters in a virtual space, enabling conversations with the user. The generated AI model is used to achieve natural, real-time conversations.

[1807] E-commerce implementation method: Purchases within a virtual space are realized through electronic commerce. Users can select products in a virtual store and complete the purchase process.

[1808] Linking mechanism: Links to real-world transportation and accommodation. Users can make reservations and arrangements within the virtual space.

[1809] Information provision fee payment method: A portion of the usage fee is paid as an information provision fee. Users can use the service by paying a fee for the information provided.

[1810] Hardware and software to be used

[1811] Hardware: Smartphones, head-mounted displays

[1812] Software: Python, 3D rendering library (e.g., some_3d_rendering_library), generative AI model, API

[1813] Specific example

[1814] If a user wants to shop on the Champs-Élysées in Paris, the system will operate according to the following steps.

[1815] 1. Users access the system using a smartphone or head-mounted display.

[1816] 2. The server retrieves data from the Champs-Élysées in Paris and generates a digital twin.

[1817] 3. Users experience the generated digital twin in VR and move freely within the virtual space.

[1818] 4. Interact with AI-generated humans in a virtual space and receive directions and product descriptions.

[1819] 5. Users select products in a virtual store and complete the purchase process through e-commerce.

[1820] 6. Book real-world transportation and accommodation as needed.

[1821] Example of a prompt

[1822] Please enter the name of the city you would like to visit. For example, if you enter "Paris," a digital twin of the Champs-Élysées in Paris will be generated, allowing you to start a virtual shopping tour.

[1823] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[1824] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1825] Step 1:

[1826] The user accesses the system using a smartphone or head-mounted display. The user enters the name of the place they want to visit. The entered place name is sent to the server.

[1827] Step 2:

[1828] The server retrieves data for a location based on the entered location name. Specifically, it requests location data via an API and parses the retrieved data. The input is the location name, and the output is detailed location data.

[1829] Step 3:

[1830] The server generates a digital twin based on the acquired data. Using a 3D rendering library, it recreates detailed location data as a virtual space. The input is detailed location data, and the output is the digital twin.

[1831] Step 4:

[1832] The server sends the generated digital twin to the user's device. The user then experiences the digital twin in VR through a smartphone or head-mounted display. The input is the digital twin, and the output is the VR experience.

[1833] Step 5:

[1834] The server generates human characters in a virtual space using a generative AI model. The generated AI enables conversation with the user. The input is the generative AI model and data from the virtual space, and the output is the AI-generated human character.

[1835] Step 6:

[1836] Users converse with AI-generated humans in a virtual space, receiving directions and product descriptions. Input consists of user questions and requests, while output is the AI-generated human's response.

[1837] Step 7:

[1838] Users select products in a virtual store and proceed with the purchase. The server facilitates the purchase through e-commerce. The input is the user's purchase information, and the output is a purchase confirmation.

[1839] Step 8:

[1840] Users book real-world transportation and accommodations as needed. The server processes this booking information and provides it to the user. The input is the user's booking information, and the output is a booking confirmation.

[1841] Step 9:

[1842] The server collects a portion of the usage fee from the user as an information provision fee. The input is the user's usage information, and the output is a confirmation of fee collection.

[1843] In this way, users can experience detailed information and the atmosphere of places they want to visit in advance, reducing anxiety when shopping online or planning trips.

[1844] (Example 2)

[1845] Next, we will describe Example 2 of the Form Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1846] Traditional virtual reality systems have limitations, such as the restricted virtual world experienced by users, making natural real-time conversation and interaction difficult. Furthermore, the lack of sufficient linkage between the virtual world and the real world, including inadequate purchasing capabilities, limits the user experience. Additionally, there is a lack of effective means to utilize user voice to facilitate natural conversations with generative artificial intelligence.

[1847] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1848] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world with a virtual reality device, means for generating humans in the simulated world with artificial intelligence and enabling conversation with the generative AI, means for realizing purchases within the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for converting the user's voice into text and sending it to the generative AI as a prompt, and means for converting the response generated by the generative AI into voice and returning it to the user. As a result, the user can enjoy natural conversations in real time, interaction within the simulated world is improved, and the link with the real world is strengthened.

[1849] A "digital twin" is a virtual model that digitally reproduces physical objects and environments in the real world.

[1850] A "virtual world" is a virtual environment that users experience through a virtual reality device, and which is different from the real world.

[1851] A "virtual reality device" is a device that allows users to experience a virtual environment visually and aurally.

[1852] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[1853] "Generative artificial intelligence" refers to artificial intelligence models that generate natural-sounding responses based on user input.

[1854] "E-commerce" refers to transactions involving the purchase and sale of goods and services via the internet.

[1855] A "prompt message" is a text-based instruction sent to a generative artificial intelligence system based on user input.

[1856] "Speech recognition software" is software used to convert speech into text.

[1857] "Speech synthesis software" is software used to convert text into speech.

[1858] This invention is a system that generates a simulated world experienced by the user through a virtual reality device and enables natural conversation with generative artificial intelligence. Specific embodiments of this system are described below.

[1859] First, the server generates a virtual world using a digital twin. This virtual world is developed using a game engine such as Unity or Unreal Engine. The generated virtual world is then delivered to the user through a virtual reality device (e.g., Oculus Rift, HTC Vive).

[1860] When a user puts on a virtual reality device, the device connects to a server and receives data from the virtual world. Based on the received data, the device renders the virtual world on the virtual reality device's display in real time. This allows the user to experience as if they were actually walking around in that place.

[1861] Humans within the simulated world are generated using generative artificial intelligence. When a user speaks to a human in the simulated world, the device captures the audio with a microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is then sent to the generative artificial intelligence as a prompt.

[1862] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates an appropriate response based on the received prompt. For example, if a user asks, "Where is the Eiffel Tower?", the generative AI will generate the response, "The Eiffel Tower is located straight down this road, and you'll turn right at the next intersection."

[1863] The device converts the generated response into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is then returned to the user through the virtual reality device's speakers. This allows the user to hear responses from people within the simulated world.

[1864] As a concrete example, suppose a user is walking around Paris wearing a virtual reality device. If the user asks a person in the virtual world, "Where is the Eiffel Tower?" near the Eiffel Tower, the following prompt will be sent to the generative artificial intelligence:

[1865] Example of a prompt:

[1866] User: "Where is the Eiffel Tower?"

[1867] The generative artificial intelligence receives this prompt and generates an appropriate response. The terminal converts this response into speech and returns it to the user. This allows the user to enjoy a natural conversation in real time.

[1868] This system enhances user interaction within the virtual world and strengthens its link to the real world. For example, it enables e-commerce transactions within the virtual world and links to real-world transportation and accommodation. Furthermore, it includes a method for paying a portion of the usage fee as an information provision fee, further enriching the user experience.

[1869] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1870] Step 1:

[1871] The user puts on a virtual reality device.

[1872] Input: The action of a user putting on a virtual reality device.

[1873] Output: The virtual reality device is activated, and the user's sight and hearing are switched to the virtual reality device.

[1874] Specific operation: The user puts the virtual reality device on their head and turns on the device. This activates the device, and the user's sight and hearing switch to information provided through the virtual reality device.

[1875] Step 2:

[1876] The device connects to the server and receives data from the simulated world.

[1877] Input: Activation signal for a virtual reality device.

[1878] Output: Data from the simulated world is sent from the server to the terminal.

[1879] Specific operation: The device connects to the server via the internet, and the server sends data of a simulated world generated by Unity or Unreal Engine to the device. The device receives this data and loads it into memory.

[1880] Step 3:

[1881] The device renders a simulated world and displays it to the user.

[1882] Input: Data from a simulated world received from the server.

[1883] Output: A simulated world is displayed on the virtual reality device's screen.

[1884] Specific operation: Based on the received data, the device renders the streets of Paris in real time on the virtual reality device's display. Through the virtual reality device, the user experiences a visual sensation as if they were actually walking through the streets of Paris.

[1885] Step 4:

[1886] The user acts within a simulated world and speaks to people within that world.

[1887] Input: User voice input.

[1888] Output: The user's voice is captured on the device.

[1889] Specific actions: The user uses the controller of the virtual reality device to move freely within the simulated world. For example, suppose the user is walking near the Eiffel Tower. The user approaches a person in the simulated world and asks, "Where is the Eiffel Tower?"

[1890] Step 5:

[1891] The device converts the user's voice into text and sends it as a prompt to the generative AI model.

[1892] Input: User's voice data.

[1893] Output: A text-formatted prompt message.

[1894] Specific operation: The device captures the user's voice using the microphone and converts it to text using speech recognition software (e.g., Google Cloud Speech-to-Text). The converted text is sent to the AI ​​model as a prompt.

[1895] Step 6:

[1896] The server generates a response using a generated AI model.

[1897] Input: A text-based prompt message.

[1898] Output: The generated response text.

[1899] Specific operation: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an appropriate response to the received prompt. For example, it might generate a response like, "Go straight down this road and turn right at the next intersection."

[1900] Step 7:

[1901] The device converts the generated response into speech and returns it to the user.

[1902] Input: The generated response text.

[1903] Output: A voice response.

[1904] Specific operation: The terminal converts the response received from the server into speech using speech synthesis software (e.g., Google Cloud Text-to-Speech). The converted speech is returned to the user through the virtual reality device's speaker. The user can hear the response from a person in the simulated world.

[1905] (Application Example 2)

[1906] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1907] Modern consumers want to enjoy shopping without visiting physical stores, but traditional online shopping has the problem of not being able to actually touch and examine products. Furthermore, it's difficult to obtain detailed information about products through interaction with store staff when shopping online. In addition, there are limited ways for users to experience the streetscapes of foreign cities and enjoy the local atmosphere from the comfort of their homes. To solve these problems, a system is needed that provides a more realistic shopping experience and interactive communication with users.

[1908] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1909] In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in VR, means for generating humans in the simulated world using AI and enabling conversation with the generative AI, means for realizing purchases within the simulated world via e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for tracking the user's location and initiating a conversation with an AI-generated character, means for processing user input and generating a response from the AI, and means for displaying the response. As a result, users can enjoy a realistic shopping experience from the comfort of their homes and obtain detailed information about products through interactive communication with store staff. They can also experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1910] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[1911] A "simulated world" is a virtual environment or space created using a digital twin.

[1912] "VR" is an abbreviation for virtual reality, a technology that allows users to experience a virtual space created using computer technology.

[1913] "AI" is an abbreviation for artificial intelligence, which is a computer system that mimics human intelligence.

[1914] "Generative AI" refers to artificial intelligence used for interacting with users and generating content.

[1915] "EC" is an abbreviation for electronic commerce, which is a system for buying and selling goods and services over the internet.

[1916] "Real-world transportation" refers to the means of transport that users use to travel in the real world.

[1917] An "accommodation facility" is a facility where users stay overnight.

[1918] "Information provision fees" are the fees paid by users for the information they provide.

[1919] "Tracking user location" means tracking the user's current location in real time.

[1920] An "AI-generated character" is a virtual person or character created by artificial intelligence.

[1921] "Processing user input" means analyzing user input data and generating an appropriate response.

[1922] "Generating responses from AI" means that artificial intelligence creates appropriate responses to user input.

[1923] "Displaying the response" means visually presenting the generated response to the user.

[1924] To implement this invention, the following system configuration and program are required.

[1925] System Configuration

[1926] 1. Hardware:

[1927] VR devices (e.g., Oculus Rift, HTC Vive)

[1928] Smartphones (e.g., iPhone, Android)

[1929] Servers (those with high-performance computing capabilities)

[1930] 2. Software:

[1931] VR engines (e.g., Unity, Unreal Engine)

[1932] AI conversation engine (e.g., OpenAI GPT-3)

[1933] Program Processing Description

[1934] The server first generates a simulated world that recreates the streets of Paris using a digital twin. Next, the user can experience this simulated world by wearing a VR device. The user's position is tracked in real time, and a conversation begins when the user approaches an AI-generated character.

[1935] User input is sent to the server via a VR device or smartphone. The server analyzes this input and generates an appropriate response using an AI conversation engine. The generated response is displayed in the user's field of view.

[1936] Specific example

[1937] When a user enters a virtual store and asks an AI-generated shop assistant, "Do you have this dress in my size?", the server receives this input and uses its AI conversation engine to generate a response such as, "Yes, this dress is available in sizes S, M, and L." This response is then displayed in the user's field of view.

[1938] Example of a prompt

[1939] User: Do you have this dress in my size?

[1940] AI: Yes, this dress is available in sizes S, M, and L.

[1941] In this way, users can enjoy a realistic shopping experience from the comfort of their homes. They can also obtain detailed information about products through interactive communication with store staff. Furthermore, they can experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1942] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1943] Step 1:

[1944] The server generates a simulated world that recreates the streets of Paris using a digital twin.

[1945] Input: Data on the streets of Paris

[1946] Data processing: Using digital twin technology, the streetscape of Paris in the real world is recreated in a virtual space.

[1947] Output: A simulated world that recreates the streets of Paris.

[1948] Step 2:

[1949] Users wear VR devices and experience a simulated world transmitted from a server.

[1950] Input: Data from a simulated world sent from the server.

[1951] Data processing: VR devices track the user's viewpoint and movements in real time to display a simulated world.

[1952] Output: A simulated world displayed in the user's field of view.

[1953] Step 3:

[1954] The server tracks the user's location in real time.

[1955] Input: User's location information

[1956] Data calculation: Calculates the user's position in real time and updates their position within the simulated world.

[1957] Output: Updated user location information

[1958] Step 4:

[1959] When a user approaches an AI-generated character, the server initiates a conversation.

[1960] Input: User's location information, AI-generated character's location information

[1961] Data calculation: Calculates the distance between the user and the character, and initiates a conversation when they are within a certain distance.

[1962] Output: Trigger for starting a conversation

[1963] Step 5:

[1964] Users send input to the server via VR devices or smartphones.

[1965] Input: User voice input or text input

[1966] Data processing: Send input data to the server.

[1967] Output: User input data sent to the server

[1968] Step 6:

[1969] The server analyzes user input and generates appropriate responses using an AI conversation engine.

[1970] Input: User input data

[1971] Data processing: Use an AI conversation engine (e.g., OpenAI GPT-3) to generate appropriate responses to user input.

[1972] Output: Generated response data

[1973] Step 7:

[1974] The server displays the generated response in the user's field of view.

[1975] Input: Generated response data

[1976] Data processing: Send response data to the VR device and display it in the user's field of view.

[1977] Output: Response displayed in the user's field of view

[1978] In this way, users can enjoy a realistic shopping experience from the comfort of their homes. They can also obtain detailed information about products through interactive communication with store staff. Furthermore, they can experience the streetscapes of foreign countries and enjoy the local atmosphere.

[1979] (Example 3)

[1980] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1981] There is a need for a system that can efficiently handle purchases and reservation procedures within a simulated world linked to the real world. Conventional systems have cumbersome processes for reflecting actions in the simulated world back into the real world, resulting in low user convenience. Furthermore, there is a need for a system that makes the experience within the simulated world more realistic and allows users to freely choose locations and times.

[1982] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a simulated world using a digital twin, means for experiencing the simulated world in virtual reality, means for generating people in the simulated world using artificial intelligence and enabling conversation with generative artificial intelligence, means for realizing purchases in the simulated world through e-commerce, means for linking with real-world transportation and accommodation, means for paying a portion of the usage fee as an information provision fee, means for performing delivery procedures in the real world when the user selects a product in the simulated world and confirms the purchase, and means for performing reservation procedures in the real world when the user selects transportation or accommodation in the simulated world. As a result, the user's actions in the simulated world are reflected quickly and efficiently in the real world.

[1983] A "digital twin" is a virtual model that digitally reproduces physical objects and environments from the real world.

[1984] A "simulated world" is a virtual reality environment created using a digital twin.

[1985] "Virtual reality" is a technology that allows users to experience an artificial environment created using computer technology.

[1986] "Artificial intelligence" is a technology in which computer systems mimic human intelligence and act accordingly.

[1987] "Generative artificial intelligence" is a system that uses artificial intelligence technology to generate characters in a simulated world and enables conversations with users.

[1988] "E-commerce" refers to a form of transaction in which goods and services are bought and sold via the internet.

[1989] "Means of transportation" refers to the means by which a user moves from one place to another.

[1990] An "accommodation facility" is a facility where users can stay temporarily.

[1991] "Information provision fees" are a portion of th...

Claims

1. A means of generating a simulated world using a digital twin, A means of experiencing the aforementioned simulated world in virtual reality, A means for generating a person within the aforementioned simulated world using artificial intelligence and enabling conversation with a generative artificial intelligence, A means of realizing the purchase activity within the aforementioned simulated world through electronic commerce, Means of linking to real-world means of transportation or accommodation, One method of paying a portion of the usage fee as an information provision fee, When a user selects an item in the virtual world and confirms the purchase, a means is provided to carry out the delivery procedure in the real world. When the user selects the means of transportation or the accommodation within the simulated world, a means for making a reservation in the real world is provided. A means for acquiring data on the place the user wants to visit and generating a digital twin of the place based on that data, A means of providing a virtual shopping tour of the desired location based on the aforementioned digital twin, A means for allowing the user to experience the aforementioned virtual shopping tour via a smartphone or head-mounted display, A system that includes this.

2. A means of generating a simulated world using a digital twin, A means of experiencing the aforementioned simulated world in virtual reality, A means for generating humans within the aforementioned simulated world using artificial intelligence and enabling conversation with a generative artificial intelligence, A means of realizing the purchase activity within the aforementioned simulated world through electronic commerce, Means of linking to real-world means of transportation or accommodation, One method of paying a portion of the usage fee as an information provision fee, A means for recognizing the user's emotions using an emotion engine, recommending products according to those emotions, and modifying the content or behavior of the human conversations within the simulated world, The emotion engine, having recognized the user's emotions, transmits the user's emotion data to a server, and the server selects candidates for the product and the means of transportation or the accommodation based on the emotion data. A means for presenting the candidates within the simulated world through conversation with the human, and for delivering the selected product in the simulated world to the real world when the user selects one of the candidates through conversation with the human within the simulated world, A system including means for booking the means of transportation or accommodation selected within the simulated world in the real world.

Images