system

Abstract

To provide a system.SOLUTION: A system comprises: means for enabling a user to experience a tourist spot from home with the usage of a VR technology; means for enabling commodity retrieval, commodity purchase, and settlement in a virtual shop of the tourist spot and communication with a virtual shop assistant of the virtual shop with the usage of a generation-based AI; means for analyzing an emotion of the user from a facial expression and a voice of the user with the usage of an emotion engine; means for generating a commodity recommendation based on a retrieval condition of the commodity retrieval and the emotion of the user with the usage of the generation-based AI; and means for realizing real-time communication with the virtual shop assistant.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a problem in that people who are short on money or time, people with disabilities, and people who cannot travel due to travel restrictions are unable to have a real travel experience. [Means for solving the problem]

[0005] To solve this problem, we provide a system that uses VR technology to allow users to experience tourist spots from home, and a system that uses generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks. Furthermore, the generative AI recommends products based on the user's purchase and search history, and the VR technology enables tracking of the user's movements and gaze, dynamically changing the scenery of tourist spots based on this, recreating a realistic travel experience. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15]FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, the processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0010] Examples include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0011] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0012] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g.,

[0013] hard disk), or magnetic tape.

[0014] In the following embodiments, the coded communication I / F (Interface) refers to a communication processor and

[0015] The communication I / F is an interface that includes a network interface (NIC) and an antenna. The communication I / F controls communication between multiple computers. Examples of communication standards that can be applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0016] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0017] [First embodiment]

[0018] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0019] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0021] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0022] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0023] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0024] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0025] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0026] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0027] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0028] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0029] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0030] "Example 1"

[0031] An example of the present invention is a system that allows users to experience tourist spots from their homes. Specifically, by using a VR headset, users can experience tourist spots through 360-degree panoramic images. In addition, the system tracks the user's line of sight and position information, and changes the image of the tourist spot accordingly.

[0032] The images change dynamically, allowing users to experience the sights as if they were walking through a tourist spot from the comfort of their own home.

[0033] "Example 2"

[0034] This invention also includes a system that uses generative AI to provide internet search tools, product purchases, payments, and communication with virtual store clerks in a virtual shop. Specifically, when a user searches for or purchases products in a virtual shop, the generative AI learns the user's behavior and recommends products that match the user's preferences from the next time. Communication with the virtual store clerk is also possible, allowing the user to ask questions about products and complete the purchase process. This allows the user to recreate the shopping experience of a real store from the comfort of their own home.

[0035] "Example 3"

[0036] As a concrete example, consider a case where a user experiences tourist attractions in Kyoto from the comfort of their own home. The user puts on a VR headset and experiences the sights of Kyoto through a 360-degree panoramic image. As the user moves their gaze, the image changes dynamically, giving the user the experience of walking through the area. The user can also search for Kyoto's famous products in a virtual shop and purchase products recommended by generative AI. By communicating with a virtual store clerk, the user can obtain detailed product information and complete the purchase process.

[0037] The processing flow of each embodiment will be described below.

[0038] "Example 1"

[0039] Step 1: The user puts on the VR headset.

[0040] Step 2: The system provides a 360-degree panoramic image for users to experience the tourist attractions.

[0041] Step 3: The system tracks the user's eye movements and location.

[0042] Step 4: The image of the tourist spot changes dynamically accordingly.

[0043] "Example 2"

[0044] Step 1: User searches for a product in the virtual shop.

[0045] Step 2: The generative AI learns the user's behavior and recommends products that match the user's preferences next time.

[0046] Step 3: The user communicates with the virtual store associate.

[0047] Step 4: The user gets more information about the product and completes the purchase.

[0048] "Example 3"

[0049] Step 1: If a user wants to experience Kyoto's tourist attractions from home, they first put on a VR headset.

[0050] Step 2: Experience Kyoto's tourist attractions through 360-degree panoramic images.

[0051] Step 3: As the user moves their gaze, the image changes dynamically.

[0052] Step 4: The user searches for Kyoto specialties in the virtual shop.

[0053] Step 5: Purchase the product recommended by the generative AI.

[0054] Step 6: Communicate with the virtual store associate to obtain more information about the product and complete the purchase.

[0055] Example 1

[0056] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0057] Traditional tourism experiences required users to actually go to the actual location, which was problematic in terms of time and expense. Even in virtual shops, users could not actually touch and feel the products, which often led to anxiety when making a purchase. Furthermore, there was a lack of systems that dynamically changed the images in response to the user's gaze and movements, preventing a realistic experience.

[0058] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0059] In this invention, the server includes: means for allowing users to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop; means for tracking the user's eye movements and location information and dynamically changing the scenery of the tourist spot based on the movements; means for acquiring 360-degree panoramic video data of the tourist spot and streaming it to the user's device; and means for displaying the video using a VR headset and rendering the video in real time based on the user's eye gaze data. This allows users to have a realistic tourist experience from the comfort of their own home and purchase products safely in the virtual shop.

[0060] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[0061] "Generative AI" stands for generative artificial intelligence, an artificial intelligence technology that generates new information and content based on user input and data.

[0062] A "virtual shop" is a virtual store operated on the Internet, which is a system that allows users to search for, purchase, and pay for products online.

[0063] An "Internet search tool" is software or an application used to search for information on the Internet, allowing users to efficiently find specific information.

[0064] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, answering their questions and recommending products.

[0065] "Eye movement" refers to the movement of the user's eyes, and is data that indicates the direction in which the user is looking.

[0066] "Location information" refers to data that indicates a user's physical location and is information used to identify where the user is located.

[0067] "360-degree panoramic video data" refers to video data that covers all directions, allowing users to look around in any direction.

[0068] "Streaming" is a technology that plays data while transferring it in real time, allowing users to use the data instantly without having to download it.

[0069] A "VR headset" is a device that allows users to experience a virtual reality environment, and when worn, it immerses them in a virtual space.

[0070] "Real-time rendering" refers to the technology by which a computer instantly generates and displays images and graphics, allowing images to be updated without delay in response to the user's movements.

[0071] This invention is a system that allows users to experience tourist spots from the comfort of their own homes, and is realized by combining VR technology and generative AI. Specific embodiments of this system are described below.

[0072] First, the server acquires 360-degree panoramic video data of tourist attractions. This data can be acquired from the Internet or a dedicated database. For example, high-resolution panoramic images of tourist attractions can be downloaded using the Google Street View API.

[0073] The server then streams the captured 360-degree panoramic video data to the user's device using protocols such as HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH).

[0074] The user wears a VR headset at home, and the device is connected to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[0075] The device uses sensors built into the VR headset to track the user's gaze and location. For example, it uses the headset's gyro sensor and accelerometer to detect the user's head movement. The acquired gaze data is analyzed and the image is dynamically updated to match the user's gaze. This allows the user to view different parts of the tourist attraction depending on their gaze.

[0076] Generative AI will enable internet search tools, product purchases and payments, and communication with virtual store staff in virtual shops. It will also have the ability to recommend products based on users' purchase and search history. This allows users to purchase products with confidence within the virtual shop.

[0077] As a concrete example, if a user wants to experience Kinkaku-ji Temple in Kyoto, the following operations take place: The server obtains 360-degree panoramic video data of Kinkaku-ji Temple using the Google Street View API and streams it to the user's device using the HTTP Live Streaming (HLS) protocol. The user puts on a VR headset, and the device displays the image of Kinkaku-ji Temple using a game engine. The device uses the VR headset's gyro sensor to track the user's eye movement; if the user moves their head to the right, the device detects this movement and shifts the image to the right. This allows the user to see different parts of Kinkaku-ji Temple depending on their eye movement, giving the user the experience of walking around the temple in person.

[0078] Example prompts to input to a generative AI model:

[0079] "Please explain how a user can experience Kinkaku-ji Temple from the comfort of their own home using a VR headset. Please include how the image changes dynamically in response to the user's gaze."

[0080] In this way, users can have a realistic tourist experience from the comfort of their own home, and can purchase products safely in virtual shops.

[0081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0082] Step 1:

[0083] The server acquires 360-degree panoramic video data of tourist spots.

[0084] Specifically, the server downloads high-resolution panoramic images of tourist spots from the Internet or a dedicated database, for example, using the Google Street View API.

[0085] Input: Name and location of tourist spot

[0086] Output: 360-degree panoramic video data

[0087] Step 2:

[0088] The server streams the acquired video data to the user's device.

[0089] Specifically, the server uses protocols such as HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) to send video data to the user's device in real time.

[0090] Input: 360-degree panoramic video data

[0091] Output: Video data streamed to the user's device

[0092] Step 3:

[0093] The user puts on the VR headset and the device displays the image.

[0094] Specifically, the user wears a VR headset at home, and the device connects to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[0095] Input: Streamed video data

[0096] Output: Image displayed on a VR headset

[0097] Step 4:

[0098] The device tracks the user's eye movements and location.

[0099] Specifically, the device uses the gyro sensor and acceleration sensor built into the VR headset to detect the user's head movements.

[0100] Input: User's eye movements and location information

[0101] Output: Gaze data and position data

[0102] Step 5:

[0103] The device dynamically updates the video based on gaze data.

[0104] Specifically, the device analyzes the acquired gaze data and dynamically updates the image to match the user's gaze. For example, if the user moves their head to the left, the device detects this movement and shifts the image to the left.

[0105] Input: Gaze data and position data

[0106] Output: Dynamically updated video

[0107] Step 6:

[0108] Users experience tourist destinations.

[0109] Specifically, users experience tourist spots through a VR headset. The images change dynamically according to the user's line of sight, giving the user the experience of walking through the tourist spots in real life.

[0110] Input: Dynamically updated video

[0111] Output: User's sightseeing experience

[0112] In this way, users can have a realistic tourist experience from the comfort of their own home.

[0113] (Application example 1)

[0114] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional tourism experience systems and virtual shop systems using virtual reality (VR) technology have faced issues such as a lack of realism and convenience when users experience tourist spots or purchase products from the comfort of their own homes. Furthermore, the lack of interactive experiences utilizing the user's gaze or location information limits the user experience. Furthermore, the display of product information and the purchasing process are cumbersome, requiring improvements to increase user satisfaction.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0117] In this invention, the server includes a means for allowing users to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks, and a means for tracking the user's gaze and location information and displaying detailed information when the user approaches a product. This allows users to enjoy a realistic tourist experience and virtual shopping from the comfort of their own home. Furthermore, an interactive experience utilizing gaze and location information is provided, and the display of product information and the purchase process are simplified, improving the user experience.

[0118] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[0119] "Generative AI" refers to generative artificial intelligence, which has the ability to generate new information and content based on user input and data.

[0120] A "virtual shop" is a virtual store that exists on the Internet, a virtual space where users can browse and purchase products online.

[0121] An "Internet search tool" is software or an application used to search for information on the Internet.

[0122] "Product purchase and payment" refers to the process by which a user selects a product, completes the purchase process, and then pays.

[0123] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, providing product information and support.

[0124] "Tracking gaze and location information" is a technology that detects the direction of the user's gaze and body position in real time, and the system operates based on that information.

[0125] "Show detailed information" means that when a user approaches a specific product, detailed product information such as a description, specifications, and reviews will be displayed on the screen.

[0126] To implement the present invention, the following system configuration and program are required.

[0127] First, the user puts on a VR headset and experiences tourist attractions and virtual shops from the comfort of their own home. The server then uses VR technology to provide the means for the user to experience the tourist attractions from home. This means includes the ability to generate 360-degree panoramic images and dynamically change the images by tracking the user's line of sight and location. Specifically, the images are rendered in real time using a game engine.

[0128] Next, the server uses generative AI to provide internet search tools for the virtual shop, as well as the means to purchase and pay for products and communicate with virtual store staff. The generative AI has the ability to recommend products based on the user's purchase and search history, allowing users to easily find products that suit them. Natural language processing models such as GPT-3 (registered trademark) are used for generative AI.

[0129] Furthermore, the server tracks the user's gaze and location information, providing a means to display detailed information about a product when the user approaches it. This means includes gaze tracking technology and location tracking technology. For gaze tracking, hardware such as an eye tracker is used, and for location tracking, sensors built into the VR headset are used. As a result, when the user directs their gaze at a specific product, detailed information about that product is displayed.

[0130] As a concrete example, consider a scene in which a user finds a "smartphone" in a virtual shop and checks its detailed information. The user puts on a VR headset and walks around the virtual shop. When the user directs their gaze towards the smartphone, detailed information about the smartphone (e.g., "The latest model smartphone. Equipped with a high-resolution camera and a long-lasting battery.") is displayed. An example of a prompt sentence to realize this scene is as follows:

[0131] Example prompt sentence:

[0132] The user puts on a VR headset and walks around the virtual shop. When the user looks at the smartphone, detailed information about the smartphone (e.g., "Latest model smartphone. Equipped with a high-resolution camera and long-lasting battery life.") is displayed. Please generate a game engine C script to realize this scene.

[0133] In this way, users can enjoy realistic sightseeing experiences and virtual shopping from the comfort of their own homes, and the interactive experience using gaze and location information is provided, simplifying product information display and purchasing procedures, improving the user experience.

[0134] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0135] Step 1:

[0136] The user puts on the VR headset and launches the virtual shop application.

[0137] Input: User actions

[0138] Output: VR headset boots, virtual shop application starts

[0139] Specific operation: The user puts on a VR headset and presses a button to launch the application, which starts the VR headset and launches the virtual shop application.

[0140] Step 2:

[0141] The server tracks the user's gaze and location in real time.

[0142] Input: Gaze and position data from the VR headset

[0143] Output: User's gaze direction and position information

[0144] How it works: The sensors built into the VR headset detect the user's gaze direction and location information and send that data to the server, which processes this data in real time to determine the user's gaze direction and location information.

[0145] Step 3:

[0146] The server identifies the product in the user's line of sight and obtains its detailed information.

[0147] Input: User's gaze direction and position information

[0148] Output: Detailed information about the product in the line of sight

[0149] Specific operation: The server identifies the product that the user is looking at based on the user's gaze direction and location information, and retrieves detailed information about the identified product (e.g., product name, description, price, reviews, etc.) from the database.

[0150] Step 4:

[0151] The server sends the obtained detailed information to the VR headset and displays it to the user.

[0152] Input: Product details

[0153] Output: Product details displayed in a VR headset

[0154] Specific operation: The server sends the acquired product details to the VR headset, which displays the received information in the user's field of view, allowing the user to check the details.

[0155] Step 5:

[0156] The user selects a product and completes the purchase procedure.

[0157] Input: User selection

[0158] Output: Purchase completed

[0159] Specific operation: The user selects a product using their eyes or a controller and presses the purchase button. The server receives the user's selection and begins the purchase process. The server then enters payment information and completes the purchase process.

[0160] Step 6:

[0161] The server updates the purchase history, and the generative AI stores the data for the next product recommendation.

[0162] Input: Purchase history data

[0163] Output: Updated purchase history, saved data

[0164] How it works: The server updates the user's purchase history to a database, and the generative AI saves the data necessary for the next product recommendation, allowing it to recommend products that are appropriate for the user's next shopping experience.

[0165] Example 2

[0166] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Conventional virtual shop systems have struggled to recreate the shopping experience of a real store from the comfort of your own home. They also lacked the functionality to learn user behavior patterns and recommend the best products for each individual user. Furthermore, when communicating with virtual store clerks, it was difficult to provide appropriate answers to user questions. To solve these issues, it is necessary to effectively collect and analyze user behavior data and utilize generative AI models.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0169] In this invention, the server includes: means for allowing a user to experience tourist spots from home; means for using a generative AI to enable internet search tools, product purchases and payments in a virtual shop, and communication with a virtual store clerk; means for collecting user behavior data and learning the user's behavior patterns using a generative AI model; means for recommending products based on the user's behavior patterns; means for using the generative AI model to preferentially display highly relevant products when the user searches for a product; and means for using the generative AI model to generate appropriate answers when the user asks a question to the virtual store clerk. This allows users to recreate the shopping experience in an actual store from the comfort of their own home, making it possible to recommend products that are best suited to each user and provide appropriate answers when communicating with the virtual store clerk.

[0170] "User" means an individual or organization that uses the system to search for products in the virtual shop, purchase products, make payments, and communicate with virtual store staff.

[0171] The "server" is a computer system that collects, stores, and analyzes user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[0172] "Generative AI" is an artificial intelligence technology that learns from user behavior data and recommends products that match the user's preferences, or acts as a virtual store clerk to generate appropriate answers to the user's questions.

[0173] A "virtual shop" is a virtual store on the Internet where users can search for, browse, purchase, and pay for products.

[0174] "Behavioral data" refers to information about the operations performed by users in the virtual shop, such as search history, purchase history, and browsing history.

[0175] "Behavioral patterns" are data that indicate a user's preferences and tendencies, learned by generative AI based on user behavior data.

[0176] "Product recommendation" is a function in which generative AI presents appropriate products to users based on their behavioral patterns.

[0177] A "virtual store clerk" is a virtual store clerk that uses generative AI to generate appropriate answers to users' questions and communicates with them.

[0178] "Internet Search Tool" is a function that allows users to search for products within the virtual shop.

[0179] "Checkout" is the process by which a user makes payment for the products selected in the virtual shop.

[0180] A "prompt sentence" is text data input to a generative AI model, and is an instruction sentence that enables the AI ​​to generate an appropriate response.

[0181] MODE FOR CARRYING OUT THE INVENTION

[0182] This invention is a virtual shop system using generative AI, which aims to recreate the shopping experience of a real store from the comfort of your own home. The system collects user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[0183] Hardware and software used

[0184] Server: A computer system that collects, stores, and analyzes user behavior data and uses generative AI models to learn user behavior patterns. Specifically, it uses a high-performance database server and a GPU server to run the AI ​​model.

[0185] Terminal: A device that allows users to access the virtual shop, search for products, view them, purchase them, make payments, and communicate with virtual store staff. Specifically, this includes PCs, smartphones, tablets, etc.

[0186] Generative AI model: An artificial intelligence technology that learns from user behavior data to recommend products that match the user's preferences and generate appropriate answers to user questions as a virtual store clerk. Specifically, it uses natural language processing models such as GPT-4 (registered trademark).

[0187] Specific operation of the system

[0188] 1. Collecting User Behavior Data:

[0189] Users can access the virtual shop to search and browse products.

[0190] The device records the user's operations in real time and transmits them to the server.

[0191] The server stores the received data in a database, specifically recording information such as the user ID, search keywords, viewed product ID, and viewing time.

[0192] 2. Learning user behavior patterns:

[0193] The server periodically extracts user behavior data from the database.

[0194] The server uses a generative AI model (e.g., GPT-4) to analyze user behavior patterns, specifically learning what products users prefer and what time of day they access the site.

[0195] The server stores the learning results as a user profile.

[0196] 3. Perform a product search:

[0197] Users enter the product name or category into the virtual shop's search bar.

[0198] The terminal transmits the input search keyword to the server.

[0199] The server searches the database and generates a list of relevant products, specifically retrieving product IDs, product names, prices, stock status, and other information that matches the search keywords.

[0200] 4. Product recommendation:

[0201] The server prioritizes the display of relevant products based on the user's behavioral patterns. Specifically, it matches the user profile with search results and places products that match the user's preferences at the top of the list.

[0202] The terminal displays the product list received from the server to the user.

[0203] 5. Product Purchase and Payment:

[0204] Users select the products they want to purchase and add them to their cart.

[0205] The terminal transmits the cart information to the server.

[0206] The server manages the cart information and proceeds with the purchase process. Specifically, it checks the user's shipping address and payment information and processes the payment.

[0207] The user enters payment information and confirms the purchase.

[0208] The server notifies the user that the payment process is complete.

[0209] 6. Communication with virtual store associates:

[0210] Users can use the chat function within the virtual shop to ask questions to the virtual store clerk.

[0211] The terminal sends the user's question to the server.

[0212] The server uses a generative AI model to generate appropriate answers to user questions. Specifically, it analyzes the content of the question and generates answers from related product information and an FAQ database.

[0213] The terminal displays the response received from the server to the user.

[0214] Examples of specific examples and prompts

[0215] Examples:

[0216] When a user searches for "smartphone," the device sends the keyword "smartphone" to the server.

[0217] The server searches the database and generates a list of products related to "smartphone," such as "smartphone A" and "smartphone B."

[0218] The server displays "Smartphone A" at the top of the list based on the user's past search and purchase history.

[0219] When a user asks the virtual store clerk, "How long is the battery life of this smartphone?", the device sends the question to the server.

[0220] The server uses a generative AI model to generate an answer such as "This smartphone's battery life is approximately 24 hours" and sends it to the device.

[0221] The terminal displays the response received from the server to the user.

[0222] Example prompt sentence:

[0223] "When a user searches on their smartphone, how can I display recommended products based on their past search history?"

[0224] "Create a prompt that will allow the virtual store clerk to generate an appropriate answer to the user's question."

[0225] In this way, users can recreate the shopping experience of a real store from the comfort of their own home.

[0226] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0227] Step 1: Collecting user behavior data

[0228] Users can access the virtual shop to search and browse products.

[0229] The device records the user's operations in real time and transmits them to the server.

[0230] Input: User operation information (search keywords, viewed product ID, viewing time, etc.)

[0231] The server stores the received data in a database.

[0232] Output: User behavior data stored in a database

[0233] Step 2: Learn user behavior patterns

[0234] The server periodically extracts user behavior data from the database.

[0235] Input: User behavior data stored in a database

[0236] The server uses a generative AI model (e.g., GPT-4) to analyze user behavior patterns, specifically learning what products users prefer and what time of day they access the site.

[0237] Data processing: behavioral data analysis, pattern recognition

[0238] The server stores the learning results as a user profile.

[0239] Output: User profile

[0240] Step 3: Perform a product search

[0241] Users enter the product name or category into the virtual shop's search bar.

[0242] Input: Search keyword

[0243] The terminal transmits the input search keyword to the server.

[0244] The server searches the database and generates a list of relevant products, specifically retrieving product IDs, product names, prices, stock status, and other information that matches the search keywords.

[0245] Data processing: Keyword search, product information extraction

[0246] Output: Product list

[0247] Step 4: Product Recommendation

[0248] The server prioritizes the display of relevant products based on the user's behavioral patterns. Specifically, it matches the user profile with search results and places products that match the user's preferences at the top of the list.

[0249] Input: User profile, product list

[0250] Data processing: Matching profiles with search results and prioritizing them

[0251] The terminal displays the product list received from the server to the user.

[0252] Output: A prioritized list of products

[0253] Step 5: Purchase and payment

[0254] Users select the products they want to purchase and add them to their cart.

[0255] Input: Selected product information

[0256] The terminal transmits the cart information to the server.

[0257] The server manages the cart information and proceeds with the purchase. Specifically, it checks the user's shipping address and payment information and processes the payment.

[0258] Data processing: cart information management, payment processing

[0259] The user enters payment information and confirms the purchase.

[0260] The server notifies the user that the payment process is complete.

[0261] Output: Purchase completion notification

[0262] Step 6: Communicate with the virtual sales associate

[0263] Users can use the chat function within the virtual shop to ask questions to the virtual store clerk.

[0264] Input: User question

[0265] The terminal sends the user's question to the server.

[0266] The server uses a generative AI model to generate appropriate answers to user questions. Specifically, it analyzes the content of the question and generates answers from related product information and an FAQ database.

[0267] Data processing: Question analysis and answer generation

[0268] The terminal displays the response received from the server to the user.

[0269] Output: Answer from the virtual store clerk

[0270] (Application example 2)

[0271] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Conventional virtual shop systems have had difficulty recreating the shopping experience of a real store from the comfort of your own home. Furthermore, they lacked the functionality to recommend products that matched the user's preferences or the ability to communicate with virtual store staff. This led to issues such as low user satisfaction and a lack of promotion of virtual shop use.

[0273] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0274] In this invention, the server includes a means for allowing users to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with a virtual store clerk, a means for learning the user's behavior and recommending products that suit the user's preferences from the next time, and a means for asking questions about products and completing purchase procedures through a chat function with the virtual store clerk. This allows users to recreate the shopping experience of an actual store from the comfort of their own home, and further enables product recommendations that suit the user's preferences and smooth communication with the virtual store clerk.

[0275] "VR technology" refers to virtual reality technology, which allows users to experience a virtual environment generated by a computer.

[0276] "Generative AI" refers to artificial intelligence that uses generative models, a technology that generates new information and content based on user behavior and data.

[0277] A "virtual shop" is a virtual store that exists on the Internet, where users can search for, purchase, and pay for products online.

[0278] An "Internet search tool" is software or an application used to search for information on the Internet.

[0279] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[0280] "Payment" refers to the process of paying for goods or services.

[0281] A "virtual store clerk" is an artificial intelligence or avatar that communicates with users within a virtual shop.

[0282] "Learning user behavior" is a technology that analyzes a user's past behavioral data (such as search history and purchase history) and predicts future behavior.

[0283] "Product recommendation" is the act of suggesting products suitable for a user based on the user's preferences and past behavioral data.

[0284] The "chat function" is a function for communicating in real time using text and voice.

[0285] "Questions and purchasing procedures" refers to the user's actions of asking a virtual store clerk a question about a product or completing procedures for purchasing.

[0286] A system for implementing this invention includes a server, a user terminal, and a generative AI model. The server collects and stores user behavior data and uses the generative AI model to recommend products and communicate with virtual store clerks. The user terminal is a device such as a smartphone or tablet, and provides an interface for users to access the virtual shop and perform product searches, purchases, and payments.

[0287] Server Processing

[0288] The server collects user behavior data (search history, purchase history, etc.) and saves it in JSON format. This allows it to learn the user's preferences and behavioral patterns. The server uses a generative AI model (e.g., GPT-3 by OpenAI (registered trademark)) to recommend products based on the user behavior data. Specifically, it sends the following prompt to the generative AI model:

[0289] User behavior data: {"Search history": ["Smartphone", "Laptop"], "Purchase history": ["Earphones"]}

[0290] Please tell me what products you recommend.

[0291] The generative AI model will then recommend suitable products to the user based on this prompt.

[0292] User terminal processing

[0293] The user terminal provides a virtual shop interface and supports users in searching, purchasing, and paying for products. If a user has a question about a product, they can send it to the virtual shop assistant through a chat function. For example, they can send the following question to the virtual shop assistant:

[0294] I'm looking for a new smartphone. Any recommendations?

[0295] The generative AI model generates an appropriate answer to this question and provides it to the user.

[0296] Hardware and software used

[0297] Hardware: Servers, smartphones, tablets

[0298] Software: OpenAI API, Python, JSON

[0299] Specific examples

[0300] Consider a case where a user accesses a virtual shop using their smartphone and searches for a smartphone. When the user asks the virtual shop assistant, "I'm looking for a new smartphone. Do you have any recommendations?", the generative AI model will recommend an appropriate smartphone based on the user's past behavioral data. In this way, the user can recreate the shopping experience of a real store from the comfort of their own home.

[0301] This system allows users to recreate the shopping experience of a real store from the comfort of their own home, and also enables product recommendations tailored to the user's preferences and smooth communication with virtual store clerks.

[0302] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0303] Step 1:

[0304] A user accesses the virtual shop using a terminal.

[0305] Input: User login information

[0306] Output: Virtual shop home screen

[0307] Specific operation: A user launches the virtual shop application on a smartphone or tablet and enters their login information. The server authenticates the user and displays the home screen.

[0308] Step 2:

[0309] The user searches for a product.

[0310] Input: Search keyword (e.g. "smartphone")

[0311] Output: A list of search results

[0312] Specific operation: The user enters "smartphone" in the search bar and presses the search button. The server retrieves related product information from the database and displays the search results on the user's device.

[0313] Step 3:

[0314] The server collects and stores user behavior data.

[0315] Input: User search history, purchase history

[0316] Output: JSON format behavior data file

[0317] How it works: Every time a user searches for or purchases a product, the server collects that information and stores it in JSON format. For example, "smartphone" is saved as a search history and "earphones" as a purchase history.

[0318] Step 4:

[0319] The server recommends products using a generative AI model.

[0320] Input: User behavior data (JSON format)

[0321] Output: A list of recommended products

[0322] Specific operation: The server reads the saved behavioral data and sends a prompt to the generative AI model. For example, it sends a prompt like, "User behavioral data: {"Search history": ["Smartphone"], "Purchase history": ["Earphones"]} Please recommend some products." The generative AI model recommends products based on this prompt and returns the results to the server.

[0323] Step 5:

[0324] The user uses the chat function with the virtual store clerk.

[0325] Input: User question (e.g., "I'm looking for a new smartphone. Any recommendations?")

[0326] Output: Answer from the virtual store clerk

[0327] Specific operation: The user enters a question in the chat window and presses the send button. The server sends this question to the generative AI model, which generates an appropriate answer. The generated answer is displayed on the user's device.

[0328] Step 6:

[0329] The user purchases the product and makes the payment.

[0330] Input: Select the product you want to purchase, payment information

[0331] Output: Purchase confirmation message

[0332] Specific operation: The user selects the product they want to purchase and adds it to their cart. They then enter their payment information, such as credit card information, on the payment screen and press the purchase button. The server processes the payment and displays a confirmation message on the user's device that the purchase has been completed.

[0333] Step 7:

[0334] The server updates the user's behavior data.

[0335] Input: New search history, purchase history

[0336] Output: Updated JSON behavior data file

[0337] Specific operation: Every time a user searches for or purchases a new product, the server collects that information and updates the existing behavioral data file. For example, "laptop" is added to the user's search history and "smartphone" is added to the user's purchase history.

[0338] Example 3

[0339] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional tourism experience systems using VR technology lack a sense of realism due to insufficient user gaze tracking and dynamic image changes. Furthermore, when searching for and purchasing products in virtual shops, product recommendations and detailed information tailored to user needs are insufficient. This leads to issues such as lower user satisfaction and a decrease in system usage.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0342] In this invention, the server includes means for allowing a user to experience a tourist spot from home, means for using a generative AI model to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop, means for tracking the user's gaze and dynamically changing the scenery of the tourist spot based on that gaze, and means for using the generative AI model to recommend products based on the user's search query. This allows the user to experience images that change in real time according to their gaze movement, and enables them to receive more appropriate product recommendations and detailed information when searching and purchasing in the virtual shop.

[0343] "User" refers to an individual who uses the system to engage in tourism experiences or purchase products.

[0344] "Tourist destinations" refer to places that users can virtually visit using VR technology.

[0345] A "generative AI model" refers to an artificial intelligence algorithm that recommends products based on a user's search queries and history.

[0346] A "virtual shop" refers to an online store where users can search for and purchase products in a virtual space.

[0347] "Internet Search Tool" refers to a function that allows users to search for products within the virtual shop.

[0348] "Product purchase" refers to the act of a user purchasing a product selected within a virtual shop.

[0349] "Payment" refers to the payment procedure carried out by a User when purchasing a Product.

[0350] A "virtual store clerk" refers to a virtual character that communicates with users within a virtual shop and provides product information.

[0351] "Eye tracking" refers to a technology that detects a user's eye movements in real time and transmits that data to a system.

[0352] "Dynamic change" refers to updating images and information in real time according to the user's gaze and movements.

[0353] "360-degree panoramic video" refers to video that allows users to see in all directions.

[0354] "Search query" refers to the keywords or phrases that users enter when searching for products within a virtual shop.

[0355] "Recommending products" refers to the generative AI model suggesting appropriate products based on a user's search query and history.

[0356] This invention is a system that allows users to experience tourist spots from their homes and search for and purchase products in virtual shops. Specific embodiments of this system are described below.

[0357] Hardware and Software

[0358] Hardware

[0359] VR headset: A device worn by a user, such as a VR headset, that allows the user to immerse themselves in a virtual space.

[0360] Eye tracking sensor: Built into the VR headset, it tracks the user's gaze in real time.

[0361] software

[0362] 360-degree panoramic video playback software: When the user moves their gaze, the video changes dynamically.

[0363] Generative AI models: An example is GPT-4, which recommends products based on a user's search queries and history.

[0364] Data processing and calculation

[0365] User gaze tracking

[0366] The device (VR headset) uses a built-in eye tracking sensor to track the user's gaze in real time. The gaze data is processed within the device and sent to a server.

[0367] Dynamic changes in images

[0368] The server receives gaze data sent from the device. Based on the gaze data, the server calculates a 360-degree panoramic image corresponding to the direction the user is looking. For example, if the user looks left, the server calculates the image of the left side. The calculated image is streamed to the device in real time.

[0369] Search and buy in the virtual shop

[0370] A user visits a virtual shop in a VR space and enters a product name (e.g., Yatsuhashi) in the search bar. The search query is sent to a server. The server uses a generative AI model (e.g., GPT-4) to recommend products based on the user's search query. The user asks the virtual store clerk a question via voice or text, for example, "What is the expiration date of this Yatsuhashi?" The server uses the generative AI model to generate a response from the virtual store clerk and provides it to the user. The user selects the product they want to purchase and completes the purchase. The server processes the payment information and notifies the user that the purchase is complete.

[0371] Specific examples

[0372] The user puts on a VR headset at home and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user moves their gaze to the left, the device sends gaze data to the server, which then calculates and streams a video of the stage at Kiyomizu-dera Temple on the left. Next, when the user searches for "yatsuhashi" in a virtual shop, the server uses GPT-4 to recommend "recommended yatsuhashi." The user asks the virtual store clerk, "What is the expiration date of this yatsuhashi?" and receives the response generated by the server. Finally, the user selects a product and completes the purchase process.

[0373] Prompt Sentence Examples

[0374] "The user puts on a VR headset and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user moves their gaze to the left, the image changes to show the stage of Kiyomizu-dera Temple. Next, when the user searches for 'yatsuhashi' in the virtual shop, the generative AI model recommends 'recommended yatsuhashi'. The user asks the virtual shop assistant, 'What is the expiration date of this yatsuhashi?' After obtaining the detailed information, they proceed with the purchase."

[0375] In this way, the user can experience Kyoto's tourist spots in real life and purchase local specialties from the comfort of their own home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0376] Step 1:

[0377] The user puts on the VR headset.

[0378] Input: The user puts on the VR headset.

[0379] Output: The user is connected to the virtual space.

[0380] What happens: The user puts on a VR headset at home and logs into the system, ready to be immersed in the virtual space.

[0381] Step 2:

[0382] The device tracks the user's gaze.

[0383] Input: User's eye movements.

[0384] Output: Gaze data is generated and sent to the server.

[0385] How it works: The eye-tracking sensor built into the VR headset detects the user's gaze in real time, processes the data on the device, and sends it to a server.

[0386] Step 3:

[0387] The server receives the gaze data and calculates the corresponding 360-degree panoramic image.

[0388] Input: Gaze data sent from the device.

[0389] Output: Computed 360-degree panoramic image.

[0390] Specific operation: The server receives the gaze data and calculates the 360-degree panoramic image corresponding to the direction the user is looking. For example, if the user looks left, the server calculates the image of the left side.

[0391] Step 4:

[0392] The server streams the calculated video to the device.

[0393] Input: Computed 360° panoramic image.

[0394] Output: The video is displayed in the user's VR headset.

[0395] How it works: The server streams the calculated 360-degree panoramic image to the device in real time and displays it on the user's VR headset, allowing the user to experience a dynamic change in the image according to their line of sight.

[0396] Step 5:

[0397] A user searches for a product in a virtual shop.

[0398] Input: The product name entered by the user in the search bar (e.g., Yatsuhashi).

[0399] Output: The search query is sent to the server.

[0400] Specific operation: A user accesses a virtual shop in a VR space and enters a product name in the search bar. The search query is sent to the server.

[0401] Step 6:

[0402] The server uses the generative AI model to recommend products.

[0403] Input: The user's search query.

[0404] Output: A list of recommended products.

[0405] How it works: The server uses a generative AI model (e.g., GPT-4) to recommend products based on the user's search query. For example, if you search for "yatsuhashi," the server will list "recommended yatsuhashi."

[0406] Step 7:

[0407] The user communicates with a virtual store clerk.

[0408] Input: User question (e.g., What is the expiration date of this yatsuhashi?).

[0409] Output: Response from the virtual store clerk.

[0410] Specific operation: The user asks a question to a virtual clerk in the virtual shop by voice or text. The server uses the generative AI model to generate a response from the virtual clerk and provides it to the user.

[0411] Step 8:

[0412] The user purchases the product.

[0413] Input: User selected product and payment information.

[0414] Output: Confirmation of successful purchase.

[0415] Specific behavior: The user selects the product they want to purchase and completes the purchase. The server processes the payment information and notifies the user that the purchase is complete.

[0416] (Application example 3)

[0417] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0418] Conventional tourism experience systems using virtual reality (VR) technology often lack a sense of visual immersion when users experience tourist destinations from home. Furthermore, product search and purchasing procedures in virtual shops lacked sufficient product recommendations based on user preferences and detailed information. Furthermore, communication with virtual store clerks was limited, making it difficult to improve the user experience.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for allowing a user to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop, a means for dynamically changing images according to the user's line of sight, a means for the generative AI to recommend products based on the user's preferences, and a means for providing detailed product information through dialogue with a virtual store clerk. This allows a user to experience tourist spots with a high level of immersion even from the comfort of their own home, and improves the shopping experience in the virtual shop.

[0420] "VR technology" is a technology for realizing virtual reality, allowing users to immerse themselves in a computer-generated virtual environment and experience the same as the real world.

[0421] "Generative AI" is a type of artificial intelligence that has the ability to generate new information and content based on user input and data.

[0422] A "virtual shop" is a virtual store that exists on the Internet, where users can search, browse and purchase products online.

[0423] An "Internet search tool" is software or an application for searching information on the Internet, allowing users to efficiently find specific information.

[0424] "Product purchase and payment" refers to the process in which a user purchases a selected product and pays for it.

[0425] A "virtual store clerk" is a virtual assistant that interacts with users in a virtual shop, providing information about products and assisting them in the purchasing process.

[0426] "Means for dynamically changing images according to the user's line of sight" refers to technology that tracks the user's line of sight and changes the displayed image in real time according to the direction of that gaze.

[0427] "Means for recommending products based on user preferences" refers to technology that analyzes a user's past purchase history, search history, and other data to automatically suggest products that the user may be interested in.

[0428] "Means for providing detailed product information through dialogue with a virtual store clerk" refers to technology that allows users to obtain detailed information about a product, such as its features, specifications, and usage, by dialogue with a virtual store clerk.

[0429] A system for implementing the present invention has the following configuration.

[0430] System Configuration

[0431] 1. Hardware

[0432] VR headset: A device that allows a user to immerse themselves in a virtual reality environment. An example is a VR headset.

[0433] Camera: A device for tracking the user's gaze.

[0434] Server: A computer system for processing data and running generative AI models.

[0435] 2. Software

[0436] OpenAI API: An API for running generative AI models.

[0437] OpenCV: A library for image processing.

[0438] Flask: A web framework used to build server-side applications that allow interaction with virtual store associates.

[0439] Data processing and calculation

[0440] 1. Importing 360-degree panoramic footage

[0441] The server uses OpenCV to load the 360-degree panoramic video, which is displayed when the user puts on the VR headset.

[0442] 2. Dynamic image changes according to the user's gaze

[0443] The camera tracks the user's gaze and sends the data to a server, which uses OpenCV to dynamically change the image based on the user's gaze.

[0444] 3. Product Recommendations Using Generative AI

[0445] The server uses the OpenAI API to recommend products based on the user's preferences, analyzing the user's purchase and search history to suggest suitable products.

[0446] 4. Interacting with a virtual store clerk

[0447] The server uses Flask to interact with the virtual store clerk, receives messages from the user, and uses the OpenAI API to generate responses from the virtual store clerk.

[0448] Specific examples

[0449] When a user types, "I'm looking for Kyoto specialties," the generative AI responds, "The recommended specialties are yatsuhashi and matcha." When a user types, "Please tell me more about yatsuhashi," the virtual store clerk provides more information, saying, "Yatsuhashi is a traditional Japanese sweet from Kyoto, characterized by its cinnamon-flavored bean paste."

[0450] Prompt Sentence Examples

[0451] User: Looking for Kyoto specialties

[0452] Virtual Salesperson: Recommended local products are Yatsuhashi and Matcha.

[0453] User: Please tell me more about Yatsuhashi

[0454] Virtual Salesperson: Yatsuhashi is a traditional Japanese sweet from Kyoto, characterized by its cinnamon-flavored bean paste.

[0455] In this way, users can experience tourist destinations in a highly immersive way from the comfort of their own home, and the shopping experience in virtual shops can also be improved.

[0456] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0457] Step 1:

[0458] The server loads the 360-degree panoramic video using OpenCV. The input is the file path of the 360-degree panoramic video, and the output is the loaded image data. Specifically, the server loads the image file from the specified file path and stores the image data in memory.

[0459] Step 2:

[0460] The camera tracks the user's gaze and sends the data to the server. The input is the user's gaze data, and the output is the gaze data sent to the server. Specifically, the camera detects the user's gaze direction in real time and sends the data to the server.

[0461] Step 3:

[0462] The server uses OpenCV to dynamically change the image according to the user's gaze. The input is gaze data and loaded image data, and the output is image data shifted according to the gaze. Specifically, the server shifts the image based on the gaze data and generates an image that corresponds to the direction the user is looking.

[0463] Step 4:

[0464] The server uses the OpenAI API to recommend products based on the user's preferences. The input is the user's purchase history and search history, and the output is a list of recommended products. Specifically, the server analyzes the user's history data and sends prompts to the generative AI model to receive product recommendations.

[0465] Step 5:

[0466] The server uses Flask to realize interaction with the virtual store clerk. The input is a message from the user, and the output is the response of the virtual store clerk. Specifically, the server receives a message from the user, sends a prompt to the OpenAI API to generate a response for the virtual store clerk, and returns it to the user.

[0467] Step 6:

[0468] The user wears a VR headset and watches the video sent from the server. The input is the shifted image data sent from the server, and the output is the user's visual experience. Specifically, the user experiences a dynamically changing image according to their line of sight through the VR headset.

[0469] Step 7:

[0470] A user searches for a product in a virtual shop and purchases a product recommended by a generative AI. The input is the user's search query and a list of recommended products, and the output is the completion of the purchase process. Specifically, the user searches for a product in the virtual shop, selects a recommended product, and then completes the purchase process.

[0471] Step 8:

[0472] The user obtains detailed product information through a dialogue with a virtual store clerk. The input is a question from the user and the response from the virtual store clerk, and the output is detailed product information. Specifically, the user asks a question to the virtual store clerk and obtains detailed product information through the response generated by the generative AI model.

[0473] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0474] "Example 1"

[0475] In one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and optimizes the sightseeing experience based on the emotions. Specifically, when a user feels joy, the system provides scenery of a tourist spot that amplifies the joy. When a user feels depressed, the system provides scenery that soothes the user's mood.

[0476] "Example 2"

[0477] Furthermore, the emotion engine can be used to recommend products in a virtual shop. When a user feels happy, the system will recommend products that will increase that happiness. When a user feels depressed, the system will recommend products that will ease that mood.

[0478] "Example 3"

[0479] As a concrete example, consider the case where a user experiences Kyoto's tourist attractions from home. The user puts on a VR headset and experiences Kyoto's tourist attractions through a 360-degree panoramic image. At this time, the emotion engine senses the user's joy and dynamically changes the image according to that joy. For example, if the user feels joy from a particular view, that view will be displayed more vividly. Furthermore, in the virtual shop, Kyoto's famous products will be recommended according to the user's joy.

[0480] The processing flow of each embodiment will be described below.

[0481] "Example 1"

[0482] Step 1: The user puts on the VR headset and starts the system.

[0483] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[0484] Step 3: The emotion engine dynamically changes the scenery of the tourist spot based on the user's emotions.

[0485] "Example 2"

[0486] Step 1: A user visits a virtual shop.

[0487] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[0488] Step 3: The sentiment engine recommends products based on user sentiment.

[0489] "Example 3"

[0490] Step 1: The user puts on a VR headset and experiences Kyoto's tourist attractions.

[0491] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[0492] Step 3: The emotion engine dynamically changes the video based on the user's emotions.

[0493] Step 4: In the virtual shop, the emotion engine recommends Kyoto specialties based on the user's emotions.

[0494] Example 1

[0495] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0496] Conventional VR tourism experience systems were unable to dynamically change the scenery based on eye movements or location information when users experienced tourist spots from home. They also lacked the ability to recognize users' emotions and optimize the tourism experience based on those emotions. Furthermore, they lacked a means to provide integrated internet search tools, product purchase and payment options, and communication with virtual store staff in virtual shops. To address these issues, the present invention provides a new system.

[0497] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0498] In this invention, the server includes: means for allowing a user to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; means for tracking the user's eye movements and location information and dynamically changing the scenery of the tourist spot based on that; and means for recognizing the user's emotions and optimizing the tourist experience based on those emotions. This allows the user to have a tourist experience that changes dynamically based on their eye movements and location information from the comfort of their own home, and further enables them to enjoy an optimal tourist experience according to their emotions.

[0499] "VR technology" is a technology for realizing virtual reality, allowing users to have a reality-like experience in a virtual space.

[0500] "Generative AI" is a type of artificial intelligence that generates new information and content based on user input and past data.

[0501] A "virtual shop" is a virtual store operated on the Internet, an online platform where users can search for, purchase, and pay for products.

[0502] "Gaze movement" refers to the movement of the user's eyes and is information that indicates the direction in which the user is looking.

[0503] "Location information" is data that indicates the user's physical location and is information used to identify where the user is in the virtual space.

[0504] "Emotion recognition" is a technology that reads emotions from a user's facial expressions, voice, etc., and is a means of determining the user's emotional state.

[0505] "Tourism experience optimization" is the process of adjusting video and audio to enhance the tourism experience based on user emotions and behavior.

[0506] This invention is a VR system that allows users to experience tourist destinations from the comfort of their own homes. Users wear a VR headset and can experience tourist destinations through 360-degree panoramic images. The system also tracks the user's gaze and location, dynamically changing the images of the tourist destinations. Furthermore, by incorporating an emotion engine, the system optimizes the tourist experience based on the user's emotions.

[0507] Hardware and software used

[0508] Hardware: VR headset, eye tracking sensor, position tracking sensor

[0509] Software: Game engine, emotion recognition software

[0510] Data processing and calculation

[0511] Eye tracking and location acquisition

[0512] When a user puts on a VR headset, eye-tracking and position-tracking sensors capture the user's eye movements and location in real time, determining in which direction the user is looking and where they are.

[0513] Dynamic changes in images

[0514] Using a game engine, the system dynamically generates images of tourist spots based on the acquired gaze and position information. For example, if the user turns left, the image changes to show the scenery on the left.

[0515] Emotion Recognition and Experience Optimization

[0516] Using emotion recognition software (such as Affectiva), the system recognizes emotions from the user's facial expressions and voice. Based on the recognized emotions, the system optimizes the video and audio of tourist spots. For example, if the user is happy, the system will provide scenery and music that amplify that happiness.

[0517] Specific examples

[0518] Example 1

[0519] A user puts on a VR headset and experiences the Eiffel Tower in Paris. When the user looks up, the device receives that information and the server generates an image of the top of the Eiffel Tower. The device displays that image on the VR headset. When the user smiles, the device receives that information and the server generates an illuminated night view of the Eiffel Tower. The device displays that image on the VR headset.

[0520] Example 2

[0521] When a user is feeling depressed, the device receives this information, and the server generates a video of a tranquil Japanese garden and calming music. The device then displays the video and music on the VR headset, which helps soothe the user's mood.

[0522] Prompt Sentence Examples

[0523] Prompt 1: "Generate a scenario in which a user puts on a VR headset and experiences the Eiffel Tower in Paris. When the user looks up, the top of the Eiffel Tower should be visible, and when the user smiles, the Eiffel Tower should be illuminated at night."

[0524] Prompt 2: "Generate a scenario that displays a video of a tranquil Japanese garden when the user is feeling depressed. Also play calming music to soothe the user's mood."

[0525] This system allows users to experience sightseeing spots from the comfort of their own home, as if they were actually visiting them. The introduction of an emotion engine provides an optimal sightseeing experience based on the user's emotions.

[0526] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0527] Step 1:

[0528] The user puts on the VR headset.

[0529] Input: The user puts on the VR headset.

[0530] Output: VR system startup.

[0531] What happens: The user puts on the VR headset and the system starts automatically.

[0532] Step 2:

[0533] The device activates the eye tracking and position tracking sensors.

[0534] Input: VR headset wearing information.

[0535] Output: Activation of the eye tracking and position tracking sensors.

[0536] Specific operation: The device activates the gaze tracking sensor and position tracking sensor built into the VR headset and prepares to acquire the user's gaze and position information.

[0537] Step 3:

[0538] The device acquires the user's gaze and location information in real time.

[0539] Input: User's eye movements and location.

[0540] Output: Real-time gaze and position data.

[0541] Specific operation: The device uses eye tracking and position tracking sensors to capture the user's eye movements and position information in real time and saves them as data.

[0542] Step 4:

[0543] The server receives the acquired gaze and location information and dynamically generates images of tourist spots.

[0544] Input: Gaze and location data sent from the device.

[0545] Output: Dynamically generated video of tourist attractions.

[0546] Specific operation: The server receives gaze and position information sent from the device and dynamically generates a video of the tourist spot using a game engine. For example, if the user looks left, the left view is generated.

[0547] Step 5:

[0548] The device displays the generated image on the VR headset.

[0549] Input: Generated video sent from the server.

[0550] Output: Video displayed in a VR headset.

[0551] Specific operation: The device receives the generated video sent from the server and displays it on the VR headset, allowing the user to experience images of tourist spots that respond to their line of sight.

[0552] Step 6:

[0553] The device activates emotion recognition software to recognize the user's emotions.

[0554] Input: User's facial expressions and voice.

[0555] Output: Recognized emotion data.

[0556] Specific operation: The device activates emotion recognition software and recognizes emotions from the user's facial expressions and voice. For example, if the user smiles, that information is captured.

[0557] Step 7:

[0558] The server optimizes the video and audio based on the recognized emotion.

[0559] Input: Emotion data sent from the device.

[0560] Output: Optimized video and audio.

[0561] How it works: The server receives emotion data sent from the device and optimizes the video and audio based on the recognized emotion. For example, if the user is happy, the system generates scenery and music that amplifies that joy.

[0562] Step 8:

[0563] The device displays optimized video and audio on the VR headset.

[0564] Input: Optimized video and audio sent from the server.

[0565] Output: Optimized video and audio displayed on a VR headset.

[0566] Specific operation: The device receives the optimized video and audio sent from the server and displays them on the VR headset, allowing the user to have an optimal sightseeing experience according to their emotions.

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] Conventional virtual reality (VR) systems and virtual shops using generative AI did not dynamically change or optimize the display based on the user's gaze, location, or emotions. This limited the user experience, making it difficult to provide an optimal shopping experience for each individual user. Furthermore, because the display content was not optimized based on the user's emotions, it was not possible to increase user satisfaction.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for using VR technology to enable a user to experience tourist spots from home, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks, a means for tracking the user's gaze and location information and dynamically changing the display content based on that, and a means for recognizing the user's emotions and optimizing the display content based on those emotions. This makes it possible to dynamically change the display content based on the user's gaze, location information, and emotions, providing an optimal shopping experience for each individual user.

[0571] "VR technology" stands for virtual reality technology, which allows users to experience computer-generated virtual environments.

[0572] "Generative AI" stands for generative artificial intelligence, an artificial intelligence technology that generates new information and content based on user input and data.

[0573] A "virtual shop" is a virtual store that exists on the Internet, where users can search for, purchase, and pay for products online.

[0574] An "Internet search tool" is software or an application used to search for information on the Internet.

[0575] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[0576] "Payment" is the process by which a user makes payment when purchasing a product.

[0577] A "virtual store clerk" is a virtual character that communicates with users within the virtual shop and provides product information and support.

[0578] "Gaze and location tracking" is a technology that detects the user's gaze movements and location in real time and collects that data.

[0579] "Dynamic change" refers to changing the displayed content in real time based on the user's gaze and location information.

[0580] "Emotion recognition" is a technology that detects a user's emotional state from their facial expressions and behavior.

[0581] "Optimizing display content" refers to adjusting the displayed information and content to the optimal form based on the user's emotions and behavior.

[0582] The system for implementing this invention has the following configuration. First, the user puts on a VR headset and accesses a virtual shop from home. The VR headset is equipped with cameras and sensors that track the user's line of sight and location information in real time. This allows the user to walk freely around the virtual shop and view the products in a 360-degree panoramic view.

[0583] The server uses generative AI to enable internet search tools, product purchases and payments, and communication with virtual store staff in the virtual shop. The generative AI also has the ability to recommend products based on the user's purchase and search history. It also tracks the user's gaze and location information and dynamically changes the content displayed based on that information. For example, when a user focuses their gaze on a particular product, detailed information and reviews of that product are displayed.

[0584] The server also uses an emotion recognition model to recognize the user's emotions in real time. The emotion recognition model is pre-trained using machine learning libraries such as TENSORFLOW (registered trademark). The display content is optimized based on the user's emotions. For example, if the user is happy, it will display special promotions and recommended products, and if the user is depressed, it will provide relaxing music and scenery.

[0585] A specific example is a system that displays detailed information about products that a user looks at in a virtual shop and optimizes the shopping experience based on the user's emotions using an emotion recognition model.This system displays detailed information about products that a user looks at in a virtual shop and optimizes the shopping experience based on the user's emotions using an emotion recognition model.

[0586] An example prompt is, "Develop an application that displays detailed information about products when a user looks at them and uses an emotion recognition model to optimize the shopping experience based on the user's emotions."

[0587] In this way, the content displayed can be dynamically changed based on the user's gaze, location, and emotions, providing an optimal shopping experience for each individual user.

[0588] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0589] Step 1:

[0590] The user wears a VR headset and accesses the virtual shop. The VR headset is equipped with cameras and sensors to track the user's gaze and position in real time. The input is the user's gaze data and position data, which are sent from the VR headset to a server. The output is an initial display of the virtual shop based on the user's gaze and position.

[0591] Step 2:

[0592] The server uses generative AI to enable internet search tools within the virtual shop, product purchases and payments, and communication with virtual store staff. Inputs include the user's search queries, purchase history, and search history. The generative AI processes this data and recommends the most suitable products and information to the user. The output is a product list and detailed information that is displayed in the user's field of view.

[0593] Step 3:

[0594] The server tracks the user's gaze and location information and dynamically changes the display content based on that information. The inputs include gaze data and location data, which are updated in real time. The server analyzes this data to identify products and areas of interest to the user. The output displays detailed information and reviews about the products the user is looking at.

[0595] Step 4:

[0596] The server uses an emotion recognition model to recognize the user's emotions in real time. Inputs include the user's facial expression data and voice data. The emotion recognition model is pre-trained using machine learning libraries such as TensorFlow (registered trademark) and analyzes this data to identify the user's emotional state. The output is optimization of the display content based on the user's emotions.

[0597] Step 5:

[0598] The server optimizes the display content based on the user's emotions. The input is emotional data obtained from an emotion recognition model. Based on this data, the server displays special promotions and recommended products if the user is happy, and provides relaxing music and scenery if the user is depressed. The output is the optimal display content based on the user's emotions.

[0599] Step 6:

[0600] The user selects a product in the virtual shop and completes the purchase. The inputs include the user's selection data and payment information. The server processes this data and completes the purchase. The output is displayed to the user as a purchase confirmation message and a receipt.

[0601] Step 7:

[0602] The server stores the user's purchase history and search history and uses it for the next shopping experience. The input is the user's purchase data and search data. The server stores this data in a database, and the generative AI uses this data for the next shopping experience. The output is that the user's history data is updated.

[0603] Example 2

[0604] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0605] Conventional online shopping systems have difficulty recreating the shopping experience of a physical store from the comfort of your own home. Furthermore, they do not recommend products based on the user's emotions, which prevents them from increasing user satisfaction. Furthermore, they have the problem of being unable to effectively utilize the user's purchase history and search history to reflect in the next recommendation.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0607] In this invention, the server includes: a means for allowing users to experience tourist spots from home using VR technology; a means for using generative AI to enable internet search tools, product purchases and payments in a virtual shop, and communication with virtual store clerks; a means for receiving a user's search query and recommending products using a generative AI model; a means for analyzing the user's emotions using an emotion engine and recommending additional products based on those emotions; and a means for saving the user's purchase history and using it for future recommendations. This allows users to recreate the shopping experience of a real store from the comfort of their own home, enabling product recommendations based on the user's emotions and increasing user satisfaction. Furthermore, the server can effectively utilize the user's purchase history and search history to reflect in future recommendations.

[0608] "VR technology" is a technology that uses virtual reality to allow users to experience an environment different from the real world.

[0609] "Generative AI" is an artificial intelligence technology that generates new information and content based on user input and behavioral data.

[0610] A "virtual shop" is a store virtually built on the Internet, and is an online platform where users can search for and purchase products.

[0611] An "Internet search tool" is software or functionality that allows users to search for specific information or products on the Internet.

[0612] "Purchase of product" refers to the act of a user purchasing a product selected in a virtual shop.

[0613] "Payment" is the process by which a user makes payment when purchasing a product.

[0614] A "virtual store clerk" is an artificial intelligence or software that interacts with users in a virtual shop, providing information about products and assisting with the purchasing process.

[0615] A "search query" is a search term or keyword that a user enters into a search tool.

[0616] A "generative AI model" is an algorithm or program that forms the core of generative AI and generates new information based on user input data.

[0617] An "emotion engine" is a technology that analyzes a user's emotional state and provides appropriate responses and recommendations based on the results.

[0618] "Purchase history" is a record of products a user has purchased in the past.

[0619] "Search history" is a record of searches a user has conducted in the past.

[0620] "Recommendation" is the act of presenting appropriate products or information based on a user's behavioral data and emotional state.

[0621] This invention is a system that allows users to recreate the shopping experience of a real store from the comfort of their own home. The system combines VR technology, generative AI, and an emotion engine.

[0622] The server hosts a generative AI model and collects and analyzes user behavior data. The generative AI model uses a common generative AI algorithm (e.g., a natural language processing model). The server stores data when users search for and purchase products in the virtual shop and uses it for future recommendations. It also uses an emotion engine (e.g., an emotion recognition API) to analyze the user's emotional state and recommend products based on that.

[0623] The device (user's PC or smartphone) provides the virtual shop interface and accepts operations that allow users to search for and purchase products. The device communicates with the server and displays recommendation results from the generative AI model and the results of communication with the virtual store clerk. Users access the virtual shop through their device to search for and purchase products.

[0624] Users access the virtual shop to search for and purchase products. For example, if a user types "I'm looking for the latest smartphone," the generative AI model will recommend smartphones that fit the user's preferences based on the user's past search and purchase history. Furthermore, if the user is happy, the emotion engine will analyze their emotions and recommend related products (e.g., smartphone accessories) to amplify their joy.

[0625] As a concrete example, consider the case where a user accesses a virtual shop and searches for the "latest smartphone." The server uses a generative AI model to recommend the "latest smartphone." Furthermore, the emotion engine analyzes the user's joy and recommends related products such as "smartphone cases" and "wireless earphones" to amplify that joy. The user selects the recommended "latest smartphone" and completes the purchase process. The server processes the payment and saves the purchase history.

[0626] An example of a prompt is as follows:

[0627] "A user is searching for the latest smartphone in a virtual shop. Based on their past search and purchase history, recommend smartphones that match their preferences. Also, if the user is happy, recommend related products that will amplify that emotion."

[0628] In this way, by clarifying the roles of the server, terminal, and user, and explaining the flow of data processing and calculation using specific hardware and software, it becomes easier to understand the overall picture of the system.

[0629] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0630] Step 1:

[0631] The user visits the virtual shop.

[0632] Users access the virtual shop's website or app using their PC or smartphone. If user login information is required, a login screen is displayed. The input is the user's access information, and the output is the virtual shop's home screen.

[0633] Step 2:

[0634] A user searches for a product.

[0635] A user enters a keyword into the search bar and clicks the search button, for example, "latest smartphones." The input is the user's search query, and the output is the submission of the search query.

[0636] Step 3:

[0637] The server receives the user's search query.

[0638] The server receives the user's search query and queries the database. The input is the user's search query and the output is the query to the database.

[0639] Step 4:

[0640] The server uses the generative AI model to recommend products.

[0641] The server uses a generative AI model to recommend products based on the user's search query. The generative AI model also takes into account the user's past search and purchase history. The input is search results from the database and the user's historical data, and the output is a list of recommended products.

[0642] Step 5:

[0643] The server analyzes the user's emotions using an emotion engine.

[0644] The server uses an emotion engine to analyze the user's emotional state, for example, to determine whether the user is happy or depressed. The input is the user's behavioral and facial expression data, and the output is the analyzed emotional state.

[0645] Step 6:

[0646] The server then recommends additional products based on the sentiment.

[0647] The server recommends additional products that match the user's emotions based on the analysis results of the emotion engine. For example, if the user is happy, it recommends related products that will amplify that happiness. The input is the analyzed emotional state, and the output is a list of additional recommended products.

[0648] Step 7:

[0649] The device displays the recommendations to the user.

[0650] The terminal displays the recommendation results received from the server to the user. The user can check the recommended products on the screen. The input is the recommendation results from the server, and the output is the recommendation results displayed on the user's screen.

[0651] Step 8:

[0652] The user selects a product and completes the purchase.

[0653] The user selects the recommended products and completes the checkout process, which includes entering shipping and payment information. The input is the user's selection and purchase information, and the output is the completion of the checkout process.

[0654] Step 9:

[0655] The server processes the payment.

[0656] The server receives the user's payment information and processes the payment. Once the payment process is complete, a confirmation email is sent to the user. The input is the user's payment information, and the output is a notification that the payment has been completed.

[0657] Step 10:

[0658] The server stores your purchase history and uses it for future recommendations.

[0659] The server stores the user's purchase history in a database. The stored data is used for the next recommendation. The input is the user's purchase history, and the output is the updated database.

[0660] (Application example 2)

[0661] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0662] Conventional virtual shop systems did not adequately recommend products based on the user's preferences and emotions, limiting the user experience. Communication with virtual store clerks was also limited, making it difficult to fully replicate the shopping experience in a real store. Furthermore, there was a lack of technology to analyze user emotions and reflect them in product recommendations, which prevented users from increasing satisfaction.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0664] In this invention, the server includes: a means for allowing a user to experience tourist spots from home using VR technology; a means for enabling internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop using generative AI; a means for recommending products based on the user's emotions using an emotion engine; a means for analyzing emotions from the user's facial expressions and voice; a means for generating product recommendations based on the user's queries and emotions using a generative AI model; and a means for real-time communication with the virtual store clerk. This enables product recommendations based on the user's preferences and emotions, recreating the shopping experience in a real store. Furthermore, analyzing the user's emotions and reflecting them in product recommendations can increase user satisfaction.

[0665] "VR technology" is a technology that uses virtual reality to allow users to experience a virtual space that is different from the real world.

[0666] "Generative AI" is an artificial intelligence technology that generates new information and content based on user input and actions.

[0667] A "virtual shop" is a store virtually constructed on the Internet, and is an online platform where users can search for, purchase, and pay for products.

[0668] An "Internet search tool" is software or a system that allows users to search for information or products on the Internet.

[0669] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[0670] "Payment" is the process by which a user makes payment when purchasing a product.

[0671] A "virtual store clerk" is a virtual entity that communicates with users within the virtual shop, answering questions and assisting them with the purchasing process.

[0672] An "emotion engine" is a technology that analyzes a user's emotions and responds or makes recommendations based on those emotions.

[0673] "Means for analyzing emotions from a user's facial expressions and voice" refers to technology that uses a camera or microphone to capture a user's facial expressions and voice, and then analyzes them to identify emotions.

[0674] A "generative AI model" is a specific implementation of generative AI, a model for generating new information and content based on user queries and emotions.

[0675] A "user query" is a search or question that a user enters into the system.

[0676] "Product recommendation" is the act of a system suggesting appropriate products to a user based on the user's preferences and behavior.

[0677] "Real-time communication" is a form of communication that allows users and systems to exchange information instantly.

[0678] A system for carrying out this invention has the following configuration: A server includes means for allowing a user to experience tourist spots from home using VR technology, means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with a virtual store clerk, means for recommending products according to the user's emotions using an emotion engine, means for analyzing emotions from the user's facial expressions and voice, means for generating product recommendations based on the user's queries and emotions using a generative AI model, and means for real-time communication with a virtual store clerk.

[0679] Hardware and Software Configuration

[0680] Hardware: Smartphone, Head-Mounted Display (HMD), Camera, Microphone

[0681] Software: OpenAI API, emotion recognition library, OpenCV

[0682] Data processing and calculation

[0683] 1. Data capture: The user's facial expressions and voice are captured using a camera and microphone, allowing for real-time emotional data to be obtained.

[0684] 2. Emotion Analysis: Using an emotion recognition library, we analyze captured facial and voice data to identify the user's emotions.

[0685] 3. Use of generative AI models: Based on user queries and sentiment data, prompts are sent to generative AI models to generate product recommendations.

[0686] 4. Communication with a virtual store clerk: A generative AI model is used to generate answers from a virtual store clerk in response to user questions, and these answers are provided to the user in real time.

[0687] Specific examples

[0688] Product recommendations based on user queries and sentiment

[0689] User query: "Looking for a new smartphone"

[0690] User emotion: "Delight"

[0691] Prompt for the generative AI model:

[0692] User Query: Looking for a new smartphone

[0693] User Emotion: Delight

[0694] Please tell me what products you recommend.

[0695] Communication with virtual store associates

[0696] User Question: "What is the battery life of this phone?"

[0697] Prompt for the generative AI model:

[0698] User Question: What is the battery life of this phone?

[0699] Please tell us the answer of the virtual salesperson.

[0700] In this way, the server utilizes generative AI models based on the user's queries and emotions to provide appropriate product recommendations and enable communication with virtual store associates, allowing users to recreate the shopping experience of a real store from the comfort of their own home.

[0701] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0702] Step 1:

[0703] Users access the virtual shop using a smartphone or head-mounted display (HMD). They input their facial expressions and voice through a camera and microphone, which captures the user's real-time emotional data.

[0704] Step 2:

[0705] The device sends the captured facial and voice data to an emotion recognition library, which analyzes the data and identifies the user's emotions. The input is facial and voice data, and the output is the user's emotion (e.g., joy, sadness).

[0706] Step 3:

[0707] A user inputs a query to search for a product in the virtual shop. For example, the text input is "I'm looking for a new smartphone." The input is the user's query.

[0708] Step 4:

[0709] The server sends the user's query and emotion data to the generative AI model, which generates prompts and product recommendations based on these inputs. The input is the user's query and emotion data, and the output is a list of recommended products.

[0710] Step 5:

[0711] The server displays the product recommendations obtained from the generative AI model to the user. The user can check the list of recommended products and view detailed information. The input is the list of recommended products, and the output is the user's browsing behavior.

[0712] Step 6:

[0713] The user inputs a question to the virtual store clerk. For example, text input is made such as "What is the battery life of this smartphone?" The input is the user's question.

[0714] Step 7:

[0715] The server sends the user's question to the generative AI model, which generates prompts based on the question and generates answers from the virtual store clerk. The input is the user's question, and the output is the answer from the virtual store clerk.

[0716] Step 8:

[0717] The server displays the answer of the virtual clerk obtained from the generative AI model to the user. The user can confirm the answer and continue asking questions. The input is the answer of the virtual clerk, and the output is the user's next action.

[0718] Step 9:

[0719] If the user decides to make a purchase, the server initiates the checkout process: the user enters payment information, and the server processes the payment through the payment system. The input is the user's payment information, and the output is confirmation of the successful purchase.

[0720] Step 10:

[0721] After a purchase is completed, the server stores the user's purchase history and search history in a database. This improves the accuracy of recommendations from next time onwards. The input is the user's purchase history and search history, and the output is an updated database.

[0722] Example 3

[0723] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0724] Conventional tourism experience systems using VR technology allow users to visually experience tourist destinations, but they are unable to dynamically change the images according to the user's emotions or gaze, making it difficult to provide a realistic experience. Furthermore, there is also the issue of not being able to optimally recommend products that take the user's emotions into account when recommending products in virtual shops or during the purchasing process. This makes it difficult to increase user satisfaction.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0726] In this invention, the server includes: means for allowing a user to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; means for acquiring the user's gaze information and dynamically changing the scenery of the tourist spot based on that; means for analyzing the user's facial expressions and voice to determine their emotions and dynamically adjusting the images according to those emotions; and means for the generative AI to recommend products based on the user's emotions. This allows the user to experience images that change in real time according to their gaze and emotions, and to receive optimal product recommendations based on their emotions.

[0727] "VR technology" is a technology for realizing virtual reality, allowing users to visually and experientially interact in a virtual space using dedicated headsets and devices.

[0728] "Generative AI" is a type of artificial intelligence, a system that has the ability to generate new information and recommendations based on user input data and history.

[0729] A "virtual shop" is a virtual store that exists on the Internet and is an online platform where users can search, browse and purchase products.

[0730] An "Internet search tool" is software or a system that allows users to search for specific information or products on the Internet.

[0731] "Product purchase and payment" refers to the process of purchasing the product selected by the user and making the payment.

[0732] A "virtual store clerk" is a virtual character or system that communicates with users within a virtual shop and explains and recommends products.

[0733] "Gaze information" is data that indicates the direction the user is looking, and is information obtained through VR headsets and other devices.

[0734] "Determining emotions by analyzing facial expressions and voice" is the process of identifying a user's emotional state at that time by analyzing their facial expressions and tone of voice.

[0735] "Dynamic image adjustment" refers to changing the color, brightness, contrast, etc. of the image in real time based on the user's emotions and gaze information.

[0736] "Product recommendation" is the process of suggesting the most suitable product based on the user's past behavior and current emotional state.

[0737] The present invention provides a system that allows users to experience tourist spots from their own homes and purchase products in virtual shops. Specific embodiments of this system are described below.

[0738] Hardware and software used

[0739] Hardware: VR headset (e.g., a general VR device), PC, or smartphone

[0740] Software: 360-degree panoramic video playback software, generative AI model (e.g., general generative AI), emotion engine

[0741] System Overview

[0742] Users wear a VR headset and experience tourist attractions through 360-degree panoramic images. The system dynamically adjusts video and product recommendations based on real-time capture of the user's gaze, facial expressions, and voice. In the virtual shop, a generative AI model recommends optimal products based on the user's emotions and history, and provides detailed information through interaction with a virtual sales associate.

[0743] Data processing and calculation

[0744] 1. Video playback:

[0745] The server streams 360-degree panoramic video data of tourist spots to the user's device.

[0746] The device acquires the user's gaze information through the VR headset's sensors and sends it to the server.

[0747] The server dynamically plays back the video based on the line-of-sight information.

[0748] 2. Emotion analysis:

[0749] The device uses the camera and microphone built into the VR headset to capture the user's facial expressions and voice.

[0750] The emotion engine analyzes this data to determine the user's emotional state.

[0751] The server dynamically adjusts the color and brightness of the video based on the emotion analysis results.

[0752] 3. Product recommendation:

[0753] When a user visits the virtual shop, a generative AI model recommends the most suitable products based on the user's emotions and history.

[0754] Users can communicate with virtual store clerks via voice and text to obtain detailed product information.

[0755] When the user decides to make a purchase, the terminal sends data to the server to complete the purchase procedure, and the purchase is completed.

[0756] Specific examples

[0757] The user wears a standard VR device at home and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user is moved by the view from the stage of Kiyomizu-dera Temple and smiles, the device sends the facial expression to the emotion engine. The emotion engine analyzes the user's joy, and the server adjusts the image to make it more vivid. The user then searches for yatsuhashi in a virtual shop and purchases the product recommended by the generation AI. Through a conversation with a virtual store clerk, the user obtains detailed information about the yatsuhashi and completes the purchase process.

[0758] Prompt Sentence Examples

[0759] "When a user is moved by the view from the stage of Kiyomizu-dera Temple and smiles, the emotion engine should detect that joy, and the server should display the image more vividly. Also, when a user searches for yatsuhashi in a virtual shop, the generative AI should recommend the most suitable product and provide detailed information through a conversation with a virtual store clerk."

[0760] In this way, the system allows the user to experience the tourist spot in real life and purchase local specialties from the comfort of their own home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0761] Step 1:

[0762] The user puts on the VR headset.

[0763] Input: The user puts on the VR headset.

[0764] Output: The VR headset is started and the system is ready to capture the user's visual information.

[0765] Specific operation: The user puts on the VR headset and turns it on. The headset starts up and completes the initial setup.

[0766] Step 2:

[0767] The server streams the 360-degree panoramic video.

[0768] Input: Video request from the user's device.

[0769] Output: 360-degree panoramic video streamed to the user's device.

[0770] Specific operation: The server sends 360-degree panoramic video data of the tourist attraction to the user's device, and the device plays the video.

[0771] Step 3:

[0772] The device sends the user's gaze information to the server.

[0773] Input: User gaze information captured by sensors in the VR headset.

[0774] Output: Gaze information sent to the server.

[0775] Specific operation: The device acquires the user's gaze information in real time through the VR headset's sensors and sends that information to the server.

[0776] Step 4:

[0777] The server plays the video according to the line of sight.

[0778] Input: Gaze information sent from the device.

[0779] Output: Images are played according to the user's gaze.

[0780] Specific operation: Based on the gaze information received by the server, the server selects the video in the direction the user is looking and sends it to the device, which then plays the video.

[0781] Step 5:

[0782] The device sends the user's facial expressions and voice to the emotion engine.

[0783] Input: User facial and voice data captured by cameras and microphones in the VR headset.

[0784] Output: Facial and voice data sent to the emotion engine.

[0785] Specific operation: The device uses the camera and microphone of the VR headset to capture the user's facial expressions and voice, and sends that data to the emotion engine.

[0786] Step 6:

[0787] The emotion engine analyzes the user's emotions.

[0788] Input: Facial expression and voice data sent from the device.

[0789] Output: Analysis results showing the user's emotional state.

[0790] Specific behavior: The emotion engine analyzes the facial and voice data it receives to determine the user's emotional state (e.g., joy, surprise, sadness).

[0791] Step 7:

[0792] The server dynamically adjusts the video based on the emotion analysis results.

[0793] Input: Sentiment analysis results from the sentiment engine.

[0794] Output: The adjusted video is sent to the user's device.

[0795] Specific operation: The server receives the emotion analysis results, dynamically adjusts the color and brightness of the image according to the user's emotions, and sends the image to the device.

[0796] Step 8:

[0797] The user visits the virtual shop.

[0798] Input: A user request to access the virtual shop.

[0799] Output: The virtual shop interface is displayed on the user's device.

[0800] Specific operation: The user accesses a virtual shop in the VR space, and the shop interface is displayed on the device.

[0801] Step 9:

[0802] A generative AI model recommends products based on user emotions.

[0803] Input: Sentiment analysis results and user history data.

[0804] Output: A list of products recommended to the user.

[0805] Specific operation: The generative AI model selects the most suitable products and generates a recommendation list based on the results of sentiment analysis and the user's purchase and search history.

[0806] Step 10:

[0807] Users communicate with virtual store clerks.

[0808] Input: User voice or text input.

[0809] Output: Response from the virtual store clerk and detailed product information.

[0810] Specific actions: The user sends a question or request to the virtual store clerk, who responds via voice or text and provides detailed product information.

[0811] Step 11:

[0812] The user purchases the product.

[0813] Input: User's purchasing decision and payment information.

[0814] Output: Confirmation of successful purchase and information about the purchased item.

[0815] What happens: The user decides to make a purchase and enters payment information. The device processes the purchase, sends the data to the server, and the purchase is completed.

[0816] (Application example 3)

[0817] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0818] Conventional tourism experience systems using virtual reality (VR) technology do not provide dynamic image changes or product recommendations based on the user's emotions or gaze, limiting the user experience. Furthermore, virtual shops do not provide product recommendations that take the user's emotions into account, making it difficult for users to find the perfect product. This reduces user satisfaction and prevents the virtual experience from being fully appealing.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0820] In this invention, the server includes: a means for allowing users to experience tourist spots from home using VR technology; a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; a means for analyzing user emotions in real time and dynamically changing the display of images and products according to those emotions; and a means for the generative AI to recommend products based on the user's emotion data and gaze data. This enables dynamic image changes and product recommendations according to the user's emotion and gaze.

[0821] "VR technology" is a technology for realizing virtual reality, allowing users to have a reality-like experience in a virtual space.

[0822] "Generative AI" is a type of artificial intelligence, a technology that has the ability to generate new information and content based on user data.

[0823] A "virtual shop" is a virtual store that exists on the Internet and is an online platform where users can browse and purchase products.

[0824] An "Internet search tool" is software or an application used to search for information on the Internet.

[0825] "Product purchase and payment" refers to the process by which a user selects a product and pays for it.

[0826] A "virtual store clerk" is a virtual character that communicates with users within the virtual shop, providing product information and assisting with the purchasing process.

[0827] "Analyzing user emotions in real time" is the process of detecting a user's emotional state in real time and analyzing that information.

[0828] "Dynamic change in the display of images and products" means changing the appearance of displayed images and products in real time according to the user's emotions and line of sight.

[0829] "User emotional data" refers to data that indicates the user's emotional state, and is information obtained from brain waves, facial expressions, and the like.

[0830] "Gaze data" is data that indicates the movement of the user's gaze, and is information that detects the direction in which the user is looking.

[0831] "Recommending products" means selecting and presenting the most suitable products to a user based on the user's data.

[0832] The system for implementing the present invention uses the following hardware and software.

[0833] Hardware used

[0834] VR headset: A device that allows users to experience virtual reality spaces.

[0835] Emotion sensor: A device that acquires the user's brainwave data and analyzes their emotions.

[0836] Software used

[0837] Game engine: A platform for developing VR content.

[0838] TensorFlow: A machine learning framework for implementing emotion recognition models.

[0839] OpenAI GPT-4: A tool that implements a generative AI model to recommend the best products to users.

[0840] Data processing and calculation

[0841] 1. Viewing VR content:

[0842] The server uses a game engine to generate a 360-degree panoramic image and transmits it to the VR headset.

[0843] The user's gaze data is acquired from the VR headset, and the image changes dynamically according to the gaze.

[0844] 2. Emotion recognition:

[0845] The user's brainwave data is acquired from the emotion sensor and sent to the server.

[0846] The server uses TensorFlow to implement an emotion recognition model and analyzes user emotions in real time.

[0847] If it senses the user's joy, it will adjust the images and product display accordingly.

[0848] 3. Product Recommendations by Generative AI:

[0849] The server inputs the user's emotional and gaze data as prompts into OpenAI GPT-4.

[0850] A generative AI model recommends the best products for users and displays them in a virtual shop.

[0851] Specific examples

[0852] 1. User puts on VR headset:

[0853] The user puts on a VR headset and enters a virtual store.

[0854] A 360-degree panoramic image is displayed, and the image changes dynamically as the user moves their gaze.

[0855] 2. Emotion recognition and dynamic changes in images:

[0856] The emotion sensor acquires the user's brainwave data and sends it to the server.

[0857] The server analyzes the user's delight using a TensorFlow model.

[0858] If a user is excited about a particular product, it will be displayed more vividly.

[0859] 3. Product Recommendations by Generative AI:

[0860] The server inputs the user's emotion data and gaze data as prompts into OpenAI GPT-4.

[0861] A generative AI model recommends the best products for users and displays them in a virtual shop.

[0862] Prompt Sentence Examples

[0863] User emotion data: Happiness 0.8, Surprise 0.2

[0864] User gaze data: Product A 5 seconds, Product B 3 seconds

[0865] Recommend the best products to your users.

[0866] In this way, users can enjoy a virtual shopping experience from the comfort of their own home.

[0867] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0868] Step 1:

[0869] The user puts on a VR headset and enters a virtual store.

[0870] Input: User actions

[0871] Output: 360-degree panoramic image of a virtual store

[0872] Specific operation: The server sends a 360-degree panoramic image generated using a game engine to the VR headset, allowing the user to enter the virtual space.

[0873] Step 2:

[0874] The user's gaze data is acquired from the VR headset, and the image changes dynamically according to the gaze.

[0875] Input: User gaze data

[0876] Output: Dynamically changing video

[0877] Specific operation: The server acquires gaze data from the VR headset in real time and uses the game engine to change the image according to the gaze.

[0878] Step 3:

[0879] The user's brain wave data is acquired from the emotion sensor and sent to the server.

[0880] Input: User's brainwave data

[0881] Output: EEG data sent to the server

[0882] Specific operation: The emotion sensor acquires the user's brainwave data in real time and transmits it to the server.

[0883] Step 4:

[0884] The server implements an emotion recognition model using TensorFlow to analyze the user's emotions in real time.

[0885] Input: EEG data

[0886] Output: User emotion data

[0887] How it works: The server uses a TensorFlow model to analyze EEG data and detect the user's emotional state in real time.

[0888] Step 5:

[0889] If the system senses the user's joy, it adjusts the images and product display according to that emotion.

[0890] Input: User emotion data

[0891] Output: Adjusted video and product display

[0892] Specific operation: The server uses the game engine based on the emotion data to make the images and products that the user finds pleasing more vivid.

[0893] Step 6:

[0894] The server inputs the user's emotional and gaze data as prompts into OpenAI GPT-4.

[0895] Input: User emotion data, gaze data

[0896] Output: prompts to the generative AI model

[0897] Specific operation: The server converts emotion data and gaze data into text prompts and inputs them into OpenAI GPT-4.

[0898] Step 7:

[0899] A generative AI model recommends the best products to users and displays them in a virtual shop.

[0900] Input: prompt to generative AI model

[0901] Output: Recommended products

[0902] Specific operation: OpenAI GPT-4 generates the most suitable product for the user based on the prompts, and the server displays the product information in a virtual shop.

[0903] Prompt Sentence Examples

[0904] User emotion data: Happiness 0.8, Surprise 0.2

[0905] User gaze data: Product A 5 seconds, Product B 3 seconds

[0906] Recommend the best products to your users.

[0907] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0908] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0909] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0910] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0911] [Second embodiment]

[0912] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0913] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0914] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0915] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0916] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0917] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0918] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0919] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0920] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0921] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0922] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0923] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0924] "Example 1"

[0925] An example of the present invention is a system that allows users to experience tourist spots from their own homes. Specifically, using a VR headset, users can experience tourist spots through 360-degree panoramic images. The system also tracks the user's eye movements and location information, and the images of the tourist spots change dynamically accordingly. This allows users to experience the tourist spots as if they were walking around them from the comfort of their own homes.

[0926] "Example 2"

[0927] This invention also includes a system that uses generative AI to provide internet search tools, product purchases, payments, and communication with virtual store clerks in a virtual shop. Specifically, when a user searches for or purchases products in a virtual shop, the generative AI learns the user's behavior and recommends products that match the user's preferences from the next time. Communication with the virtual store clerk is also possible, allowing the user to ask questions about products and complete the purchase process. This allows the user to recreate the shopping experience of a real store from the comfort of their own home.

[0928] "Example 3"

[0929] As a concrete example, consider a case where a user experiences tourist attractions in Kyoto from the comfort of their own home. The user puts on a VR headset and experiences the sights of Kyoto through a 360-degree panoramic image. As the user moves their gaze, the image changes dynamically, giving the user the experience of walking through the area. The user can also search for Kyoto's famous products in a virtual shop and purchase products recommended by generative AI. By communicating with a virtual store clerk, the user can obtain detailed product information and complete the purchase process.

[0930] The processing flow of each embodiment will be described below.

[0931] "Example 1"

[0932] Step 1: The user puts on the VR headset.

[0933] Step 2: The system provides a 360-degree panoramic image for users to experience the tourist attractions.

[0934] Step 3: The system tracks the user's eye movements and location.

[0935] Step 4: The image of the tourist spot changes dynamically accordingly.

[0936] "Example 2"

[0937] Step 1: User searches for a product in the virtual shop.

[0938] Step 2: The generative AI learns the user's behavior and recommends products that match the user's preferences next time.

[0939] Step 3: The user communicates with the virtual store associate.

[0940] Step 4: The user gets more information about the product and completes the purchase.

[0941] "Example 3"

[0942] Step 1: If a user wants to experience Kyoto's tourist attractions from home, they first put on a VR headset.

[0943] Step 2: Experience Kyoto's tourist attractions through 360-degree panoramic images.

[0944] Step 3: As the user moves their gaze, the image changes dynamically.

[0945] Step 4: The user searches for Kyoto specialties in the virtual shop.

[0946] Step 5: Purchase the product recommended by the generative AI.

[0947] Step 6: Communicate with the virtual store associate to obtain more information about the product and complete the purchase.

[0948] Example 1

[0949] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0950] Traditional tourism experiences required users to actually go to the actual location, which was problematic in terms of time and expense. Even in virtual shops, users could not actually touch and feel the products, which often led to anxiety when making a purchase. Furthermore, there was a lack of systems that dynamically changed the images in response to the user's gaze and movements, preventing a realistic experience.

[0951] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0952] In this invention, the server includes: means for allowing users to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop; means for tracking the user's eye movements and location information and dynamically changing the scenery of the tourist spot based on the movements; means for acquiring 360-degree panoramic video data of the tourist spot and streaming it to the user's device; and means for displaying the video using a VR headset and rendering the video in real time based on the user's eye gaze data. This allows users to have a realistic tourist experience from the comfort of their own home and purchase products safely in the virtual shop.

[0953] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[0954] "Generative AI" stands for generative artificial intelligence, an artificial intelligence technology that generates new information and content based on user input and data.

[0955] A "virtual shop" is a virtual store operated on the Internet, which is a system that allows users to search for, purchase, and pay for products online.

[0956] An "Internet search tool" is software or an application used to search for information on the Internet, allowing users to efficiently find specific information.

[0957] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, answering their questions and recommending products.

[0958] "Eye movement" refers to the movement of the user's eyes, and is data that indicates the direction in which the user is looking.

[0959] "Location information" refers to data that indicates a user's physical location and is used to identify where the user is located.

[0960] "360-degree panoramic video data" refers to video data that covers all directions, allowing users to look around in any direction.

[0961] "Streaming" is a technology that plays data while transferring it in real time, allowing users to use the data instantly without having to download it.

[0962] A "VR headset" is a device that allows users to experience a virtual reality environment, and when worn, it immerses them in a virtual space.

[0963] "Real-time rendering" refers to the technology by which a computer instantly generates and displays images and graphics, allowing images to be updated without delay in response to the user's movements.

[0964] This invention is a system that allows users to experience tourist spots from the comfort of their own homes, and is realized by combining VR technology and generative AI. Specific embodiments of this system are described below.

[0965] First, the server acquires 360-degree panoramic video data of tourist attractions. This data can be obtained from the Internet or a dedicated database. For example, you can download high-resolution panoramic images of tourist attractions using the Google Street View API.

[0966] The server then streams the captured 360-degree panoramic video data to the user's device using protocols such as HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH).

[0967] The user wears a VR headset at home, and the device is connected to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[0968] The device uses sensors built into the VR headset to track the user's gaze and location. For example, it uses the headset's gyro sensor and accelerometer to detect the user's head movement. The acquired gaze data is analyzed and the image is dynamically updated to match the user's gaze. This allows the user to view different parts of the tourist attraction depending on their gaze.

[0969] Generative AI will enable internet search tools, product purchases and payments, and communication with virtual store staff in virtual shops. It will also have the ability to recommend products based on users' purchase and search history. This allows users to purchase products with confidence within the virtual shop.

[0970] As a concrete example, if a user wants to experience Kinkaku-ji Temple in Kyoto, the following operations take place: The server obtains 360-degree panoramic video data of Kinkaku-ji Temple using the Google Street View API and streams it to the user's device using the HTTP Live Streaming (HLS) protocol. The user puts on a VR headset, and the device displays the image of Kinkaku-ji Temple using a game engine. The device uses the VR headset's gyro sensor to track the user's eye movement; if the user moves their head to the right, the device detects this movement and shifts the image to the right. This allows the user to see different parts of Kinkaku-ji Temple depending on their eye movement, giving the user the experience of walking around the temple in person.

[0971] Example prompts to input to a generative AI model:

[0972] "Please explain how a user can experience Kinkaku-ji Temple from the comfort of their own home using a VR headset. Please include how the image changes dynamically in response to the user's gaze."

[0973] In this way, users can have a realistic tourist experience from the comfort of their own home, and can purchase products safely in virtual shops.

[0974] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0975] Step 1:

[0976] The server acquires 360-degree panoramic video data of tourist spots.

[0977] Specifically, the server downloads high-resolution panoramic images of tourist spots from the Internet or a dedicated database, for example, using the Google Street View API.

[0978] Input: Name and location of tourist spot

[0979] Output: 360-degree panoramic video data

[0980] Step 2:

[0981] The server streams the acquired video data to the user's device.

[0982] Specifically, the server uses protocols such as HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) to send video data to the user's device in real time.

[0983] Input: 360-degree panoramic video data

[0984] Output: Video data streamed to the user's device

[0985] Step 3:

[0986] The user puts on the VR headset and the device displays the image.

[0987] Specifically, the user wears a VR headset at home, and the device connects to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[0988] Input: Streamed video data

[0989] Output: Image displayed on a VR headset

[0990] Step 4:

[0991] The device tracks the user's eye movements and location.

[0992] Specifically, the device uses the gyro sensor and acceleration sensor built into the VR headset to detect the user's head movements.

[0993] Input: User's eye movements and location information

[0994] Output: Gaze data and position data

[0995] Step 5:

[0996] The device dynamically updates the video based on gaze data.

[0997] Specifically, the device analyzes the acquired gaze data and dynamically updates the image to match the user's gaze. For example, if the user moves their head to the left, the device detects this movement and shifts the image to the left.

[0998] Input: Gaze data and position data

[0999] Output: Dynamically updated video

[1000] Step 6:

[1001] Users experience tourist destinations.

[1002] Specifically, users experience tourist spots through a VR headset. The images change dynamically according to the user's line of sight, giving the user the experience of walking through the tourist spots in real life.

[1003] Input: Dynamically updated video

[1004] Output: User's sightseeing experience

[1005] In this way, users can have a realistic tourist experience from the comfort of their own home.

[1006] (Application example 1)

[1007] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1008] Conventional tourism experience systems and virtual shop systems using virtual reality (VR) technology have faced issues such as a lack of realism and convenience when users experience tourist spots or purchase products from the comfort of their own homes. Furthermore, the lack of interactive experiences utilizing the user's gaze or location information limits the user experience. Furthermore, the display of product information and the purchasing process are cumbersome, requiring improvements to increase user satisfaction.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1010] In this invention, the server includes a means for allowing users to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks, and a means for tracking the user's gaze and location information and displaying detailed information when the user approaches a product. This allows users to enjoy a realistic tourist experience and virtual shopping from the comfort of their own home. Furthermore, an interactive experience utilizing gaze and location information is provided, and the display of product information and the purchase process are simplified, improving the user experience.

[1011] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[1012] "Generative AI" refers to generative artificial intelligence, which has the ability to generate new information and content based on user input and data.

[1013] A "virtual shop" is a virtual store that exists on the Internet, a virtual space where users can browse and purchase products online.

[1014] An "Internet search tool" is software or an application used to search for information on the Internet.

[1015] "Product purchase and payment" refers to the process by which a user selects a product, completes the purchase process, and then pays.

[1016] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, providing product information and support.

[1017] "Tracking gaze and location information" is a technology that detects the direction of the user's gaze and the position of their body in real time, and the system operates based on that information.

[1018] "Show detailed information" means that when a user approaches a specific product, detailed product information such as a description, specifications, and reviews will be displayed on the screen.

[1019] To implement the present invention, the following system configuration and program are required.

[1020] First, the user puts on a VR headset and experiences tourist attractions and virtual shops from the comfort of their own home. The server then uses VR technology to provide the means for the user to experience the tourist attractions from home. This means includes the ability to generate 360-degree panoramic images and dynamically change the images by tracking the user's line of sight and location. Specifically, the images are rendered in real time using a game engine.

[1021] Next, the server uses generative AI to provide internet search tools for the virtual shop, as well as the means to purchase and pay for products and communicate with virtual store staff. Generative AI has the ability to recommend products based on the user's purchase and search history, allowing users to easily find products that suit them. Natural language processing models such as GPT-3 are used for generative AI.

[1022] Furthermore, the server tracks the user's gaze and location information, providing a means to display detailed information about a product when the user approaches it. This means includes gaze tracking technology and location tracking technology. For gaze tracking, hardware such as an eye tracker is used, and for location tracking, sensors built into the VR headset are used. As a result, when the user directs their gaze at a specific product, detailed information about that product is displayed.

[1023] As a concrete example, consider a scene in which a user finds a "smartphone" in a virtual shop and checks its detailed information. The user puts on a VR headset and walks around the virtual shop. When the user directs their gaze towards the smartphone, detailed information about the smartphone (e.g., "The latest model smartphone. Equipped with a high-resolution camera and a long-lasting battery.") is displayed. An example of a prompt sentence to realize this scene is as follows:

[1024] Example prompt sentence:

[1025] The user puts on a VR headset and walks around the virtual shop. When the user looks at the smartphone, detailed information about the smartphone (e.g., "Latest model smartphone. Equipped with a high-resolution camera and long-lasting battery life.") is displayed. Please generate a game engine C script to realize this scene.

[1026] In this way, users can enjoy realistic sightseeing experiences and virtual shopping from the comfort of their own homes, and the interactive experience using gaze and location information is provided, simplifying product information display and purchasing procedures, improving the user experience.

[1027] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1028] Step 1:

[1029] The user puts on the VR headset and launches the virtual shop application.

[1030] Input: User actions

[1031] Output: VR headset boots, virtual shop application starts

[1032] Specific operation: The user puts on a VR headset and presses a button to launch the application, which starts the VR headset and launches the virtual shop application.

[1033] Step 2:

[1034] The server tracks the user's gaze and location in real time.

[1035] Input: Gaze and position data from the VR headset

[1036] Output: User's gaze direction and position information

[1037] How it works: The sensors built into the VR headset detect the user's gaze direction and location and send that data to the server, which processes the data in real time to determine the user's gaze direction and location.

[1038] Step 3:

[1039] The server identifies the product in the user's line of sight and obtains its detailed information.

[1040] Input: User's gaze direction and position information

[1041] Output: Detailed information about the product in the line of sight

[1042] Specific operation: The server identifies the product that the user is looking at based on the user's gaze direction and location information, and retrieves detailed information about the identified product (e.g., product name, description, price, reviews, etc.) from the database.

[1043] Step 4:

[1044] The server sends the obtained detailed information to the VR headset and displays it to the user.

[1045] Input: Product details

[1046] Output: Product details displayed in a VR headset

[1047] Specific operation: The server sends the acquired product details to the VR headset, which displays the received information in the user's field of view, allowing the user to check the details.

[1048] Step 5:

[1049] The user selects a product and completes the purchase procedure.

[1050] Input: User selection

[1051] Output: Purchase completed

[1052] Specific operation: The user selects a product using their eyes or a controller and presses the purchase button. The server receives the user's selection and begins the purchase process. The server then enters payment information and completes the purchase process.

[1053] Step 6:

[1054] The server updates the purchase history, and the generative AI stores the data for the next product recommendation.

[1055] Input: Purchase history data

[1056] Output: Updated purchase history, saved data

[1057] How it works: The server updates the user's purchase history to a database, and the generative AI saves the data necessary for the next product recommendation, allowing it to recommend products that are appropriate for the user's next shopping experience.

[1058] Example 2

[1059] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1060] Conventional virtual shop systems have struggled to recreate the shopping experience of a real store from the comfort of your own home. They also lacked the functionality to learn user behavior patterns and recommend the best products for each individual user. Furthermore, when communicating with virtual store clerks, it was difficult to provide appropriate answers to user questions. To solve these issues, it is necessary to effectively collect and analyze user behavior data and utilize generative AI models.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1062] In this invention, the server includes: means for allowing a user to experience tourist spots from home; means for using a generative AI to enable internet search tools, product purchases and payments in a virtual shop, and communication with a virtual store clerk; means for collecting user behavior data and learning the user's behavior patterns using a generative AI model; means for recommending products based on the user's behavior patterns; means for using the generative AI model to preferentially display highly relevant products when the user searches for a product; and means for using the generative AI model to generate appropriate answers when the user asks a question to the virtual store clerk. This allows users to recreate the shopping experience in an actual store from the comfort of their own home, making it possible to recommend products that are best suited to each user, and providing appropriate answers when communicating with the virtual store clerk.

[1063] "User" means an individual or organization that uses the system to search for products in the virtual shop, purchase products, make payments, and communicate with virtual store staff.

[1064] The "server" is a computer system that collects, stores, and analyzes user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[1065] "Generative AI" is an artificial intelligence technology that learns from user behavior data and recommends products that match the user's preferences, or acts as a virtual store clerk to generate appropriate answers to the user's questions.

[1066] A "virtual shop" is a virtual store on the Internet where users can search for, browse, purchase, and pay for products.

[1067] "Behavioral data" refers to information about the operations performed by users in the virtual shop, such as search history, purchase history, and browsing history.

[1068] "Behavioral patterns" are data that indicate a user's preferences and tendencies, learned by generative AI based on user behavior data.

[1069] "Product recommendation" is a function in which generative AI presents appropriate products to users based on their behavioral patterns.

[1070] A "virtual store clerk" is a virtual store clerk that uses generative AI to generate appropriate answers to users' questions and communicates with them.

[1071] "Internet Search Tool" is a function that allows users to search for products within the virtual shop.

[1072] "Checkout" is the process by which a user makes payment for the products selected in the virtual shop.

[1073] A "prompt sentence" is text data input to a generative AI model, and is an instruction sentence that enables the AI ​​to generate an appropriate response.

[1074] MODE FOR CARRYING OUT THE INVENTION

[1075] This invention is a virtual shop system using generative AI, which aims to recreate the shopping experience of a real store from the comfort of your own home. The system collects user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[1076] Hardware and software used

[1077] Server: A computer system that collects, stores, and analyzes user behavior data and uses generative AI models to learn user behavior patterns. Specifically, it uses a high-performance database server and a GPU server to run the AI ​​model.

[1078] Terminal: A device that allows users to access the virtual shop, search for products, view them, purchase them, make payments, and communicate with virtual store staff. Specifically, this includes PCs, smartphones, tablets, etc.

[1079] Generative AI model: An artificial intelligence technology that learns from user behavior data to recommend products that match the user's preferences and generate appropriate answers to user questions as a virtual store clerk. Specifically, it uses natural language processing models such as GPT-4.

[1080] Specific operation of the system

[1081] 1. Collecting User Behavior Data:

[1082] Users can access the virtual shop to search and browse products.

[1083] The device records the user's operations in real time and transmits them to the server.

[1084] The server stores the received data in a database, specifically recording information such as the user ID, search keywords, viewed product ID, and viewing time.

[1085] 2. Learning user behavior patterns:

[1086] The server periodically extracts user behavior data from the database.

[1087] The server uses a generative AI model (e.g., GPT-4) to analyze user behavior patterns, specifically learning what products users prefer and what time of day they access the site.

[1088] The server stores the learning results as a user profile.

[1089] 3. Perform a product search:

[1090] Users enter the product name or category into the virtual shop's search bar.

[1091] The terminal transmits the input search keyword to the server.

[1092] The server searches the database and generates a list of relevant products, specifically retrieving product IDs, product names, prices, stock status, and other information that matches the search keywords.

[1093] 4. Product recommendation:

[1094] The server prioritizes the display of relevant products based on the user's behavioral patterns. Specifically, it matches the user profile with search results and places products that match the user's preferences at the top of the list.

[1095] The terminal displays the product list received from the server to the user.

[1096] 5. Product Purchase and Payment:

[1097] Users select the products they want to purchase and add them to their cart.

[1098] The terminal transmits the cart information to the server.

[1099] The server manages the cart information and proceeds with the purchase. Specifically, it checks the user's shipping address and payment information and processes the payment.

[1100] The user enters payment information and confirms the purchase.

[1101] The server notifies the user that the payment process is complete.

[1102] 6. Communication with virtual store associates:

[1103] Users can use the chat function within the virtual shop to ask questions to the virtual store clerk.

[1104] The terminal sends the user's question to the server.

[1105] The server uses a generative AI model to generate appropriate answers to user questions. Specifically, it analyzes the content of the question and generates answers from related product information and an FAQ database.

[1106] The terminal displays the response received from the server to the user.

[1107] Examples of specific examples and prompts

[1108] Examples:

[1109] When a user searches for "smartphone," the device sends the keyword "smartphone" to the server.

[1110] The server searches the database and generates a list of products related to "smartphone," such as "smartphone A" and "smartphone B."

[1111] The server displays "Smartphone A" at the top of the list based on the user's past search and purchase history.

[1112] When a user asks the virtual store clerk, "How long is the battery life of this smartphone?", the device sends the question to the server.

[1113] The server uses a generative AI model to generate an answer such as "This smartphone's battery life is approximately 24 hours" and sends it to the device.

[1114] The terminal displays the response received from the server to the user.

[1115] Example prompt sentence:

[1116] "When a user searches on their smartphone, how can I display recommended products based on their past search history?"

[1117] "Create a prompt that will allow the virtual store clerk to generate an appropriate answer to the user's question."

[1118] In this way, users can recreate the shopping experience of a real store from the comfort of their own home.

[1119] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1120] Step 1: Collecting user behavior data

[1121] Users can access the virtual shop to search and browse products.

[1122] The terminal records the user's operations in real time and transmits them to the server.

[1123] Input: User operation information (search keywords, viewed product ID, viewing time, etc.)

[1124] The server stores the received data in a database.

[1125] Output: User behavior data stored in a database

[1126] Step 2: Learn user behavior patterns

[1127] The server periodically extracts user behavior data from the database.

[1128] Input: User behavior data stored in a database

[1129] The server uses a generative AI model (e.g., GPT-4) to analyze user behavior patterns, specifically learning what products users prefer and what time of day they access the site.

[1130] Data processing: behavioral data analysis, pattern recognition

[1131] The server stores the learning results as a user profile.

[1132] Output: User profile

[1133] Step 3: Perform a product search

[1134] Users enter the product name or category into the virtual shop's search bar.

[1135] Input: Search keyword

[1136] The terminal transmits the input search keyword to the server.

[1137] The server searches the database and generates a list of relevant products, specifically retrieving product IDs, product names, prices, stock status, and other information that matches the search keywords.

[1138] Data processing: Keyword search, product information extraction

[1139] Output: Product list

[1140] Step 4: Product Recommendation

[1141] The server prioritizes the display of relevant products based on the user's behavioral patterns. Specifically, it matches the user profile with search results and places products that match the user's preferences at the top of the list.

[1142] Input: User profile, product list

[1143] Data processing: Matching profiles with search results and prioritizing them

[1144] The terminal displays the product list received from the server to the user.

[1145] Output: A prioritized list of products

[1146] Step 5: Purchase and payment

[1147] Users select the products they want to purchase and add them to their cart.

[1148] Input: Selected product information

[1149] The terminal transmits the cart information to the server.

[1150] The server manages the cart information and proceeds with the purchase. Specifically, it checks the user's shipping address and payment information and processes the payment.

[1151] Data processing: cart information management, payment processing

[1152] The user enters payment information and confirms the purchase.

[1153] The server notifies the user that the payment process is complete.

[1154] Output: Purchase completion notification

[1155] Step 6: Communicate with the virtual sales associate

[1156] Users can use the chat function within the virtual shop to ask questions to the virtual store clerk.

[1157] Input: User question

[1158] The terminal sends the user's question to the server.

[1159] The server uses a generative AI model to generate appropriate answers to user questions. Specifically, it analyzes the content of the question and generates answers from related product information and an FAQ database.

[1160] Data processing: Question analysis and answer generation

[1161] The terminal displays the response received from the server to the user.

[1162] Output: Answer from the virtual store clerk

[1163] (Application example 2)

[1164] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1165] Conventional virtual shop systems have had difficulty recreating the shopping experience of a real store from the comfort of your own home. Furthermore, they lacked the functionality to recommend products that matched the user's preferences or the ability to communicate with virtual store staff. This led to issues such as low user satisfaction and a lack of promotion of virtual shop use.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1167] In this invention, the server includes a means for allowing users to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with a virtual store clerk, a means for learning the user's behavior and recommending products that suit the user's preferences from the next time, and a means for asking questions about products and completing purchase procedures through a chat function with the virtual store clerk. This allows users to recreate the shopping experience of an actual store from the comfort of their own home, and further enables product recommendations that suit the user's preferences and smooth communication with the virtual store clerk.

[1168] "VR technology" refers to virtual reality technology, which allows users to experience a virtual environment generated by a computer.

[1169] "Generative AI" refers to artificial intelligence that uses generative models, a technology that generates new information and content based on user behavior and data.

[1170] A "virtual shop" is a virtual store that exists on the Internet, where users can search for, purchase, and pay for products online.

[1171] An "Internet search tool" is software or an application used to search for information on the Internet.

[1172] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[1173] "Payment" refers to the process of paying for goods or services.

[1174] A "virtual store clerk" is an artificial intelligence or avatar that communicates with users within a virtual shop.

[1175] "Learning user behavior" is a technology that analyzes a user's past behavioral data (such as search history and purchase history) and predicts future behavior.

[1176] "Product recommendation" is the act of suggesting products suitable for a user based on the user's preferences and past behavioral data.

[1177] The "chat function" is a function for communicating in real time using text and voice.

[1178] "Questions and purchasing procedures" refers to the user's actions of asking a virtual store clerk a question about a product or completing procedures for purchasing.

[1179] A system for implementing this invention includes a server, a user terminal, and a generative AI model. The server collects and stores user behavior data and uses the generative AI model to recommend products and communicate with virtual store clerks. The user terminal is a device such as a smartphone or tablet, and provides an interface for users to access the virtual shop and perform product searches, purchases, and payments.

[1180] Server Processing

[1181] The server collects user behavior data (search history, purchase history, etc.) and saves it in JSON format. This allows it to learn the user's preferences and behavioral patterns. The server uses a generative AI model (e.g., OpenAI's GPT-3) to recommend products based on the user behavior data. Specifically, it sends the following prompt to the generative AI model:

[1182] User behavior data: {"Search history": ["Smartphone", "Laptop"], "Purchase history": ["Earphones"]}

[1183] Please tell me what products you recommend.

[1184] The generative AI model will then recommend suitable products to the user based on this prompt.

[1185] User terminal processing

[1186] The user terminal provides a virtual shop interface and supports users in searching, purchasing, and paying for products. If a user has a question about a product, they can send it to the virtual shop assistant through a chat function. For example, they can send the following question to the virtual shop assistant:

[1187] I'm looking for a new smartphone. Any recommendations?

[1188] The generative AI model generates an appropriate answer to this question and provides it to the user.

[1189] Hardware and software used

[1190] Hardware: Servers, smartphones, tablets

[1191] Software: OpenAI API, Python, JSON

[1192] Specific examples

[1193] Consider a case where a user accesses a virtual shop using their smartphone and searches for a smartphone. When the user asks the virtual shop assistant, "I'm looking for a new smartphone. Do you have any recommendations?", the generative AI model will recommend an appropriate smartphone based on the user's past behavioral data. In this way, the user can recreate the shopping experience of a real store from the comfort of their own home.

[1194] This system allows users to recreate the shopping experience of a real store from the comfort of their own home, and also enables product recommendations tailored to the user's preferences and smooth communication with virtual store clerks.

[1195] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1196] Step 1:

[1197] A user accesses the virtual shop using a terminal.

[1198] Input: User login information

[1199] Output: Virtual shop home screen

[1200] Specific operation: A user launches the virtual shop application on a smartphone or tablet and enters their login information. The server authenticates the user and displays the home screen.

[1201] Step 2:

[1202] The user searches for a product.

[1203] Input: Search keyword (e.g. "smartphone")

[1204] Output: A list of search results

[1205] Specific operation: The user enters "smartphone" in the search bar and presses the search button. The server retrieves related product information from the database and displays the search results on the user's device.

[1206] Step 3:

[1207] The server collects and stores user behavior data.

[1208] Input: User search history, purchase history

[1209] Output: JSON format behavior data file

[1210] How it works: Every time a user searches for or purchases a product, the server collects that information and stores it in JSON format. For example, "smartphone" is saved as a search history and "earphones" as a purchase history.

[1211] Step 4:

[1212] The server recommends products using a generative AI model.

[1213] Input: User behavior data (JSON format)

[1214] Output: A list of recommended products

[1215] Specific operation: The server reads the saved behavioral data and sends a prompt to the generative AI model. For example, it sends a prompt like, "User behavioral data: {"Search history": ["Smartphone"], "Purchase history": ["Earphones"]} Please recommend some products." The generative AI model recommends products based on this prompt and returns the results to the server.

[1216] Step 5:

[1217] The user uses the chat function with the virtual store clerk.

[1218] Input: User question (e.g., "I'm looking for a new smartphone. Any recommendations?")

[1219] Output: Answer from the virtual store clerk

[1220] Specific operation: The user enters a question in the chat window and presses the send button. The server sends this question to the generative AI model, which generates an appropriate answer. The generated answer is displayed on the user's device.

[1221] Step 6:

[1222] The user purchases the product and makes payment.

[1223] Input: Select the product you want to purchase, payment information

[1224] Output: Purchase confirmation message

[1225] Specific operation: The user selects the product they want to purchase and adds it to their cart. They then enter payment information, such as credit card information, on the payment screen and press the purchase button. The server processes the payment and displays a confirmation message on the user's device that the purchase has been completed.

[1226] Step 7:

[1227] The server updates the user's behavior data.

[1228] Input: New search history, purchase history

[1229] Output: Updated JSON behavior data file

[1230] Specific operation: Every time a user searches for or purchases a new product, the server collects that information and updates the existing behavioral data file. For example, "laptop" is added to the user's search history and "smartphone" is added to the user's purchase history.

[1231] Example 3

[1232] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1233] Conventional tourism experience systems using VR technology lack a sense of realism due to insufficient user gaze tracking and dynamic image changes. Furthermore, when searching for and purchasing products in virtual shops, product recommendations and detailed information tailored to user needs are insufficient. This leads to issues such as lower user satisfaction and a decrease in system usage.

[1234] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1235] In this invention, the server includes means for allowing a user to experience a tourist spot from home, means for using a generative AI model to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop, means for tracking the user's gaze and dynamically changing the scenery of the tourist spot based on that gaze, and means for using the generative AI model to recommend products based on the user's search query. This allows the user to experience images that change in real time according to their gaze movement, and enables them to receive more appropriate product recommendations and detailed information when searching and purchasing in the virtual shop.

[1236] "User" refers to an individual who uses the system to engage in tourism experiences or purchase products.

[1237] "Tourist destinations" refer to places that users can virtually visit using VR technology.

[1238] A "generative AI model" refers to an artificial intelligence algorithm that recommends products based on a user's search queries and history.

[1239] A "virtual shop" refers to an online store where users can search for and purchase products in a virtual space.

[1240] "Internet Search Tool" refers to a function that allows users to search for products within the virtual shop.

[1241] "Product purchase" refers to the act of a user purchasing a product selected within a virtual shop.

[1242] "Payment" refers to the payment procedure carried out by a User when purchasing a Product.

[1243] A "virtual store clerk" refers to a virtual character that communicates with users within a virtual shop and provides product information.

[1244] "Eye tracking" refers to a technology that detects a user's eye movements in real time and transmits that data to a system.

[1245] "Dynamic change" refers to updating images and information in real time according to the user's gaze and movements.

[1246] "360-degree panoramic video" refers to video that allows users to see in all directions.

[1247] "Search query" refers to the keywords or phrases that users enter when searching for products within a virtual shop.

[1248] "Recommending products" refers to the generative AI model suggesting appropriate products based on a user's search query and history.

[1249] This invention is a system that allows users to experience tourist spots from their homes and search for and purchase products in virtual shops. Specific embodiments of this system are described below.

[1250] Hardware and Software

[1251] Hardware

[1252] VR headset: A device worn by a user, such as a VR headset, that allows the user to immerse themselves in a virtual space.

[1253] Eye tracking sensor: Built into the VR headset, it tracks the user's gaze in real time.

[1254] software

[1255] 360-degree panoramic video playback software: When the user moves their gaze, the video changes dynamically.

[1256] Generative AI models: An example is GPT-4, which recommends products based on a user's search queries and history.

[1257] Data processing and calculation

[1258] User gaze tracking

[1259] The device (VR headset) uses a built-in eye tracking sensor to track the user's gaze in real time. The gaze data is processed within the device and sent to a server.

[1260] Dynamic changes in images

[1261] The server receives gaze data sent from the device. Based on the gaze data, the server calculates a 360-degree panoramic image corresponding to the direction the user is looking. For example, if the user looks left, the server calculates the image of the left side. The calculated image is streamed to the device in real time.

[1262] Search and buy in the virtual shop

[1263] A user visits a virtual shop in a VR space and enters a product name (e.g., Yatsuhashi) in the search bar. The search query is sent to a server. The server uses a generative AI model (e.g., GPT-4) to recommend products based on the user's search query. The user asks the virtual store clerk a question via voice or text, for example, "What is the expiration date of this Yatsuhashi?" The server uses the generative AI model to generate a response from the virtual store clerk and provides it to the user. The user selects the product they want to purchase and completes the purchase. The server processes the payment information and notifies the user that the purchase is complete.

[1264] Specific examples

[1265] The user puts on a VR headset at home and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user moves their gaze to the left, the device sends gaze data to the server, which then calculates and streams a video of the stage at Kiyomizu-dera Temple on the left. Next, when the user searches for "yatsuhashi" in a virtual shop, the server uses GPT-4 to recommend "recommended yatsuhashi." The user asks the virtual store clerk, "What is the expiration date of this yatsuhashi?" and receives the response generated by the server. Finally, the user selects a product and completes the purchase process.

[1266] Prompt Sentence Examples

[1267] "The user puts on a VR headset and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user moves their gaze to the left, the image changes to show the stage of Kiyomizu-dera Temple. Next, when the user searches for 'yatsuhashi' in the virtual shop, the generative AI model recommends 'recommended yatsuhashi'. The user asks the virtual shop assistant, 'What is the expiration date of this yatsuhashi?' After obtaining the detailed information, they proceed with the purchase."

[1268] In this way, the user can experience Kyoto's tourist spots in real life and purchase local specialties from the comfort of their own home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1269] Step 1:

[1270] The user puts on the VR headset.

[1271] Input: The user puts on the VR headset.

[1272] Output: The user is connected to the virtual space.

[1273] What happens: The user puts on a VR headset at home and logs into the system, ready to be immersed in the virtual space.

[1274] Step 2:

[1275] The device tracks the user's gaze.

[1276] Input: User's eye movements.

[1277] Output: Gaze data is generated and sent to the server.

[1278] How it works: The eye-tracking sensor built into the VR headset detects the user's gaze in real time, processes the data on the device, and sends it to a server.

[1279] Step 3:

[1280] The server receives the gaze data and calculates the corresponding 360-degree panoramic image.

[1281] Input: Gaze data sent from the device.

[1282] Output: Computed 360-degree panoramic image.

[1283] Specific operation: The server receives the gaze data and calculates the 360-degree panoramic image corresponding to the direction the user is looking. For example, if the user looks left, the server calculates the image of the left side.

[1284] Step 4:

[1285] The server streams the calculated video to the device.

[1286] Input: Computed 360° panoramic image.

[1287] Output: The video is displayed in the user's VR headset.

[1288] How it works: The server streams the calculated 360-degree panoramic image to the device in real time and displays it on the user's VR headset, allowing the user to experience a dynamic change in the image according to their line of sight.

[1289] Step 5:

[1290] A user searches for a product in a virtual shop.

[1291] Input: The product name entered by the user in the search bar (e.g., Yatsuhashi).

[1292] Output: The search query is sent to the server.

[1293] Specific operation: A user accesses a virtual shop in the VR space and enters a product name in the search bar. The search query is sent to the server.

[1294] Step 6:

[1295] The server uses the generative AI model to recommend products.

[1296] Input: The user's search query.

[1297] Output: A list of recommended products.

[1298] How it works: The server uses a generative AI model (e.g., GPT-4) to recommend products based on the user's search query. For example, if you search for "yatsuhashi," the server will list "recommended yatsuhashi."

[1299] Step 7:

[1300] The user communicates with a virtual store clerk.

[1301] Input: User question (e.g., What is the expiration date of this yatsuhashi?).

[1302] Output: Response from the virtual store clerk.

[1303] Specific operation: The user asks a question to a virtual clerk in the virtual shop by voice or text. The server uses the generative AI model to generate a response from the virtual clerk and provides it to the user.

[1304] Step 8:

[1305] The user purchases the product.

[1306] Input: User selected product and payment information.

[1307] Output: Confirmation of successful purchase.

[1308] Specific behavior: The user selects the product they want to purchase and completes the purchase. The server processes the payment information and notifies the user that the purchase is complete.

[1309] (Application example 3)

[1310] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1311] Conventional tourism experience systems using virtual reality (VR) technology often lack a sense of visual immersion when users experience tourist destinations from home. Furthermore, product search and purchasing procedures in virtual shops lacked sufficient product recommendations based on user preferences and detailed information. Furthermore, communication with virtual store clerks was limited, making it difficult to improve the user experience.

[1312] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for allowing a user to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop, a means for dynamically changing images according to the user's line of sight, a means for the generative AI to recommend products based on the user's preferences, and a means for providing detailed product information through dialogue with a virtual store clerk. This allows a user to experience tourist spots with a high level of immersion even from the comfort of their own home, and improves the shopping experience in the virtual shop.

[1313] "VR technology" is a technology for realizing virtual reality, allowing users to immerse themselves in a computer-generated virtual environment and experience the same as the real world.

[1314] "Generative AI" is a type of artificial intelligence that has the ability to generate new information and content based on user input and data.

[1315] A "virtual shop" is a virtual store that exists on the Internet, where users can search, browse and purchase products online.

[1316] An "Internet search tool" is software or an application for searching information on the Internet, allowing users to efficiently find specific information.

[1317] "Product purchase and payment" refers to the process in which a user purchases a selected product and pays for it.

[1318] A "virtual store clerk" is a virtual assistant that interacts with users in a virtual shop, providing information about products and assisting them in the purchasing process.

[1319] "Means for dynamically changing images according to the user's line of sight" refers to technology that tracks the user's line of sight and changes the displayed image in real time according to the direction of that gaze.

[1320] "Means for recommending products based on user preferences" refers to technology that analyzes a user's past purchase history, search history, and other data to automatically suggest products that the user may be interested in.

[1321] "Means for providing detailed product information through dialogue with a virtual store clerk" refers to technology that allows users to obtain detailed information about a product, such as its features, specifications, and usage, by dialogue with a virtual store clerk.

[1322] A system for implementing the present invention has the following configuration.

[1323] System Configuration

[1324] 1. Hardware

[1325] VR headset: A device that allows a user to immerse themselves in a virtual reality environment. An example is a VR headset.

[1326] Camera: A device for tracking the user's gaze.

[1327] Server: A computer system for processing data and running generative AI models.

[1328] 2. Software

[1329] OpenAI API: An API for running generative AI models.

[1330] OpenCV: A library for image processing.

[1331] Flask: A web framework used to build server-side applications that allow interaction with virtual store associates.

[1332] Data processing and calculation

[1333] 1. Importing 360-degree panoramic footage

[1334] The server uses OpenCV to load the 360-degree panoramic video, which is displayed when the user puts on the VR headset.

[1335] 2. Dynamic image changes according to the user's gaze

[1336] The camera tracks the user's gaze and sends the data to a server, which uses OpenCV to dynamically change the image based on the user's gaze.

[1337] 3. Product Recommendations Using Generative AI

[1338] The server uses the OpenAI API to recommend products based on the user's preferences, analyzing the user's purchase and search history to suggest suitable products.

[1339] 4. Interacting with a virtual store clerk

[1340] The server uses Flask to interact with the virtual store clerk, receives messages from the user, and uses the OpenAI API to generate responses from the virtual store clerk.

[1341] Specific examples

[1342] When a user types, "I'm looking for Kyoto specialties," the generative AI responds, "The recommended specialties are yatsuhashi and matcha." When a user types, "Please tell me more about yatsuhashi," the virtual store clerk provides more information, saying, "Yatsuhashi is a traditional Japanese sweet from Kyoto, characterized by its cinnamon-flavored bean paste."

[1343] Prompt Sentence Examples

[1344] User: Looking for Kyoto specialties

[1345] Virtual Salesperson: Recommended local products are Yatsuhashi and Matcha.

[1346] User: Please tell me more about Yatsuhashi

[1347] Virtual Salesperson: Yatsuhashi is a traditional Japanese sweet from Kyoto, characterized by its cinnamon-flavored bean paste.

[1348] In this way, users can experience tourist destinations in a highly immersive way from the comfort of their own home, and the shopping experience in virtual shops can also be improved.

[1349] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1350] Step 1:

[1351] The server loads the 360-degree panoramic video using OpenCV. The input is the file path of the 360-degree panoramic video, and the output is the loaded image data. Specifically, the server loads the image file from the specified file path and stores the image data in memory.

[1352] Step 2:

[1353] The camera tracks the user's gaze and sends the data to the server. The input is the user's gaze data, and the output is the gaze data sent to the server. Specifically, the camera detects the user's gaze direction in real time and sends the data to the server.

[1354] Step 3:

[1355] The server uses OpenCV to dynamically change the image according to the user's gaze. The input is gaze data and loaded image data, and the output is image data shifted according to the gaze. Specifically, the server shifts the image based on the gaze data and generates an image that corresponds to the direction the user is looking.

[1356] Step 4:

[1357] The server uses the OpenAI API to recommend products based on the user's preferences. The input is the user's purchase history and search history, and the output is a list of recommended products. Specifically, the server analyzes the user's history data and sends prompts to the generative AI model to receive product recommendations.

[1358] Step 5:

[1359] The server uses Flask to realize interaction with the virtual store clerk. The input is a message from the user, and the output is the response of the virtual store clerk. Specifically, the server receives a message from the user, sends a prompt to the OpenAI API to generate a response for the virtual store clerk, and returns it to the user.

[1360] Step 6:

[1361] The user wears a VR headset and watches the video sent from the server. The input is the shifted image data sent from the server, and the output is the user's visual experience. Specifically, the user experiences a dynamically changing image according to their line of sight through the VR headset.

[1362] Step 7:

[1363] A user searches for a product in a virtual shop and purchases a product recommended by a generative AI. The input is the user's search query and a list of recommended products, and the output is the completion of the purchase process. Specifically, the user searches for a product in the virtual shop, selects a recommended product, and then completes the purchase process.

[1364] Step 8:

[1365] The user obtains detailed product information through a dialogue with a virtual store clerk. The input is a question from the user and the response from the virtual store clerk, and the output is detailed product information. Specifically, the user asks a question to the virtual store clerk and obtains detailed product information through the response generated by the generative AI model.

[1366] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1367] "Example 1"

[1368] In one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and optimizes the sightseeing experience based on the emotions. Specifically, when a user feels joy, the system provides scenery of a tourist spot that amplifies the joy. When a user feels depressed, the system provides scenery that soothes the user's mood.

[1369] "Example 2"

[1370] Furthermore, the emotion engine can be used to recommend products in a virtual shop. When a user feels happy, the system will recommend products that will increase that happiness. When a user feels depressed, the system will recommend products that will ease that mood.

[1371] "Example 3"

[1372] As a concrete example, consider the case where a user experiences Kyoto's tourist attractions from home. The user puts on a VR headset and experiences Kyoto's tourist attractions through a 360-degree panoramic image. At this time, the emotion engine senses the user's joy and dynamically changes the image according to that joy. For example, if the user feels joy from a particular view, that view will be displayed more vividly. Furthermore, in the virtual shop, Kyoto's famous products will be recommended according to the user's joy.

[1373] The processing flow of each embodiment will be described below.

[1374] "Example 1"

[1375] Step 1: The user puts on the VR headset and starts the system.

[1376] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[1377] Step 3: The emotion engine dynamically changes the scenery of the tourist spot based on the user's emotions.

[1378] "Example 2"

[1379] Step 1: A user visits a virtual shop.

[1380] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[1381] Step 3: The sentiment engine recommends products based on user sentiment.

[1382] "Example 3"

[1383] Step 1: The user puts on a VR headset and experiences Kyoto's tourist attractions.

[1384] Step 2: The system activates the emotion engine and recognizes the user's emotion.

[1385] Step 3: The emotion engine dynamically changes the video based on the user's emotions.

[1386] Step 4: In the virtual shop, the emotion engine recommends Kyoto specialties based on the user's emotions.

[1387] Example 1

[1388] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1389] Conventional VR tourism experience systems were unable to dynamically change the scenery based on eye movements or location information when users experienced tourist spots from home. They also lacked the ability to recognize users' emotions and optimize the tourism experience based on those emotions. Furthermore, they lacked a means to provide integrated internet search tools, product purchase and payment options, and communication with virtual store staff in virtual shops. To address these issues, the present invention provides a new system.

[1390] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1391] In this invention, the server includes: means for allowing a user to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; means for tracking the user's eye movements and location information and dynamically changing the scenery of the tourist spot based on that; and means for recognizing the user's emotions and optimizing the tourist experience based on those emotions. This allows the user to have a tourist experience that changes dynamically based on their eye movements and location information from the comfort of their own home, and further enables them to enjoy an optimal tourist experience according to their emotions.

[1392] "VR technology" is a technology for realizing virtual reality, allowing users to have a reality-like experience in a virtual space.

[1393] "Generative AI" is a type of artificial intelligence that generates new information and content based on user input and past data.

[1394] A "virtual shop" is a virtual store operated on the Internet, an online platform where users can search for, purchase, and pay for products.

[1395] "Gaze movement" refers to the movement of the user's eyes and is information that indicates the direction in which the user is looking.

[1396] "Location information" is data that indicates the user's physical location and is information used to identify where the user is in the virtual space.

[1397] "Emotion recognition" is a technology that reads emotions from a user's facial expressions, voice, etc., and is a means of determining the user's emotional state.

[1398] "Tourism experience optimization" is the process of adjusting video and audio to enhance the tourism experience based on user emotions and behavior.

[1399] This invention is a VR system that allows users to experience tourist destinations from the comfort of their own homes. Users wear a VR headset and can experience tourist destinations through 360-degree panoramic images. The system also tracks the user's gaze and location, dynamically changing the images of the tourist destinations. Furthermore, by incorporating an emotion engine, the system optimizes the tourist experience based on the user's emotions.

[1400] Hardware and software used

[1401] Hardware: VR headset, eye tracking sensor, position tracking sensor

[1402] Software: Game engine, emotion recognition software

[1403] Data processing and calculation

[1404] Eye tracking and location acquisition

[1405] When a user puts on a VR headset, eye-tracking and position-tracking sensors capture the user's eye movements and location in real time, determining in which direction the user is looking and where they are.

[1406] Dynamic changes in images

[1407] Using a game engine, the system dynamically generates images of tourist spots based on the acquired gaze and position information. For example, if the user turns left, the image changes to show the scenery on the left.

[1408] Emotion Recognition and Experience Optimization

[1409] Using emotion recognition software (such as Affectiva), the system recognizes emotions from the user's facial expressions and voice. Based on the recognized emotions, the system optimizes the video and audio of tourist spots. For example, if the user is happy, the system will provide scenery and music that amplify that happiness.

[1410] Specific examples

[1411] Example 1

[1412] A user puts on a VR headset and experiences the Eiffel Tower in Paris. When the user looks up, the device receives that information and the server generates an image of the top of the Eiffel Tower. The device displays that image on the VR headset. When the user smiles, the device receives that information and the server generates an illuminated night view of the Eiffel Tower. The device displays that image on the VR headset.

[1413] Example 2

[1414] When a user is feeling depressed, the device receives this information, and the server generates a video of a tranquil Japanese garden and calming music. The device then displays the video and music on the VR headset, which helps soothe the user's mood.

[1415] Prompt Sentence Examples

[1416] Prompt 1: "Generate a scenario in which a user puts on a VR headset and experiences the Eiffel Tower in Paris. When the user looks up, the top of the Eiffel Tower should be visible, and when the user smiles, the Eiffel Tower should be illuminated at night."

[1417] Prompt 2: "Generate a scenario that displays a video of a tranquil Japanese garden when the user is feeling depressed. Also play calming music to soothe the user's mood."

[1418] This system allows users to experience sightseeing spots from the comfort of their own home, as if they were actually visiting them. The introduction of an emotion engine provides an optimal sightseeing experience based on the user's emotions.

[1419] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1420] Step 1:

[1421] The user puts on the VR headset.

[1422] Input: The user puts on the VR headset.

[1423] Output: VR system startup.

[1424] What happens: The user puts on the VR headset and the system starts automatically.

[1425] Step 2:

[1426] The device activates the eye tracking and position tracking sensors.

[1427] Input: VR headset wearing information.

[1428] Output: Activation of the eye tracking and position tracking sensors.

[1429] Specific operation: The device activates the gaze tracking sensor and position tracking sensor built into the VR headset and prepares to acquire the user's gaze and position information.

[1430] Step 3:

[1431] The device acquires the user's gaze and location information in real time.

[1432] Input: User's eye movements and location.

[1433] Output: Real-time gaze and position data.

[1434] Specific operation: The device uses eye tracking and position tracking sensors to capture the user's eye movements and position information in real time and saves them as data.

[1435] Step 4:

[1436] The server receives the acquired gaze and location information and dynamically generates images of tourist spots.

[1437] Input: Gaze and location data sent from the device.

[1438] Output: Dynamically generated video of tourist attractions.

[1439] Specific operation: The server receives gaze and position information sent from the device and dynamically generates a video of the tourist spot using a game engine. For example, if the user looks left, the left view is generated.

[1440] Step 5:

[1441] The device displays the generated image on the VR headset.

[1442] Input: Generated video sent from the server.

[1443] Output: Video displayed in a VR headset.

[1444] Specific operation: The device receives the generated video sent from the server and displays it on the VR headset, allowing the user to experience images of tourist spots that respond to their line of sight.

[1445] Step 6:

[1446] The device activates emotion recognition software to recognize the user's emotions.

[1447] Input: User's facial expressions and voice.

[1448] Output: Recognized emotion data.

[1449] Specific operation: The device activates emotion recognition software and recognizes emotions from the user's facial expressions and voice. For example, if the user smiles, that information is captured.

[1450] Step 7:

[1451] The server optimizes the video and audio based on the recognized emotion.

[1452] Input: Emotion data sent from the device.

[1453] Output: Optimized video and audio.

[1454] How it works: The server receives emotion data sent from the device and optimizes the video and audio based on the recognized emotion. For example, if the user is happy, the system will generate scenery and music that amplify that joy.

[1455] Step 8:

[1456] The device displays optimized video and audio on the VR headset.

[1457] Input: Optimized video and audio sent from the server.

[1458] Output: Optimized video and audio displayed on a VR headset.

[1459] Specific operation: The device receives the optimized video and audio sent from the server and displays them on the VR headset, allowing the user to have an optimal sightseeing experience according to their emotions.

[1460] (Application example 1)

[1461] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1462] Conventional virtual reality (VR) systems and virtual shops using generative AI did not dynamically change or optimize the display based on the user's gaze, location, or emotions. This limited the user experience, making it difficult to provide an optimal shopping experience for each individual user. Furthermore, because the display content was not optimized based on the user's emotions, it was not possible to increase user satisfaction.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for using VR technology to enable a user to experience tourist spots from home, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks, a means for tracking the user's gaze and location information and dynamically changing the display content based on that, and a means for recognizing the user's emotions and optimizing the display content based on those emotions. This makes it possible to dynamically change the display content based on the user's gaze, location information, and emotions, providing an optimal shopping experience for each individual user.

[1464] "VR technology" stands for virtual reality technology, which allows users to experience computer-generated virtual environments.

[1465] "Generative AI" stands for generative artificial intelligence, an artificial intelligence technology that generates new information and content based on user input and data.

[1466] A "virtual shop" is a virtual store that exists on the Internet, where users can search for, purchase, and pay for products online.

[1467] An "Internet search tool" is software or an application used to search for information on the Internet.

[1468] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[1469] "Payment" is the process by which a user makes payment when purchasing a product.

[1470] A "virtual store clerk" is a virtual character that communicates with users within the virtual shop and provides product information and support.

[1471] "Gaze and location tracking" is a technology that detects the user's gaze movements and location in real time and collects that data.

[1472] "Dynamic change" refers to changing the displayed content in real time based on the user's gaze and location information.

[1473] "Emotion recognition" is a technology that detects a user's emotional state from their facial expressions and behavior.

[1474] "Optimizing display content" refers to adjusting the displayed information and content to the optimal form based on the user's emotions and behavior.

[1475] The system for implementing this invention has the following configuration. First, the user puts on a VR headset and accesses a virtual shop from home. The VR headset is equipped with cameras and sensors that track the user's line of sight and location information in real time. This allows the user to walk freely around the virtual shop and view the products in a 360-degree panoramic view.

[1476] The server uses generative AI to enable internet search tools, product purchases and payments, and communication with virtual store staff in the virtual shop. The generative AI also has the ability to recommend products based on the user's purchase and search history. It also tracks the user's gaze and location information and dynamically changes the content displayed based on that information. For example, when a user focuses their gaze on a particular product, detailed information and reviews of that product are displayed.

[1477] The server also uses an emotion recognition model to recognize the user's emotions in real time. The emotion recognition model is pre-trained using machine learning libraries such as TensorFlow. Based on the user's emotions, the display content is optimized. For example, if the user is happy, it will display special promotions and recommended products, and if the user is depressed, it will provide relaxing music and scenery.

[1478] A specific example is a system that displays detailed information about products that a user looks at in a virtual shop and optimizes the shopping experience based on the user's emotions using an emotion recognition model.This system displays detailed information about products that a user looks at in a virtual shop and optimizes the shopping experience based on the user's emotions using an emotion recognition model.

[1479] An example prompt is, "Develop an application that displays detailed information about products when a user looks at them and uses an emotion recognition model to optimize the shopping experience based on the user's emotions."

[1480] In this way, the content displayed can be dynamically changed based on the user's gaze, location, and emotions, providing an optimal shopping experience for each individual user.

[1481] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1482] Step 1:

[1483] The user wears a VR headset and accesses the virtual shop. The VR headset is equipped with cameras and sensors to track the user's gaze and position in real time. The input is the user's gaze data and position data, which are sent from the VR headset to a server. The output is an initial display of the virtual shop based on the user's gaze and position.

[1484] Step 2:

[1485] The server uses generative AI to enable internet search tools within the virtual shop, product purchases and payments, and communication with virtual store staff. Inputs include the user's search queries, purchase history, and search history. The generative AI processes this data and recommends the most suitable products and information to the user. The output is a product list and detailed information that is displayed in the user's field of view.

[1486] Step 3:

[1487] The server tracks the user's gaze and location information and dynamically changes the display content based on that information. The inputs include gaze data and location data, which are updated in real time. The server analyzes this data to identify products and areas of interest to the user. The output displays detailed information and reviews about the products the user is looking at.

[1488] Step 4:

[1489] The server uses an emotion recognition model to recognize the user's emotions in real time. Inputs include the user's facial expression data and voice data. The emotion recognition model is pre-trained using machine learning libraries such as TensorFlow and analyzes this data to identify the user's emotional state. The output is optimization of the display content based on the user's emotions.

[1490] Step 5:

[1491] The server optimizes the display content based on the user's emotions. The input is emotional data obtained from an emotion recognition model. Based on this data, the server displays special promotions and recommended products if the user is happy, and provides relaxing music and scenery if the user is depressed. The output is the optimal display content based on the user's emotions.

[1492] Step 6:

[1493] The user selects a product in the virtual shop and completes the purchase. The inputs include the user's selection data and payment information. The server processes this data and completes the purchase. The output is displayed to the user as a purchase confirmation message and a receipt.

[1494] Step 7:

[1495] The server stores the user's purchase history and search history and uses it for the next shopping experience. The input is the user's purchase data and search data. The server stores this data in a database, and the generative AI uses this data for the next shopping experience. The output is that the user's history data is updated.

[1496] Example 2

[1497] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1498] Conventional online shopping systems have difficulty recreating the shopping experience of a physical store from the comfort of your own home. Furthermore, they do not recommend products based on the user's emotions, which prevents them from increasing user satisfaction. Furthermore, they have the problem of being unable to effectively utilize the user's purchase history and search history to reflect in the next recommendation.

[1499] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1500] In this invention, the server includes: a means for allowing users to experience tourist spots from home using VR technology; a means for using generative AI to enable internet search tools, product purchases and payments in a virtual shop, and communication with virtual store clerks; a means for receiving a user's search query and recommending products using a generative AI model; a means for analyzing the user's emotions using an emotion engine and recommending additional products based on those emotions; and a means for saving the user's purchase history and using it for future recommendations. This allows users to recreate the shopping experience of a real store from the comfort of their own home, enabling product recommendations based on the user's emotions and increasing user satisfaction. Furthermore, the server can effectively utilize the user's purchase history and search history to reflect in future recommendations.

[1501] "VR technology" is a technology that uses virtual reality to allow users to experience an environment different from the real world.

[1502] "Generative AI" is an artificial intelligence technology that generates new information and content based on user input and behavioral data.

[1503] A "virtual shop" is a store virtually built on the Internet, and is an online platform where users can search for and purchase products.

[1504] An "Internet search tool" is software or functionality that allows users to search for specific information or products on the Internet.

[1505] "Purchase of product" refers to the act of a user purchasing a product selected in a virtual shop.

[1506] "Payment" is the process by which a user makes payment when purchasing a product.

[1507] A "virtual store clerk" is an artificial intelligence or software that interacts with users in a virtual shop, providing information about products and assisting with the purchasing process.

[1508] A "search query" is a search term or keyword that a user enters into a search tool.

[1509] A "generative AI model" is an algorithm or program that forms the core of generative AI and generates new information based on user input data.

[1510] An "emotion engine" is a technology that analyzes a user's emotional state and provides appropriate responses and recommendations based on the results.

[1511] "Purchase history" is a record of products a user has purchased in the past.

[1512] "Search history" is a record of searches a user has conducted in the past.

[1513] "Recommendation" is the act of presenting appropriate products or information based on a user's behavioral data and emotional state.

[1514] This invention is a system that allows users to recreate the shopping experience of a real store from the comfort of their own home. The system combines VR technology, generative AI, and an emotion engine.

[1515] The server hosts a generative AI model and collects and analyzes user behavior data. The generative AI model uses a common generative AI algorithm (e.g., a natural language processing model). The server stores data when users search for and purchase products in the virtual shop and uses it for future recommendations. It also uses an emotion engine (e.g., an emotion recognition API) to analyze the user's emotional state and recommend products based on that.

[1516] The device (user's PC or smartphone) provides the virtual shop interface and accepts operations that allow users to search for and purchase products. The device communicates with the server and displays recommendation results from the generative AI model and the results of communication with the virtual store clerk. Users access the virtual shop through their device to search for and purchase products.

[1517] Users access the virtual shop to search for and purchase products. For example, if a user types "I'm looking for the latest smartphone," the generative AI model will recommend smartphones that fit the user's preferences based on the user's past search and purchase history. Furthermore, if the user is happy, the emotion engine will analyze their emotions and recommend related products (e.g., smartphone accessories) to amplify their joy.

[1518] As a concrete example, consider the case where a user accesses a virtual shop and searches for the "latest smartphone." The server uses a generative AI model to recommend the "latest smartphone." Furthermore, the emotion engine analyzes the user's joy and recommends related products such as "smartphone cases" and "wireless earphones" to amplify that joy. The user selects the recommended "latest smartphone" and completes the purchase process. The server processes the payment and saves the purchase history.

[1519] An example of a prompt is as follows:

[1520] "A user is searching for the latest smartphone in a virtual shop. Based on their past search and purchase history, recommend smartphones that match their preferences. Also, if the user is happy, recommend related products that will amplify that emotion."

[1521] In this way, by clarifying the roles of the server, terminal, and user, and explaining the flow of data processing and calculation using specific hardware and software, it becomes easier to understand the overall picture of the system.

[1522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1523] Step 1:

[1524] The user visits the virtual shop.

[1525] Users access the virtual shop's website or app using their PC or smartphone. If user login information is required, a login screen is displayed. The input is the user's access information, and the output is the virtual shop's home screen.

[1526] Step 2:

[1527] A user searches for a product.

[1528] A user enters a keyword into the search bar and clicks the search button, for example, "latest smartphones." The input is the user's search query, and the output is the submission of the search query.

[1529] Step 3:

[1530] The server receives the user's search query.

[1531] The server receives the user's search query and queries the database. The input is the user's search query and the output is the query to the database.

[1532] Step 4:

[1533] The server uses the generative AI model to recommend products.

[1534] The server uses a generative AI model to recommend products based on the user's search query. The generative AI model also takes into account the user's past search and purchase history. The input is search results from the database and the user's historical data, and the output is a list of recommended products.

[1535] Step 5:

[1536] The server analyzes the user's emotions using an emotion engine.

[1537] The server uses an emotion engine to analyze the user's emotional state, for example, to determine whether the user is happy or depressed. The input is the user's behavioral and facial expression data, and the output is the analyzed emotional state.

[1538] Step 6:

[1539] The server then recommends additional products based on the sentiment.

[1540] The server recommends additional products that match the user's emotions based on the analysis results of the emotion engine. For example, if the user is happy, it recommends related products that will amplify that happiness. The input is the analyzed emotional state, and the output is a list of additional recommended products.

[1541] Step 7:

[1542] The device displays the recommendations to the user.

[1543] The terminal displays the recommendation results received from the server to the user. The user can check the recommended products on the screen. The input is the recommendation results from the server, and the output is the recommendation results displayed on the user's screen.

[1544] Step 8:

[1545] The user selects a product and completes the purchase.

[1546] The user selects the recommended products and completes the checkout process, which includes entering shipping and payment information. The input is the user's selection and purchase information, and the output is the completion of the checkout process.

[1547] Step 9:

[1548] The server processes the payment.

[1549] The server receives the user's payment information and processes the payment. Once the payment process is complete, a confirmation email is sent to the user. The input is the user's payment information, and the output is a notification that the payment has been completed.

[1550] Step 10:

[1551] The server stores your purchase history and uses it for future recommendations.

[1552] The server stores the user's purchase history in a database. The stored data is used for the next recommendation. The input is the user's purchase history, and the output is the updated database.

[1553] (Application example 2)

[1554] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1555] Conventional virtual shop systems did not adequately recommend products based on the user's preferences and emotions, limiting the user experience. Communication with virtual store clerks was also limited, making it difficult to fully replicate the shopping experience in a real store. Furthermore, there was a lack of technology to analyze user emotions and reflect them in product recommendations, which prevented users from increasing satisfaction.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1557] In this invention, the server includes: a means for allowing a user to experience tourist spots from home using VR technology; a means for enabling internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop using generative AI; a means for recommending products based on the user's emotions using an emotion engine; a means for analyzing emotions from the user's facial expressions and voice; a means for generating product recommendations based on the user's queries and emotions using a generative AI model; and a means for real-time communication with the virtual store clerk. This enables product recommendations based on the user's preferences and emotions, recreating the shopping experience in a real store. Furthermore, analyzing the user's emotions and reflecting them in product recommendations can increase user satisfaction.

[1558] "VR technology" is a technology that uses virtual reality to allow users to experience a virtual space that is different from the real world.

[1559] "Generative AI" is an artificial intelligence technology that generates new information and content based on user input and actions.

[1560] A "virtual shop" is a store virtually constructed on the Internet, and is an online platform where users can search for, purchase, and pay for products.

[1561] An "Internet search tool" is software or a system that allows users to search for information or products on the Internet.

[1562] "Purchasing a product" refers to the act of a user selecting a product in a virtual shop and completing the purchase process.

[1563] "Payment" is the process by which a user makes payment when purchasing a product.

[1564] A "virtual store clerk" is a virtual entity that communicates with users within the virtual shop, answering questions and assisting them with the purchasing process.

[1565] An "emotion engine" is a technology that analyzes a user's emotions and responds or makes recommendations based on those emotions.

[1566] "Means for analyzing emotions from a user's facial expressions and voice" refers to technology that uses a camera or microphone to capture a user's facial expressions and voice, and then analyzes them to identify emotions.

[1567] A "generative AI model" is a specific implementation of generative AI, a model for generating new information and content based on user queries and emotions.

[1568] A "user query" is a search or question that a user enters into the system.

[1569] "Product recommendation" is the act of a system suggesting appropriate products to a user based on the user's preferences and behavior.

[1570] "Real-time communication" is a form of communication that allows users and systems to exchange information instantly.

[1571] A system for carrying out this invention has the following configuration: A server includes means for allowing a user to experience tourist spots from home using VR technology, means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with a virtual store clerk, means for recommending products according to the user's emotions using an emotion engine, means for analyzing emotions from the user's facial expressions and voice, means for generating product recommendations based on the user's queries and emotions using a generative AI model, and means for real-time communication with a virtual store clerk.

[1572] Hardware and Software Configuration

[1573] Hardware: Smartphone, Head-Mounted Display (HMD), Camera, Microphone

[1574] Software: OpenAI API, emotion recognition library, OpenCV

[1575] Data processing and calculation

[1576] 1. Data capture: The user's facial expressions and voice are captured using a camera and microphone, allowing for real-time emotional data to be obtained.

[1577] 2. Emotion Analysis: Using an emotion recognition library, we analyze captured facial and voice data to identify the user's emotions.

[1578] 3. Use of generative AI models: Based on user queries and sentiment data, prompts are sent to generative AI models to generate product recommendations.

[1579] 4. Communication with a virtual store clerk: A generative AI model is used to generate answers from a virtual store clerk in response to user questions, and these answers are provided to the user in real time.

[1580] Specific examples

[1581] Product recommendations based on user queries and sentiment

[1582] User query: "Looking for a new smartphone"

[1583] User emotion: "Delight"

[1584] Prompt for the generative AI model:

[1585] User Query: Looking for a new smartphone

[1586] User Emotion: Delight

[1587] Please tell me what products you recommend.

[1588] Communication with virtual store associates

[1589] User Question: "What is the battery life of this phone?"

[1590] Prompt for the generative AI model:

[1591] User Question: What is the battery life of this phone?

[1592] Please tell us the answer of the virtual salesperson.

[1593] In this way, the server utilizes generative AI models based on the user's queries and emotions to provide appropriate product recommendations and enable communication with virtual store associates, allowing users to recreate the shopping experience of a real store from the comfort of their own home.

[1594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1595] Step 1:

[1596] Users access the virtual shop using a smartphone or head-mounted display (HMD). They input facial expressions and voice through a camera and microphone, which captures the user's real-time emotional data.

[1597] Step 2:

[1598] The device sends the captured facial and voice data to an emotion recognition library, which analyzes the data and identifies the user's emotions. The input is facial and voice data, and the output is the user's emotion (e.g., joy, sadness).

[1599] Step 3:

[1600] A user inputs a query to search for a product in the virtual shop. For example, the text input is "I'm looking for a new smartphone." The input is the user's query.

[1601] Step 4:

[1602] The server sends the user's query and emotion data to the generative AI model, which generates prompts and product recommendations based on these inputs. The input is the user's query and emotion data, and the output is a list of recommended products.

[1603] Step 5:

[1604] The server displays the product recommendations obtained from the generative AI model to the user. The user can check the list of recommended products and view detailed information. The input is the list of recommended products, and the output is the user's browsing behavior.

[1605] Step 6:

[1606] The user inputs a question to the virtual store clerk. For example, text input is made such as "What is the battery life of this smartphone?" The input is the user's question.

[1607] Step 7:

[1608] The server sends the user's question to the generative AI model, which generates prompts based on the question and generates answers from the virtual store clerk. The input is the user's question, and the output is the answer from the virtual store clerk.

[1609] Step 8:

[1610] The server displays the answer of the virtual clerk obtained from the generative AI model to the user. The user can confirm the answer and continue asking questions. The input is the answer of the virtual clerk, and the output is the user's next action.

[1611] Step 9:

[1612] If the user decides to make a purchase, the server initiates the checkout process: the user enters payment information, and the server processes the payment through the payment system. The input is the user's payment information, and the output is confirmation of the successful purchase.

[1613] Step 10:

[1614] After a purchase is completed, the server stores the user's purchase history and search history in a database. This improves the accuracy of recommendations from next time onwards. The input is the user's purchase history and search history, and the output is an updated database.

[1615] Example 3

[1616] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1617] Conventional tourism experience systems using VR technology allow users to visually experience tourist destinations, but they are unable to dynamically change the images according to the user's emotions or gaze, making it difficult to provide a realistic experience. Furthermore, there is also the issue of not being able to optimally recommend products that take the user's emotions into account when recommending products in virtual shops or during the purchasing process. This makes it difficult to increase user satisfaction.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1619] In this invention, the server includes: means for allowing a user to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; means for acquiring the user's gaze information and dynamically changing the scenery of the tourist spot based on that; means for analyzing the user's facial expressions and voice to determine their emotions and dynamically adjusting the images according to those emotions; and means for the generative AI to recommend products based on the user's emotions. This allows the user to experience images that change in real time according to their gaze and emotions, and to receive optimal product recommendations based on their emotions.

[1620] "VR technology" is a technology for realizing virtual reality, allowing users to visually and experientially interact in a virtual space using dedicated headsets and devices.

[1621] "Generative AI" is a type of artificial intelligence, a system that has the ability to generate new information and recommendations based on user input data and history.

[1622] A "virtual shop" is a virtual store that exists on the Internet and is an online platform where users can search, browse and purchase products.

[1623] An "Internet search tool" is software or a system that allows users to search for specific information or products on the Internet.

[1624] "Product purchase and payment" refers to the process of purchasing the product selected by the user and making the payment.

[1625] A "virtual store clerk" is a virtual character or system that communicates with users within a virtual shop and explains and recommends products.

[1626] "Gaze information" is data that indicates the direction the user is looking, and is information obtained through VR headsets and other devices.

[1627] "Determining emotions by analyzing facial expressions and voice" is the process of identifying a user's emotional state at that time by analyzing their facial expressions and tone of voice.

[1628] "Dynamic image adjustment" refers to changing the color, brightness, contrast, etc. of the image in real time based on the user's emotions and gaze information.

[1629] "Product recommendation" is the process of suggesting the most suitable product based on the user's past behavior and current emotional state.

[1630] The present invention provides a system that allows users to experience tourist spots from their own homes and purchase products in virtual shops. Specific embodiments of this system are described below.

[1631] Hardware and software used

[1632] Hardware: VR headset (e.g., a general VR device), PC, or smartphone

[1633] Software: 360-degree panoramic video playback software, generative AI model (e.g., general generative AI), emotion engine

[1634] System Overview

[1635] Users wear a VR headset and experience tourist attractions through 360-degree panoramic images. The system dynamically adjusts video and product recommendations based on real-time capture of the user's gaze, facial expressions, and voice. In the virtual shop, a generative AI model recommends optimal products based on the user's emotions and history, and provides detailed information through interaction with a virtual sales associate.

[1636] Data processing and calculation

[1637] 1. Video playback:

[1638] The server streams 360-degree panoramic video data of tourist spots to the user's device.

[1639] The device acquires the user's gaze information through the VR headset's sensors and sends it to the server.

[1640] The server dynamically plays back the video based on the line-of-sight information.

[1641] 2. Emotion analysis:

[1642] The device uses the camera and microphone built into the VR headset to capture the user's facial expressions and voice.

[1643] The emotion engine analyzes this data to determine the user's emotional state.

[1644] The server dynamically adjusts the color and brightness of the video based on the emotion analysis results.

[1645] 3. Product recommendation:

[1646] When a user visits the virtual shop, a generative AI model recommends the most suitable products based on the user's emotions and history.

[1647] Users can communicate with virtual store clerks via voice and text to obtain detailed product information.

[1648] When the user decides to make a purchase, the terminal sends data to the server to complete the purchase procedure, and the purchase is completed.

[1649] Specific examples

[1650] The user wears a standard VR device at home and experiences a scene of visiting Kiyomizu-dera Temple in Kyoto. When the user is moved by the view from the stage of Kiyomizu-dera Temple and smiles, the device sends the facial expression to the emotion engine. The emotion engine analyzes the user's joy, and the server adjusts the image to make it more vivid. The user then searches for yatsuhashi in a virtual shop and purchases the product recommended by the generation AI. Through a conversation with a virtual store clerk, the user obtains detailed information about the yatsuhashi and completes the purchase process.

[1651] Prompt Sentence Examples

[1652] "When a user is moved by the view from the stage of Kiyomizu-dera Temple and smiles, the emotion engine should detect that joy, and the server should display the image more vividly. Also, when a user searches for yatsuhashi in a virtual shop, the generative AI should recommend the most suitable product and provide detailed information through a conversation with a virtual store clerk."

[1653] In this way, the system allows the user to experience the tourist spot in real life and purchase local specialties from the comfort of their own home. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1654] Step 1:

[1655] The user puts on the VR headset.

[1656] Input: The user puts on the VR headset.

[1657] Output: The VR headset is started and the system is ready to capture the user's visual information.

[1658] Specific operation: The user puts on the VR headset and turns it on. The headset starts up and completes the initial setup.

[1659] Step 2:

[1660] The server streams the 360-degree panoramic video.

[1661] Input: Video request from the user's device.

[1662] Output: 360-degree panoramic video streamed to the user's device.

[1663] Specific operation: The server sends 360-degree panoramic video data of the tourist attraction to the user's device, and the device plays the video.

[1664] Step 3:

[1665] The device sends the user's gaze information to the server.

[1666] Input: User gaze information captured by sensors in the VR headset.

[1667] Output: Gaze information sent to the server.

[1668] Specific operation: The device acquires the user's gaze information in real time through the VR headset's sensors and sends that information to the server.

[1669] Step 4:

[1670] The server plays the video according to the line of sight.

[1671] Input: Gaze information sent from the device.

[1672] Output: Images are played according to the user's gaze.

[1673] Specific operation: Based on the gaze information received by the server, the server selects the video in the direction the user is looking and sends it to the device, which then plays the video.

[1674] Step 5:

[1675] The device sends the user's facial expressions and voice to the emotion engine.

[1676] Input: User facial and voice data captured by cameras and microphones in the VR headset.

[1677] Output: Facial and voice data sent to the emotion engine.

[1678] Specific operation: The device uses the camera and microphone of the VR headset to capture the user's facial expressions and voice, and sends that data to the emotion engine.

[1679] Step 6:

[1680] The emotion engine analyzes the user's emotions.

[1681] Input: Facial expression and voice data sent from the device.

[1682] Output: Analysis results showing the user's emotional state.

[1683] Specific behavior: The emotion engine analyzes the facial and voice data it receives to determine the user's emotional state (e.g., joy, surprise, sadness).

[1684] Step 7:

[1685] The server dynamically adjusts the video based on the emotion analysis results.

[1686] Input: Sentiment analysis results from the sentiment engine.

[1687] Output: The adjusted video is sent to the user's device.

[1688] Specific operation: The server receives the emotion analysis results, dynamically adjusts the color and brightness of the image according to the user's emotions, and sends the image to the device.

[1689] Step 8:

[1690] The user visits the virtual shop.

[1691] Input: A user request to access the virtual shop.

[1692] Output: The virtual shop interface is displayed on the user's device.

[1693] Specific operation: The user accesses a virtual shop in the VR space, and the shop interface is displayed on the device.

[1694] Step 9:

[1695] A generative AI model recommends products based on user emotions.

[1696] Input: Sentiment analysis results and user history data.

[1697] Output: A list of products recommended to the user.

[1698] Specific operation: The generative AI model selects the most suitable products and generates a recommendation list based on the results of sentiment analysis and the user's purchase and search history.

[1699] Step 10:

[1700] Users communicate with virtual store clerks.

[1701] Input: User voice or text input.

[1702] Output: Response from the virtual store clerk and detailed product information.

[1703] Specific actions: The user sends a question or request to the virtual store clerk, who responds via voice or text and provides detailed product information.

[1704] Step 11:

[1705] The user purchases the product.

[1706] Input: User's purchasing decision and payment information.

[1707] Output: Confirmation of successful purchase and information about the purchased item.

[1708] What happens: The user decides to make a purchase and enters payment information. The device processes the purchase, sends the data to the server, and the purchase is completed.

[1709] (Application example 3)

[1710] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1711] Conventional tourism experience systems using virtual reality (VR) technology do not provide dynamic image changes or product recommendations based on the user's emotions or gaze, limiting the user experience. Furthermore, virtual shops do not provide product recommendations that take the user's emotions into account, making it difficult for users to find the perfect product. This reduces user satisfaction and prevents the virtual experience from being fully appealing.

[1712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1713] In this invention, the server includes: a means for allowing users to experience tourist spots from home using VR technology; a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks; a means for analyzing user emotions in real time and dynamically changing the display of images and products according to those emotions; and a means for the generative AI to recommend products based on the user's emotion data and gaze data. This enables dynamic image changes and product recommendations according to the user's emotion and gaze.

[1714] "VR technology" is a technology for realizing virtual reality, allowing users to have a reality-like experience in a virtual space.

[1715] "Generative AI" is a type of artificial intelligence, a technology that has the ability to generate new information and content based on user data.

[1716] A "virtual shop" is a virtual store that exists on the Internet and is an online platform where users can browse and purchase products.

[1717] An "Internet search tool" is software or an application used to search for information on the Internet.

[1718] "Product purchase and payment" refers to the process by which a user selects a product and pays for it.

[1719] A "virtual store clerk" is a virtual character that communicates with users within the virtual shop, providing product information and assisting with the purchasing process.

[1720] "Analyzing user emotions in real time" is the process of detecting a user's emotional state in real time and analyzing that information.

[1721] "Dynamic change in the display of images and products" means changing the appearance of displayed images and products in real time according to the user's emotions and line of sight.

[1722] "User emotional data" refers to data that indicates the user's emotional state, and is information obtained from brain waves, facial expressions, and the like.

[1723] "Gaze data" is data that indicates the movement of the user's gaze, and is information that detects the direction in which the user is looking.

[1724] "Recommending products" means selecting and presenting the most suitable products to a user based on the user's data.

[1725] The system for implementing the present invention uses the following hardware and software.

[1726] Hardware used

[1727] VR headset: A device that allows users to experience virtual reality spaces.

[1728] Emotion sensor: A device that acquires the user's brainwave data and analyzes their emotions.

[1729] Software used

[1730] Game engine: A platform for developing VR content.

[1731] TensorFlow: A machine learning framework for implementing emotion recognition models.

[1732] OpenAI GPT-4: A tool that implements a generative AI model to recommend the best products to users.

[1733] Data processing and calculation

[1734] 1. Viewing VR content:

[1735] The server uses a game engine to generate a 360-degree panoramic image and transmits it to the VR headset.

[1736] The user's gaze data is acquired from the VR headset, and the image changes dynamically according to the gaze.

[1737] 2. Emotion recognition:

[1738] The user's brainwave data is acquired from the emotion sensor and sent to the server.

[1739] The server uses TensorFlow to implement an emotion recognition model and analyzes user emotions in real time.

[1740] If it senses the user's joy, it will adjust the images and product display accordingly.

[1741] 3. Product Recommendations by Generative AI:

[1742] The server inputs the user's emotional and gaze data as prompts into OpenAI GPT-4.

[1743] A generative AI model recommends the best products for users and displays them in a virtual shop.

[1744] Specific examples

[1745] 1. User puts on VR headset:

[1746] The user puts on a VR headset and enters a virtual store.

[1747] A 360-degree panoramic image is displayed, and the image changes dynamically as the user moves their gaze.

[1748] 2. Emotion recognition and dynamic changes in images:

[1749] The emotion sensor acquires the user's brainwave data and sends it to the server.

[1750] The server analyzes the user's delight using a TensorFlow model.

[1751] If a user is excited about a particular product, it will be displayed more vividly.

[1752] 3. Product Recommendations by Generative AI:

[1753] The server inputs the user's emotion data and gaze data as prompts into OpenAI GPT-4.

[1754] A generative AI model recommends the best products for users and displays them in a virtual shop.

[1755] Prompt Sentence Examples

[1756] User emotion data: Happiness 0.8, Surprise 0.2

[1757] User gaze data: Product A 5 seconds, Product B 3 seconds

[1758] Recommend the best products to your users.

[1759] In this way, users can enjoy a virtual shopping experience from the comfort of their own home.

[1760] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1761] Step 1:

[1762] The user puts on a VR headset and enters a virtual store.

[1763] Input: User actions

[1764] Output: 360-degree panoramic image of a virtual store

[1765] Specific operation: The server sends a 360-degree panoramic image generated using a game engine to the VR headset, allowing the user to enter the virtual space.

[1766] Step 2:

[1767] The user's gaze data is acquired from the VR headset, and the image changes dynamically according to the gaze.

[1768] Input: User gaze data

[1769] Output: Dynamically changing video

[1770] Specific operation: The server acquires gaze data from the VR headset in real time and uses the game engine to change the image according to the gaze.

[1771] Step 3:

[1772] The user's brain wave data is acquired from the emotion sensor and sent to the server.

[1773] Input: User's brainwave data

[1774] Output: EEG data sent to the server

[1775] Specific operation: The emotion sensor acquires the user's brainwave data in real time and transmits it to the server.

[1776] Step 4:

[1777] The server implements an emotion recognition model using TensorFlow to analyze the user's emotions in real time.

[1778] Input: EEG data

[1779] Output: User emotion data

[1780] How it works: The server uses a TensorFlow model to analyze EEG data and detect the user's emotional state in real time.

[1781] Step 5:

[1782] If the system senses the user's joy, it adjusts the images and product display according to that emotion.

[1783] Input: User emotion data

[1784] Output: Adjusted video and product display

[1785] Specific operation: The server uses the game engine based on the emotion data to make the images and products that the user finds pleasing more vivid.

[1786] Step 6:

[1787] The server inputs the user's emotional and gaze data as prompts into OpenAI GPT-4.

[1788] Input: User emotion data, gaze data

[1789] Output: prompts to the generative AI model

[1790] Specific operation: The server converts emotion data and gaze data into text prompts and inputs them into OpenAI GPT-4.

[1791] Step 7:

[1792] A generative AI model recommends the best products to users and displays them in a virtual shop.

[1793] Input: prompt to generative AI model

[1794] Output: Recommended products

[1795] Specific operation: OpenAI GPT-4 generates the most suitable product for the user based on the prompts, and the server displays the product information in a virtual shop.

[1796] Prompt Sentence Examples

[1797] User emotion data: Happiness 0.8, Surprise 0.2

[1798] User gaze data: Product A 5 seconds, Product B 3 seconds

[1799] Recommend the best products to your users.

[1800] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1801] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1802] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1803] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1804] [Third embodiment]

[1805] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1806] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1807] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1808] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1809] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1810] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1811] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1812] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1813] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1814] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1815] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1816] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1817] "Example 1"

[1818] An example of the present invention is a system that allows users to experience tourist spots from their own homes. Specifically, using a VR headset, users can experience tourist spots through 360-degree panoramic images. The system also tracks the user's eye movements and location information, and the images of the tourist spots change dynamically accordingly. This allows users to experience the tourist spots as if they were walking around them from the comfort of their own homes.

[1819] "Example 2"

[1820] This invention also includes a system that uses generative AI to provide internet search tools, product purchases, payments, and communication with virtual store clerks in a virtual shop. Specifically, when a user searches for or purchases products in a virtual shop, the generative AI learns the user's behavior and recommends products that match the user's preferences from the next time. Communication with the virtual store clerk is also possible, allowing the user to ask questions about products and complete the purchase process. This allows the user to recreate the shopping experience of a real store from the comfort of their own home.

[1821] "Example 3"

[1822] As a concrete example, consider a case where a user experiences tourist attractions in Kyoto from the comfort of their own home. The user puts on a VR headset and experiences the sights of Kyoto through a 360-degree panoramic image. As the user moves their gaze, the image changes dynamically, giving the user the experience of walking through the area. The user can also search for Kyoto's famous products in a virtual shop and purchase products recommended by generative AI. By communicating with a virtual store clerk, the user can obtain detailed product information and complete the purchase process.

[1823] The processing flow of each embodiment will be described below.

[1824] "Example 1"

[1825] Step 1: The user puts on the VR headset.

[1826] Step 2: The system provides a 360-degree panoramic image for users to experience the tourist attractions.

[1827] Step 3: The system tracks the user's eye movements and location.

[1828] Step 4: The image of the tourist spot changes dynamically accordingly.

[1829] "Example 2"

[1830] Step 1: User searches for a product in the virtual shop.

[1831] Step 2: The generative AI learns the user's behavior and recommends products that match the user's preferences next time.

[1832] Step 3: The user communicates with the virtual store associate.

[1833] Step 4: The user gets more information about the product and completes the purchase.

[1834] "Example 3"

[1835] Step 1: If a user wants to experience Kyoto's tourist attractions from home, they first put on a VR headset.

[1836] Step 2: Experience Kyoto's tourist attractions through 360-degree panoramic images.

[1837] Step 3: As the user moves their gaze, the image changes dynamically.

[1838] Step 4: The user searches for Kyoto specialties in the virtual shop.

[1839] Step 5: Purchase the product recommended by the generative AI.

[1840] Step 6: Communicate with the virtual store associate to obtain more information about the product and complete the purchase.

[1841] Example 1

[1842] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1843] Traditional tourism experiences required users to actually go to the actual location, which was problematic in terms of time and expense. Even in virtual shops, users could not actually touch and feel the products, which often led to anxiety when making a purchase. Furthermore, there was a lack of systems that dynamically changed the images in response to the user's gaze and movements, preventing a realistic experience.

[1844] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1845] In this invention, the server includes: means for allowing users to experience tourist spots from home using VR technology; means for using generative AI to enable internet search tools, product purchases and payments, and communication with virtual store clerks in a virtual shop; means for tracking the user's eye movements and location information and dynamically changing the scenery of the tourist spot based on the movements; means for acquiring 360-degree panoramic video data of the tourist spot and streaming it to the user's device; and means for displaying the video using a VR headset and rendering the video in real time based on the user's eye gaze data. This allows users to have a realistic tourist experience from the comfort of their own home and purchase products safely in the virtual shop.

[1846] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[1847] "Generative AI" stands for generative artificial intelligence, an artificial intelligence technology that generates new information and content based on user input and data.

[1848] A "virtual shop" is a virtual store operated on the Internet, which is a system that allows users to search for, purchase, and pay for products online.

[1849] An "Internet search tool" is software or an application used to search for information on the Internet, allowing users to efficiently find specific information.

[1850] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, answering their questions and recommending products.

[1851] "Eye movement" refers to the movement of the user's eyes, and is data that indicates the direction in which the user is looking.

[1852] "Location information" refers to data that indicates a user's physical location and is used to identify where the user is located.

[1853] "360-degree panoramic video data" refers to video data that covers all directions, allowing users to look around in any direction.

[1854] "Streaming" is a technology that plays data while transferring it in real time, allowing users to use the data instantly without having to download it.

[1855] A "VR headset" is a device that allows users to experience a virtual reality environment, and when worn, it immerses them in a virtual space.

[1856] "Real-time rendering" refers to the technology by which a computer instantly generates and displays images and graphics, allowing images to be updated without delay in response to the user's movements.

[1857] This invention is a system that allows users to experience tourist spots from the comfort of their own homes, and is realized by combining VR technology and generative AI. Specific embodiments of this system are described below.

[1858] First, the server acquires 360-degree panoramic video data of tourist attractions. This data can be obtained from the Internet or a dedicated database. For example, you can download high-resolution panoramic images of tourist attractions using the Google Street View API.

[1859] The server then streams the captured 360-degree panoramic video data to the user's device using protocols such as HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH).

[1860] The user wears a VR headset at home, and the device is connected to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[1861] The device uses sensors built into the VR headset to track the user's gaze and location. For example, it uses the headset's gyro sensor and accelerometer to detect the user's head movement. The acquired gaze data is analyzed and the image is dynamically updated to match the user's gaze. This allows the user to view different parts of the tourist attraction depending on their gaze.

[1862] Generative AI will enable internet search tools, product purchases and payments, and communication with virtual store staff in virtual shops. It will also have the ability to recommend products based on users' purchase and search history. This allows users to purchase products with confidence within the virtual shop.

[1863] As a concrete example, if a user wants to experience Kinkaku-ji Temple in Kyoto, the following operations take place: The server obtains 360-degree panoramic video data of Kinkaku-ji Temple using the Google Street View API and streams it to the user's device using the HTTP Live Streaming (HLS) protocol. The user puts on a VR headset, and the device displays the image of Kinkaku-ji Temple using a game engine. The device uses the VR headset's gyro sensor to track the user's eye movement; if the user moves their head to the right, the device detects this movement and shifts the image to the right. This allows the user to see different parts of Kinkaku-ji Temple depending on their eye movement, giving the user the experience of walking around the temple in person.

[1864] Example prompts to input to a generative AI model:

[1865] "Please explain how a user can experience Kinkaku-ji Temple from the comfort of their own home using a VR headset. Please include how the image changes dynamically in response to the user's gaze."

[1866] In this way, users can have a realistic tourist experience from the comfort of their own home, and can purchase products safely in virtual shops.

[1867] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1868] Step 1:

[1869] The server acquires 360-degree panoramic video data of tourist spots.

[1870] Specifically, the server downloads high-resolution panoramic images of tourist spots from the Internet or a dedicated database, for example, using the Google Street View API.

[1871] Input: Name and location of tourist spot

[1872] Output: 360-degree panoramic video data

[1873] Step 2:

[1874] The server streams the acquired video data to the user's device.

[1875] Specifically, the server uses protocols such as HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) to send video data to the user's device in real time.

[1876] Input: 360-degree panoramic video data

[1877] Output: Video data streamed to the user's device

[1878] Step 3:

[1879] The user puts on the VR headset and the device displays the image.

[1880] Specifically, the user wears a VR headset at home, and the device connects to the VR headset and displays the streamed video, which is rendered in real time using a game engine.

[1881] Input: Streamed video data

[1882] Output: Image displayed on a VR headset

[1883] Step 4:

[1884] The device tracks the user's eye movements and location.

[1885] Specifically, the device uses the gyro sensor and acceleration sensor built into the VR headset to detect the user's head movements.

[1886] Input: User's eye movements and location information

[1887] Output: Gaze data and position data

[1888] Step 5:

[1889] The device dynamically updates the video based on gaze data.

[1890] Specifically, the device analyzes the acquired gaze data and dynamically updates the image to match the user's gaze. For example, if the user moves their head to the left, the device detects this movement and shifts the image to the left.

[1891] Input: Gaze data and position data

[1892] Output: Dynamically updated video

[1893] Step 6:

[1894] Users experience tourist destinations.

[1895] Specifically, users experience tourist spots through a VR headset. The images change dynamically according to the user's line of sight, giving the user the experience of walking through the tourist spots in real life.

[1896] Input: Dynamically updated video

[1897] Output: User's sightseeing experience

[1898] In this way, users can have a realistic tourist experience from the comfort of their own home.

[1899] (Application example 1)

[1900] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1901] Conventional tourism experience systems and virtual shop systems using virtual reality (VR) technology have faced issues such as a lack of realism and convenience when users experience tourist spots or purchase products from the comfort of their own homes. Furthermore, the lack of interactive experiences utilizing the user's gaze or location information limits the user experience. Furthermore, the display of product information and the purchasing process are cumbersome, requiring improvements to increase user satisfaction.

[1902] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1903] In this invention, the server includes a means for allowing users to experience tourist spots from home using VR technology, a means for using generative AI to enable internet search tools in a virtual shop, product purchases and payments, and communication with virtual store clerks, and a means for tracking the user's gaze and location information and displaying detailed information when the user approaches a product. This allows users to enjoy a realistic tourist experience and virtual shopping from the comfort of their own home. Furthermore, an interactive experience utilizing gaze and location information is provided, and the display of product information and the purchase process are simplified, improving the user experience.

[1904] "VR technology" stands for virtual reality technology, which allows users to immerse themselves in a computer-generated virtual environment and experience it as if it were real.

[1905] "Generative AI" refers to generative artificial intelligence, which has the ability to generate new information and content based on user input and data.

[1906] A "virtual shop" is a virtual store that exists on the Internet, a virtual space where users can browse and purchase products online.

[1907] An "Internet search tool" is software or an application used to search for information on the Internet.

[1908] "Product purchase and payment" refers to the process by which a user selects a product, completes the purchase process, and then pays.

[1909] A "virtual store clerk" is a virtual character or artificial intelligence that communicates with users within a virtual shop, providing product information and support.

[1910] "Tracking gaze and location information" is a technology that detects the direction of the user's gaze and body position in real time, and the system operates based on that information.

[1911] "Show detailed information" means that when a user approaches a specific product, detailed product information such as a description, specifications, and reviews will be displayed on the screen.

[1912] To implement the present invention, the following system configuration and program are required.

[1913] First, the user puts on a VR headset and experiences tourist attractions and virtual shops from the comfort of their own home. The server then uses VR technology to provide the means for the user to experience the tourist attractions from home. This means includes the ability to generate 360-degree panoramic images and dynamically change the images by tracking the user's line of sight and location. Specifically, the images are rendered in real time using a game engine.

[1914] Next, the server uses generative AI to provide internet search tools for the virtual shop, as well as the means to purchase and pay for products and communicate with virtual store staff. Generative AI has the ability to recommend products based on the user's purchase and search history, allowing users to easily find products that suit them. Natural language processing models such as GPT-3 are used for generative AI.

[1915] Furthermore, the server tracks the user's gaze and location information, providing a means to display detailed information about a product when the user approaches it. This means includes gaze tracking technology and location tracking technology. For gaze tracking, hardware such as an eye tracker is used, and for location tracking, sensors built into the VR headset are used. As a result, when the user directs their gaze at a specific product, detailed information about that product is displayed.

[1916] As a concrete example, consider a scene in which a user finds a "smartphone" in a virtual shop and checks its detailed information. The user puts on a VR headset and walks around the virtual shop. When the user directs their gaze towards the smartphone, detailed information about the smartphone (e.g., "The latest model smartphone. Equipped with a high-resolution camera and a long-lasting battery.") is displayed. An example of a prompt sentence to realize this scene is as follows:

[1917] Example prompt sentence:

[1918] The user puts on a VR headset and walks around the virtual shop. When the user looks at the smartphone, detailed information about the smartphone (e.g., "Latest model smartphone. Equipped with a high-resolution camera and long-lasting battery life.") is displayed. Please generate a game engine C script to realize this scene.

[1919] In this way, users can enjoy a realistic sightseeing experience and virtual shopping from the comfort of their own home, and the interactive experience using gaze and location information is provided, simplifying product information display and purchasing procedures, improving the user experience.

[1920] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1921] Step 1:

[1922] The user puts on the VR headset and launches the virtual shop application.

[1923] Input: User actions

[1924] Output: VR headset boots, virtual shop application starts

[1925] Specific operation: The user puts on a VR headset and presses a button to launch the application, which starts the VR headset and launches the virtual shop application.

[1926] Step 2:

[1927] The server tracks the user's gaze and location in real time.

[1928] Input: Gaze and position data from the VR headset

[1929] Output: User's gaze direction and position information

[1930] How it works: The sensors built into the VR headset detect the user's gaze direction and location information and send that data to the server, which processes this data in real time to determine the user's gaze direction and location information.

[1931] Step 3:

[1932] The server identifies the product in the user's line of sight and obtains its detailed information.

[1933] Input: User's gaze direction and position information

[1934] Output: Detailed information about the product in the line of sight

[1935] Specific operation: The server identifies the product that the user is looking at based on the user's gaze direction and location information, and retrieves detailed information about the identified product (e.g., product name, description, price, reviews, etc.) from the database.

[1936] Step 4:

[1937] The server sends the obtained detailed information to the VR headset and displays it to the user.

[1938] Input: Product details

[1939] Output: Product details displayed in a VR headset

[1940] Specific operation: The server sends the acquired product details to the VR headset, which displays the received information in the user's field of view, allowing the user to check the details.

[1941] Step 5:

[1942] The user selects a product and completes the purchase procedure.

[1943] Input: User selection

[1944] Output: Purchase completed

[1945] Specific operation: The user selects a product using their eyes or a controller and presses the purchase button. The server receives the user's selection and begins the purchase process. The server then enters payment information and completes the purchase process.

[1946] Step 6:

[1947] The server updates the purchase history, and the generative AI stores the data for the next product recommendation.

[1948] Input: Purchase history data

[1949] Output: Updated purchase history, saved data

[1950] How it works: The server updates the user's purchase history to a database, and the generative AI saves the data necessary for the next product recommendation, allowing it to recommend products that are appropriate for the user's next shopping experience.

[1951] Example 2

[1952] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1953] Conventional virtual shop systems have struggled to recreate the shopping experience of a real store from the comfort of your own home. They also lacked the functionality to learn user behavior patterns and recommend the best products for each individual user. Furthermore, when communicating with virtual store clerks, it was difficult to provide appropriate answers to user questions. To solve these issues, it is necessary to effectively collect and analyze user behavior data and utilize generative AI models.

[1954] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1955] In this invention, the server includes: means for allowing a user to experience tourist spots from home; means for using a generative AI to enable internet search tools, product purchases and payments in a virtual shop, and communication with a virtual store clerk; means for collecting user behavior data and learning the user's behavior patterns using a generative AI model; means for recommending products based on the user's behavior patterns; means for using the generative AI model to preferentially display highly relevant products when the user searches for a product; and means for using the generative AI model to generate appropriate answers when the user asks a question to the virtual store clerk. This allows users to recreate the shopping experience in an actual store from the comfort of their own home, making it possible to recommend products that are best suited to each user, and providing appropriate answers when communicating with the virtual store clerk.

[1956] "User" means an individual or organization that uses the system to search for products in the virtual shop, purchase products, make payments, and communicate with virtual store staff.

[1957] The "server" is a computer system that collects, stores, and analyzes user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[1958] "Generative AI" is an artificial intelligence technology that learns from user behavior data and recommends products that match the user's preferences, or acts as a virtual store clerk to generate appropriate answers to the user's questions.

[1959] A "virtual shop" is a virtual store on the Internet where users can search for, browse, purchase, and pay for products.

[1960] "Behavioral data" refers to information about the operations performed by users in the virtual shop, such as search history, purchase history, and browsing history.

[1961] "Behavioral patterns" are data that indicate a user's preferences and tendencies, learned by generative AI based on user behavior data.

[1962] "Product recommendation" is a function in which generative AI presents appropriate products to users based on their behavioral patterns.

[1963] A "virtual store clerk" is a virtual store clerk that uses generative AI to generate appropriate answers to users' questions and communicates with them.

[1964] "Internet Search Tool" is a function that allows users to search for products within the virtual shop.

[1965] "Checkout" is the process by which a user makes payment for the products selected in the virtual shop.

[1966] A "prompt sentence" is text data input to a generative AI model, and is an instruction sentence that enables the AI ​​to generate an appropriate response.

[1967] MODE FOR CARRYING OUT THE INVENTION

[1968] This invention is a virtual shop system using generative AI, which aims to recreate the shopping experience of a real store from the comfort of your own home. The system collects user behavior data, learns user behavior patterns using a generative AI model, and generates product recommendations and responses from virtual store clerks.

[1969] Hardware and software used

[1970] Server: A computer system that collects, stores, and analyzes user behavior data and uses generative AI models to learn user behavior patterns. Specifically, it uses a high-performance database server and a GPU server to run the AI ​​model.

[1971] Terminal: A device that allows users to access the virtual shop, search for products, view them, purchase them, make payments, and communicate with virtual store staff. Specifically, this includes PCs, smartphones, tablets, etc.

[1972] Generative AI model: An artificial intelligence technology that learns from user behavior data to recommend products that match the user's preferences and generate appropriate answers to user questions as a virtual store clerk. Specifically, it uses natural language processing models such as GPT-4.

[1973] Specific operation of the system

[1974] 1. Collecting User Behavior Data:

[1975] Users can access the virtual shop to search and browse products.

[1976] The device records the user's operations in real time and transmits them to the server.

[1977] The server stores the received data in a database, specifically recording information such as the user ID, search keywords, viewed product ID, and viewing time.

[1978] 2. Learning user behavior patterns:

[1979] The server periodically extracts user behavior data from the database.

[1980] The server uses a generative AI model (e.g., GPT-4) to analyze user behavior patterns, specifically learning what products users prefer and what time of day they access the site.

[1981] The server stores the learning results as a user profile.

[1982] 3. Perform a product search:

[1983] Users enter the product name or category into the virtual shop's search bar.

[1984] The terminal transmits the input search keyword to the server.

[1985] The server searches the database and generates a list of relevant products, specifically retrieving product IDs, product names, prices, stock status, and other information that matches the search keywords.

[1986] 4. Product recommendation:

[1987] The server prioritizes the display of relevant products based on the user's behavioral patterns. Specifically, it matches the user profile with search results and places products that match the user's preferences at the top of the list.

[1988] The terminal displays the product list received from the server to the user.

[1989] 5. Product Purchase and Payment:

[1990] Users select the products they want to purchase and add them to their cart.

[1991] The terminal transmits the cart information to the server.

[1992] The server manages the cart information and proceeds with the purchase. Specifically, it checks the user's shipping address and payment information and processes the payment.

[1993] The user enters payment information and confirms the purchase.

[1994] The server notifies the user that the payment process is complete.

[1995] 6. Communication with virtual store associates:

[1996] Users can use the chat function within the virtual shop to ask questions to the virtual store clerk.

[1997] The terminal sends the user's question to the server.

[1998] The server uses a generative AI model to generate appropriate answers to user questions. Specifically, it analyzes the content of the question and generates answers from related product information and an FAQ database.

[1999] The terminal displays the response received from the server to the user.

[2000] Examples of specific examples and prompts

[2001] Examples:

[2002] When a user searches for "smartphone," the device sends the keyword "smartphone" to the server.

[2003] The server searches the databa...

Claims

1. A means for users to experience tourist destinations from their homes using VR technology; A means for enabling product search, product purchase, payment, and communication with a virtual shop assistant at the virtual shop in the tourist spot using generative AI; A means for analyzing the emotion of the user from the facial expression and voice of the user using an emotion engine; A means for generating product recommendations based on the search conditions of the product search and the user's emotions using the generation system AI; A means for realizing real-time communication with the virtual store clerk; A system including:

2. The generative AI further includes a means for recommending products based on at least one of the user's purchase history and search history. The system of claim 1 .

3. The VR technology includes a means for tracking the movement of the user and the movement of the user's gaze, and dynamically changing the scenery of the tourist spot based on the tracked movement of the user and the movement of the user's gaze; and a means for analyzing the user's emotions in real time and dynamically changing the display of images and products in response to the analyzed emotions. The system of claim 1 .

Images