System

Abstract

To provide a system which allows a user to realize a wish of the user in a virtual space.SOLUTION: A system includes: means for inputting a wish that a user wants to realize; means for extracting necessary information for realizing the wish of the user by analyzing the input wish of the user; means for generating a prompt sentence which instructs generation of the extracted information; means for generating necessary information for realizing the wish of the user by using the generated prompt sentence and the generated AI model; and means for reproducing the wish of the user by using the generated information on the virtual space and providing the user with the reproduced virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] People have the desire to realize the things they want to remember at the end of their lives. Specifically, they want to talk to their grandchildren, go back to their hometown, see a beautiful ocean, etc. However, in the real world, it is difficult to make all of these desires come true. [Means for solving the problem]

[0005] The present invention provides a system that works in conjunction with the metaverse to allow users to realize their regrets in a virtual world. Specifically, the system includes a means for generating a person to talk to in front of the user, a means for providing the user with the feeling of returning to their hometown, a means for allowing the user to go to a place that the user imagines, and a means for allowing the user to feel the sensation of water. This allows the user to realize their wishes in the virtual world. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[0029] "Example 2"

[0030] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The user starts the system and inputs their desires. For example, they can input their desires, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean," in text or voice.

[0034] Step 2: The system analyzes the input desire and generates a corresponding virtual experience. For example, if the desire is "I want to talk to my grandchild," the system will generate an avatar of the user's grandchild and enable real-time conversation.

[0035] Step 3: The system provides the generated virtual experience to the user. The user can experience the virtual experience through a VR or AR device. For example, if the scenery of the user's hometown is reproduced, the user can feel as if they are there. Step 4: The system monitors the user's response and adjusts the virtual experience as necessary. For example, if the user responds that they would like to feel the touch of water while looking at a seascape, the system utilizes a means of providing haptic feedback to provide the user with the touch of water.

[0036] Example 1

[0037] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0038] In modern society, many users face the challenge of finding it difficult to realize their regrets due to physical and time constraints. For example, it is difficult to talk to far-away family members or experience the feeling of returning to one's hometown. It is also difficult for users to go to places they imagine or experience real sensations through touch. To solve these problems, a means is needed to realize users' wishes using the virtual world.

[0039] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0040] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for analyzing the user's wishes, a means for generating a virtual experience, and a means for setting haptic feedback. This allows the user to go beyond physical constraints and experience their wishes in a realistic way in the virtual world.

[0041] The "metaverse" is a virtual world built on the Internet, a virtual environment in which users can interact with other users through avatars and engage in various activities.

[0042] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in the virtual world the desires or experiences that they were unable to realize in the real world.

[0043] "Means for generating a person with whom the user can converse in front of the user" refers to a technology or method for generating an avatar of the person with whom the user wants to converse in the virtual world and simulating a conversation in real time.

[0044] "Means of giving the user the feeling of returning to their hometown" refers to technologies or methods that recreate the scenery or environment of the user's hometown in a virtual world, giving the user the feeling of being in that place.

[0045] "A means to go to the place the user imagines" refers to the technology or method that recreates the place the user imagines he or she would like to go to in the virtual world and allows the user to access that place.

[0046] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when interacting with water in the virtual world.

[0047] "Means for analyzing user wishes" refers to techniques or methods for analyzing the wishes entered by the user, understanding their content, and extracting the necessary information.

[0048] A "means for generating a virtual experience" is a technology or method for generating an experience in a virtual world based on a user's desires.

[0049] "Means for configuring haptic feedback" means techniques or methods for configuring and controlling a haptic feedback device to make a user's experience in a virtual world more realistic.

[0050] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] The present invention is a system for enabling a user to realize his / her own desires in a virtual world. A specific embodiment of this system will be described below.

[0052] System Configuration

[0053] This system consists of three main elements: a server, a terminal, and a user. The server works in conjunction with the metaverse to analyze users' desires and generate virtual experiences. The terminal provides an interface for users to access the system and enjoy the virtual experience. Users input their desires through the terminal and experience the virtual experience.

[0054] Hardware and Software Used

[0055] Server: A server with high-performance computing power is required. Specifically, a server with the computing resources to run a generative AI model (e.g., GPT-4 (registered trademark) by OpenAI (registered trademark)) is recommended.

[0056] Terminal: The device through which the user accesses the device, which may be a PC, tablet, smartphone, etc. Also includes haptic feedback devices (e.g. HaptX gloves).

[0057] Software: Game engines (e.g. Unity, Unreal Engine) are used to build virtual worlds, and generative AI models are used to perform natural language processing.

[0058] Program Processing

[0059] The server receives the user's desire and analyzes it. It uses a generative AI model to understand the user's input and extract the necessary information. The server then generates the virtual experience. Specifically, it generates an avatar of the person the user wants to talk to and simulates a real-time conversation. It also uses a game engine to build a realistic 3D environment to recreate the user's hometown or the scenery they want to see. It also controls a haptic feedback device and configures it to make the user's experience in the virtual world feel more realistic.

[0060] Examples

[0061] If a user wishes to have a conversation with their grandchild, the specific steps that are taken are as follows:

[0062] 1. User's wish input: The user inputs "I want to talk to my grandson. His grandson's name is Taro, and he is 10 years old" into the input field of the terminal.

[0063] 2. Wish analysis: The server uses a generative AI model to analyze the wish “I want to talk to my grandchild” and extract details such as the grandchild’s name and age.

[0064] 3. Generating the virtual experience: The server uses a generative AI model to generate an avatar of the user's 10-year-old grandson, Taro, and simulates a real-time conversation with him. It also uses Unity to create a 3D recreation of the user's hometown.

[0065] 4. Setting up haptic feedback: The server controls the HaptX gloves and configures them so that the user can feel the sensation of water in the virtual world.

[0066] 5. Providing the virtual experience: The server provides the generated virtual experience to the user. The user can interact with the grandchild’s avatar through the terminal display and feel the sensation of water through the HaptX gloves.

[0067] Examples of prompt statements

[0068] "I want to have a conversation with my grandson. His name is Taro and he is 10 years old."

[0069] "I want to see the scenery of my hometown. It's a small village in the north."

[0070] "I want to see a beautiful ocean. Preferably one like the ocean in the south."

[0071] This allows users to transcend physical constraints and experience their desires in a realistic way in the virtual world.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1:

[0074] The user uses the terminal to input his / her wish. For example, he / she inputs, "I want to talk to my grandson. His / her name is Taro, and he / she is 10 years old." This input is sent to the server.

[0075] Step 2:

[0076] The server analyzes the wish received from the user. The input text data is passed to a generative AI model, which understands the user's wish and extracts the information necessary to realize the user's wish. Specifically, it analyzes the wish "I want to talk to my grandchild" and the detailed information "My grandchild's name is Taro and he is 10 years old." The analysis result is passed to the next processing step.

[0077] Step 3:

[0078] The server generates the virtual experience based on the analysis results. First, it uses a generative AI model to generate an avatar of the person the user wants to talk to. Specifically, it generates an avatar of the user's 10-year-old grandson, "Taro," and simulates a real-time conversation. The data for this avatar generation is passed to the next processing step.

[0079] Step 4:

[0080] The server uses a game engine (e.g. Unity) to create a realistic 3D environment to recreate the user's hometown or scenery they want to see. For example, if a user inputs "I want to see the scenery of my hometown," a 3D model is generated based on the scenery data of that place. This 3D environment data is passed to the next processing step.

[0081] Step 5:

[0082] The server controls the haptic feedback device (e.g., HaptX gloves) and configures it to make the user's experience in the virtual world feel more realistic. For example, if the user inputs "I want to see a beautiful ocean," the server configures the haptic feedback device to reproduce the feeling of water. This configuration data is passed to the next processing step.

[0083] Step 6:

[0084] The server provides the generated virtual experience to the user. The user interacts with the generated avatar through the display of the terminal and experiences the 3D environment. The user can also feel the touch of water through a haptic feedback device. The data of this virtual experience is sent to the user, and the user can enjoy the experience in the virtual world.

[0085] (Application example 1)

[0086] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] In modern society, users need to have realistic experiences in virtual spaces without having to physically move around. However, current systems have difficulty providing sufficient tactile and visual reality when users try on or use products in virtual spaces. In addition, there is a lack of real-time interaction and haptic feedback when users virtually interact with specific places or people. This limits the user's experience and reduces satisfaction.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0089] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to try on or use products in a virtual store, a means for generating an avatar for the user, a means for generating a scene of the virtual store, and a means for providing haptic feedback. This allows the user to have a real experience in the virtual space.

[0090] The "metaverse" is an interactive digital environment built in a virtual space where users can interact with other users and content through avatars.

[0091] "A means to virtually realize the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, allowing the user to experience them.

[0092] "A means for generating a person with whom the user can converse in front of the user" refers to technology that generates an avatar or character with which the user can converse in real time in a virtual space.

[0093] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[0094] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users in a virtual space and provides the user with the experience of visiting those places.

[0095] "Means allowing the user to feel the sensation of water" refers to technology that allows the user to sense the presence of water in a virtual space through haptic feedback.

[0096] "Means for users to try on or use products in a virtual store" refers to technology that provides users with the experience of trying on or using products in a store within a virtual space.

[0097] "Means for generating a user's avatar" refers to technology for generating an avatar for use in a virtual space based on the user's appearance and characteristics.

[0098] "Means for generating a virtual store scene" refers to technology for reproducing the environment and scenery of a store within a virtual space.

[0099] A "means for providing haptic feedback" is a haptic device or technology that enables a user to physically feel their virtual experience.

[0100] In order to implement this invention, the following hardware and software are used. The hardware requires a smartphone and a haptic feedback device. The software requires the OpenAI API and Virtual Reality SDK.

[0101] The server provides a means to link with the metaverse and uses generative AI models to generate data to virtually realize the user's wishes. For example, if a user wants to talk to a specific person, an avatar of that person can be generated and the user can talk to them in real time.

[0102] The terminal provides an interface for the user to try on or try on products in the virtual store. The user can access the virtual store using a smartphone, select products, and try them on or try them on. For example, if the user wants to try on clothes, the terminal can generate an avatar of the user and dress the avatar in the selected clothes. In addition, by using a haptic feedback device, the user can actually feel the texture of the clothes.

[0103] If a user wants to go to a specific place, the server generates a scene of that place and gives the user the feeling that he or she is there. For example, if a user wants to see the sea, the server generates a scene of the sea that the user imagines and allows the user to go to that place. In this case, a haptic feedback device can be used to provide the feeling of water.

[0104] As an example, the following prompt can be used:

[0105] Avatar generation: "Generate an avatar for the user with the following data: name, height, weight"

[0106] Generate a virtual store with the following data: store name, location

[0107] Try on product: "Apply the following product to the avatar: product type, color, size on user data"

[0108] This allows users to have a realistic experience in a virtual space.

[0109] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0110] Step 1:

[0111] The server receives a user request. As input, it contains the user's desired experience (e.g., talking to a specific person, visiting a specific place). The server analyzes this request and sends the necessary data to the generative AI model. As output, it gets the analysis result.

[0112] Step 2:

[0113] The server uses a generative AI model to generate avatars and scenery based on the user's request. The input includes the user's feature data and details of the desired experience. The generative AI model processes these data and generates avatar and scenery data. The output is the generated avatar and scenery data.

[0114] Step 3:

[0115] The server sends the generated avatar and scenery data to the terminal. The input includes the generated avatar and scenery data. The server converts these data into an appropriate format and sends it to the terminal. The output is the data sent to the terminal.

[0116] Step 4:

[0117] The terminal constructs a virtual space based on the received data. The input includes avatar and scenery data sent from the server. The terminal renders this data and displays it to the user. The output is the virtual space displayed to the user.

[0118] Step 5:

[0119] The user begins to experience the virtual space through the device. The input includes the virtual space displayed on the device. The user manipulates an avatar to talk to specific people or visit specific places. The output is the result of the user's manipulation.

[0120] Step 6:

[0121] The terminal transmits the user's operation to the server. The input includes the user's operation data. The terminal transmits these data to the server to realize real-time interaction. The output is the operation data transmitted to the server.

[0122] Step 7:

[0123] The server updates the virtual space based on the user's operation data. The input includes the user's operation data. The server analyzes these data and updates the state of the virtual space. The output is the updated data of the virtual space.

[0124] Step 8:

[0125] The server sends the updated virtual space data to the terminal. As input, the updated virtual space data is included. The server converts these data into an appropriate format and sends them to the terminal. As output, the updated data sent to the terminal is obtained.

[0126] Step 9:

[0127] The terminal displays the updated virtual space to the user. As input, it contains the updated data sent from the server. The terminal renders these data and displays them to the user. As output, it obtains the updated virtual space displayed to the user.

[0128] Step 10:

[0129] The user physically feels the experience in the virtual space using the haptic feedback device. The input includes haptic feedback data sent from the terminal, which allows the user to feel the experience in the virtual space more realistically. The output is the user's haptic experience.

[0130] Example 2

[0131] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In modern society, many users want to recreate their past memories and regrets in virtual space. However, current systems have difficulty realizing users' specific desires in real time, and are unable to provide sensory experiences such as haptic feedback. For this reason, there is a demand for systems that allow users to experience virtual space in a more realistic way.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0134] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the user's regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to their hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for receiving the user's input and generating a prompt sentence using a generative AI model, a means for generating a virtual space using virtual space generation software, and a means for providing haptic feedback using a haptic feedback device. This allows the user to realize their specific desires in the virtual space in real time and further obtain a more realistic experience through haptic feedback.

[0135] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[0136] "A means of virtually realizing the things that users regret" refers to technology that enables users to recreate in a virtual space things or experiences from the past that they feel nostalgic about.

[0137] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character with which the user can converse in a virtual space.

[0138] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0139] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users within a virtual space, allowing the users to visit those places.

[0140] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[0141] "Means for receiving user input and generating a prompt sentence using a generative AI model" refers to technology that analyzes information entered by the user and generates an appropriate prompt sentence using a generative AI model.

[0142] "Means for generating a virtual space using virtual space generation software" refers to a technology that uses software for constructing a virtual space to generate a virtual space in response to a user's request.

[0143] "Means for providing haptic feedback using a haptic feedback device" refers to technology that uses a device to actually reproduce the sensation when a user touches a specific object or environment in a virtual space.

[0144] The present invention is a system that allows users to realize their wishes in a virtual space. This system works in conjunction with the metaverse and provides multiple means for users to virtually realize their wishes.

[0145] Hardware and Software Used

[0146] Hardware: Terminals (PC, smartphone, VR device), servers, haptic feedback devices (e.g. HaptX gloves)

[0147] Software: Generative AI models (e.g. GPT-4), virtual world generation software (e.g. Unity, Unreal Engine)

[0148] System Configuration

[0149] 1. User input of wishes

[0150] The user uses the terminal to input their own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0151] 2. Data collection and analysis

[0152] The server receives user input and uses a generative AI model to generate prompts. For example, if a user wants to talk to their grandchild, it generates an avatar of the grandchild and generates prompts that enable real-time conversation.

[0153] 3. Creation of Virtual Space

[0154] The server uses virtual space generation software to generate a virtual space based on the user's wishes, such as "recreating the scenery of the user's hometown" or "generating the seascape that the user imagines."

[0155] 4. Avatar Creation and Conversation

[0156] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[0157] 5. Providing haptic feedback

[0158] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[0159] Examples

[0160] Example 1: I want to talk to my grandchildren

[0161] The user types into the terminal, "I want to talk to my grandchild."

[0162] The server uses the generative AI model to generate "prompts that generate an avatar of the user's grandchild and enable real-time conversation."

[0163] The server uses virtual space generation software to generate an avatar of the grandchild and conduct the conversation. The grandchild's avatar says, "Hello, Grandpa! What would you like to talk about today?"

[0164] Example 2: I want to go back to my hometown

[0165] The user types "I want to go back to my hometown" into the terminal.

[0166] The server uses a generative AI model to generate "prompt sentences that recreate the scenery of the user's hometown."

[0167] The server uses virtual space generation software to recreate the scenery of the person's hometown, allowing the user to walk around the scenery in the virtual space.

[0168] Example 3: I want to see a beautiful ocean

[0169] The user types "I want to see a beautiful ocean" into the terminal.

[0170] The server uses a generative AI model to generate "prompt sentences that generate the seascape the user imagines and provide haptic feedback."

[0171] The server uses virtual space generation software to generate the ocean scenery and a haptic feedback device to provide the feel of water: when a user places their hand in the virtual ocean, the HaptX gloves make them feel the cold and flowing water.

[0172] The system allows users to realize their specific desires in a virtual space in real time, and provides a more realistic experience through haptic feedback.

[0173] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0174] Step 1:

[0175] User's wish input

[0176] The user uses the terminal to input his / her own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0177] Input: User's wish (in text format)

[0178] Output: User wish data

[0179] Specific actions: The user inputs their wish in text form using the device's keyboard or voice input function. For example, the user inputs "I want to talk to my grandchildren" into the input form displayed on the device screen.

[0180] Step 2:

[0181] Data collection and analysis

[0182] The server receives the user's input and generates a prompt using a generative AI model.

[0183] Input: User's wish data

[0184] Output: Prompt statement

[0185] Specific operation: The server analyzes the user's input data and instructs the generative AI model to generate "a prompt sentence that generates an avatar of the grandchild and enables real-time conversation if the user wants to talk to the grandchild."

[0186] Step 3:

[0187] Creating a Virtual Space

[0188] The server uses virtual space generation software to generate a virtual space based on the user's wishes.

[0189] Input: prompt statement

[0190] Output: Virtual space data

[0191] Specific operation: The server issues instructions to the virtual space generation software, such as "recreate the scenery of the user's hometown" or "generate the seascape that the user imagines," and constructs the virtual space.

[0192] Step 4:

[0193] Avatar creation and conversation

[0194] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[0195] Input: Prompt text, virtual space data

[0196] Output: Avatar, conversation data

[0197] How it works: The server instructs the generative AI model to generate an avatar of the user's grandchild, and then applies a dialogue model to enable real-time conversation. For example, the grandchild's avatar might say, "Hello, Grandpa! What would you like to talk about today?"

[0198] Step 5:

[0199] Providing haptic feedback

[0200] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[0201] Input: Virtual space data, user operation data

[0202] Output: Haptic feedback data

[0203] Specific operation: The server instructs the haptic feedback device to reproduce the feel of water when the user touches the ocean in the virtual space. For example, when the user puts his or her hand in the ocean in the virtual space, the haptic feedback device makes the user feel the coldness and flow of the water.

[0204] Through these steps, users can realize their specific desires in the virtual space in real time, and get a more realistic experience through haptic feedback.

[0205] (Application example 2)

[0206] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0207] In modern society, users are required to have realistic experiences in virtual spaces without having to physically move. However, conventional systems have been unable to fully satisfy users' desires, and in particular, in the shopping experience in virtual stores, it was difficult to have a real-time conversation with an assistant or feel the texture of the products. This resulted in low user satisfaction and made the experience in virtual spaces inferior to reality.

[0208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0209] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to select a product while talking to an assistant avatar in real time when enjoying shopping in a virtual store, a means for the user to walk around in a virtual store that reproduces a specific place or scenery, and a means for the user to feel the texture of the product through haptic feedback. This allows the user to enjoy a real shopping experience in a virtual space.

[0210] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[0211] "A means of virtually realizing the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, making the user feel as if they are actually experiencing them.

[0212] "A means for generating a person with whom a user can converse in front of the user" refers to a technology that enables a user to generate an avatar of a person with whom the user wants to converse in a virtual space, thereby enabling the user to converse in real time.

[0213] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[0214] "A means to go to places that users imagine" is a technology that generates the places that users imagine they would like to go to in a virtual space and provides the user with the experience of visiting those places.

[0215] "A means for allowing the user to feel the sensation of water" refers to technology that recreates the presence of water in a virtual space and allows the user to feel the sensation of that water through haptic feedback.

[0216] "A means that enables users to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store" refers to technology that enables users to select products while conversing in real time with an assistant avatar when shopping in a virtual store.

[0217] "A means for enabling users to walk around a virtual store that recreates a specific place or scenery" refers to technology that enables users to move freely within a virtual store and experience an environment that recreates a specific place or scenery.

[0218] "Means for allowing a user to feel the texture of a product through haptic feedback" refers to technology that allows a user to feel the texture of a product through a haptic feedback device when picking up the product in a virtual space.

[0219] A system for implementing this invention includes means for linking with the metaverse, means for virtually realizing a user's regrets, means for generating a person for the user to converse with in front of them, means for giving the user the feeling of returning to their hometown, means for allowing the user to go to places the user imagines, means for the user to feel the feel of water, means for a user to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store, means for a user to walk around a virtual store that recreates a specific place or scenery, and means for the user to feel the texture of products through haptic feedback.

[0220] System Program Overview

[0221] The server uses the OpenAI API, the virtual environment library (virtual_environment), and the haptic feedback library (haptic_feedback) to process and calculate data to realize the user's wishes. Specifically, the server generates prompts based on the user's wishes, generates avatars and virtual environments using generative AI models, and provides the user with a realistic sensation through a haptic feedback device.

[0222] Processing flow

[0223] 1. Generate prompts based on user desires

[0224] The server describes the user's wishes in natural language and inputs them into the generative AI model. For example, if the user wishes to "talk to their grandchildren," the server generates a prompt sentence like this:

[0225] The user wants to experience conversation with his / her grandchild.

[0226] 2. Avatar Creation

[0227] The server uses the OpenAI API to generate an avatar based on the user's wishes, and the generated avatar is then used by the user to converse in real time in a virtual space.

[0228] 3. Setting up the virtual environment

[0229] The server uses the virtual environment library to set up a virtual store for the user to experience, and the user can freely walk around the virtual store.

[0230] 4. Haptic feedback settings

[0231] The server uses a haptic feedback library to allow the user to feel the texture of the product, so that when the user picks up the product in the virtual space, they can feel the texture of the product in a realistic way.

[0232] Examples

[0233] For example, if a user has a wish to "talk to his / her grandchild," the server will generate an avatar of the grandchild and enable the user to talk to the grandchild in real time in the virtual store. In addition, when the user picks up a particular product, the user can feel the texture of the product through a haptic feedback device.

[0234] In this way, users can enjoy a real shopping experience in a virtual space.

[0235] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0236] Step 1:

[0237] The server generates a prompt based on the user's desire. If the user has the desire "I want to talk to my grandchildren," the server takes this information as input and generates a prompt like this:

[0238] The user wants to experience conversation with his / her grandchild.

[0239] This prompt is then fed into a generative AI model.

[0240] Step 2:

[0241] The server uses the generative AI model to generate an avatar based on the prompt sentence. Specifically, it calls the OpenAI API and provides the prompt sentence as input. The API analyzes the prompt sentence and generates an avatar of the user's grandchild. This avatar is used by the user to converse in real time in the virtual space. The generated avatar is sent from the server to the user's device.

[0242] Step 3:

[0243] The server uses the virtual environment library to set up a virtual store for the user to experience. If the user has a desire to "enjoy shopping in a virtual store," the server receives this information as input and calls the virtual environment library to generate a virtual store. The generated virtual store is sent to the user's terminal, and the user can freely walk around the virtual store.

[0244] Step 4:

[0245] The server uses the haptic feedback library to allow the user to feel the texture of the product. When the user picks up a particular product, the information is sent to the server. The server calls the haptic feedback library to generate data about the texture of the product. This data is sent to the user's haptic feedback device, allowing the user to feel the texture of the product in a realistic way.

[0246] Step 5:

[0247] Users can walk around the virtual store and select products while talking to the generated avatar in real time. When the user selects a product, the information is sent to the server, and the server sends detailed information about the selected product to the user's device. Based on this information, the user can decide whether to select more products or purchase them.

[0248] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0249] "Example 1"

[0250] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[0251] "Example 2"

[0252] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[0253] The process flow of each embodiment will be described below.

[0254] "Example 1"

[0255] Step 1: The user connects to the system through a VR or AR device.

[0256] Step 2: The user inputs their desired virtual experience into the system.

[0257] Step 3: The system uses an emotion engine to analyze the user's emotions based on facial expressions, tone of voice, choice of words, etc.

[0258] Step 4: The system tailors the virtual experience based on the analysis results and provides it to the user.

[0259] Step 5: The user experiences the provided virtual experience.

[0260] Step 6: The system monitors the user's reactions and adjusts the virtual experience if necessary.

[0261] Example 1

[0262] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0263] In modern society, there is a demand for a way to allow users to realize their regrets in the virtual world. In particular, it is important to realize experiences that are difficult to experience in the real world, such as talking with far-away family members, returning to one's hometown, and experiencing natural environments. There is also a demand to provide a more realistic and satisfying experience by analyzing the user's emotions and adjusting the virtual experience according to those emotions.

[0264] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place imagined by the user, a means for the user to feel the touch of water, a means for analyzing the user's emotions, a means for adjusting the virtual experience based on the analysis result, and a means for generating a virtual experience using a generation AI model. This allows the user to realize his / her desires in the virtual world and enjoy the optimal virtual experience according to his / her emotions.

[0265] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[0266] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in a virtual space the desires or experiences that they were unable to realize in the real world.

[0267] "Means for generating a person with whom the user can converse in front of the user" refers to a technique or method for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[0268] "Means of giving users the feeling of returning to their hometown" refers to technologies and methods that allow users to recreate the scenery and environment of their hometown in a virtual space, giving them the feeling as if they were there.

[0269] "Means that allow users to go to places they imagine" refers to technologies or methods that allow users to recreate specific locations or environments in a virtual space, providing the experience of being in that place.

[0270] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when touching water in a virtual space.

[0271] "Means of analyzing user emotions" refers to techniques or methods for analyzing emotions from a user's facial expressions, tone of voice, choice of words, etc.

[0272] "Means for adjusting the virtual experience based on the analysis results" refers to technologies and methods for optimizing the experience in the virtual space based on the results of an analysis of the user's emotions, and providing an experience that corresponds to the user's emotions.

[0273] A "means for generating a virtual experience using a generative AI model" is a technology or method that uses an artificial intelligence model to generate experiential scenarios or content in a virtual space based on user input data.

[0274] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0275] This invention is a system for realizing a user's regrets in a virtual world, and provides a technology that works in conjunction with the metaverse to analyze the user's emotions and optimize the virtual experience. A specific embodiment of this system will be described below.

[0276] System Configuration

[0277] The system consists of the following main components:

[0278] 1. User device:

[0279] A device that allows a user to access the system, including PCs, smartphones, tablets, VR devices, AR devices, etc.

[0280] Users access the system through a dedicated application or web interface and enter their wishes.

[0281] 2. Server:

[0282] It takes user input data and uses a generative AI model to generate a virtual experience.

[0283] An emotion engine is used to analyze the user's emotions and tailor the virtual experience based on the analysis results.

[0284] 3. Generative AI model:

[0285] As an example, use a natural language processing model such as OpenAI's GPT-4.

[0286] Generates virtual experience scenarios and content based on user input data.

[0287] 4. Emotion Engine:

[0288] It includes algorithms that analyze emotions from users' facial expressions, tone of voice, choice of words, etc.

[0289] The content of the virtual experience is adjusted based on the analysis results.

[0290] System Operation

[0291] 1. User inputs their wishes:

[0292] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0293] 2. Data analysis and generation:

[0294] The server receives the user's input data and generates the virtual experience using a generative AI model.

[0295] For example, if a user types, "I want to talk to my grandchild," the server will generate an avatar of the user's grandchild and enable a real-time conversation.

[0296] 3. Sentiment analysis using emotion engine:

[0297] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[0298] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[0299] The content of the virtual experience is adjusted based on the analysis results.

[0300] 4. Providing Virtual Experiences:

[0301] The server streams the generated virtual experience data to the user's device.

[0302] Users put on a VR headset or AR glasses and begin the virtual experience.

[0303] For example, a user could put on a VR headset and enjoy a real-time conversation with an avatar of their grandchild.

[0304] Examples and prompts

[0305] As a concrete example, a process will be described in which the user has a desire to "talk to his / her grandchild."

[0306] 1. User input:

[0307] The user types into the terminal, "I want to talk to my grandchild."

[0308] 2. Data analysis and generation:

[0309] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an avatar of the user's grandchild.

[0310] 3. How the Emotion Engine works:

[0311] The emotion engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling joy.

[0312] 4. Providing Virtual Experiences:

[0313] The server then transmits the real-time conversation between the user and the avatar of the grandchild to the user's device, and the user experiences the conversation with the grandchild through a VR device.

[0314] Example prompt:

[0315] If a user types, "I want to talk to my grandchild," generate a program that creates an avatar of the user's grandchild and enables a real-time conversation.

[0316] In this way, a system is constructed that realizes users' wishes in the virtual world.

[0317] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0318] System program processing flow

[0319] Step 1: User inputs their wishes

[0320] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0321] Input: The user's desire (e.g., "I want to talk to my grandchildren")

[0322] Output: User wish data

[0323] Specific behavior:

[0324] Users enter their wishes into a text box using a dedicated application or a web interface.

[0325] For example, type "I want to talk to my grandchildren."

[0326] Step 2: Analyze and generate data

[0327] The server receives the user's input data and generates the virtual experience using a generative AI model.

[0328] Input: User's wish data

[0329] Output: Virtual experience scenario

[0330] Specific behavior:

[0331] The server sends the user's input to a natural language processing engine, which analyzes the input.

[0332] Based on the analysis results, a generative AI model (e.g., OpenAI's GPT-4) is invoked to generate a virtual experience scenario.

[0333] Example: In response to the input "I want to talk to my grandchild," an avatar of the grandchild is generated and a conversation scenario is created.

[0334] Step 3: Sentiment analysis using the emotion engine

[0335] The server analyzes the user's emotions using an emotion engine, which analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0336] Input: User's camera video and audio data

[0337] Output: Emotion analysis results

[0338] Specific behavior:

[0339] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[0340] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[0341] The content of the virtual experience is adjusted based on the analysis results.

[0342] Example: If the user is smiling, the system will provide conversational content that amplifies the joy.

[0343] Step 4: Create the virtual experience

[0344] The server generates an optimized virtual experience based on the results of the emotion analysis, which is tailored to the user's desires and emotions.

[0345] Input: Sentiment analysis results, virtual experience scenario

[0346] Output: Optimized virtual experience data

[0347] Specific behavior:

[0348] The server integrates the results of the generative AI model and the emotion engine to create an optimal virtual experience scenario.

[0349] Example: If a user types "I want to talk to my grandchild," and sentiment analysis indicates joy, the grandchild's avatar can be set to offer happy topics.

[0350] Step 5: Offer virtual experiences

[0351] The server transmits the generated virtual experience to the user's device, where the user experiences the virtual experience through a VR or AR device.

[0352] Input: Optimized virtual experience data

[0353] Output: The user's virtual experience

[0354] Specific behavior:

[0355] The server streams the generated virtual experience data to the user's device.

[0356] Users put on a VR headset or AR glasses and begin the virtual experience.

[0357] Example: A user puts on a VR headset and enjoys a real-time conversation with an avatar of their grandchild.

[0358] In this way, a system is constructed that realizes users' wishes in the virtual world.

[0359] (Application example 1)

[0360] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] In modern society, users are expected to realize their desires in virtual space without moving physically. In addition, it is a challenge to optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience. In particular, it is important to adjust the shopping experience based on the user's emotions in virtual stores.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0363] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the experience of the virtual store based on the user's emotions. This allows the user to realize his / her desires in the virtual space without physical constraints and enjoy the optimal experience according to his / her emotions.

[0364] "Means of linking with the metaverse" refers to technology that exchanges data with the metaverse, a virtual space, and provides an interface that allows users to have virtual experiences.

[0365] "A means to virtually realize the user's regrets" is a technology that enables users to recreate in a virtual space the desires and experiences that they were unable to realize in the real world.

[0366] "Means for generating a person with whom the user can converse in front of the user" refers to a technology for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[0367] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0368] "A means to go to places the user imagines" refers to technology that generates specific places or landscapes imagined by the user in a virtual space, allowing the user to access those places.

[0369] "A means for allowing the user to feel the sensation of water" refers to haptic technology that provides the user with feedback of the actual sensation of water when they touch water in a virtual space.

[0370] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and dynamically changes the content of the virtual experience based on the results.

[0371] The "means for optimizing the virtual store experience based on the user's emotions" is a technology that analyzes the user's emotional state and adjusts the shopping experience in the virtual store according to that state.

[0372] The system for implementing the present invention aims to enable a user to realize his / her desires in a virtual space and enjoy an optimal experience according to his / her emotions. A specific embodiment of the system will be described below.

[0373] System Configuration

[0374] The system consists of the following main components:

[0375] 1. Metaverse Linkage Module

[0376] It exchanges data with the metaverse and provides an interface for users to have virtual experiences. Specifically, it works with the metaverse platform through APIs.

[0377] 2. Virtual experience generation module

[0378] This is a module that allows users to virtually realize their regrets. For example, if a user wants to talk to their grandchild, it will generate an avatar of the grandchild and enable a real-time conversation.

[0379] 3. Emotion Recognition Module

[0380] This is a module for analyzing emotions from the user's facial expressions, tone of voice, etc. Specifically, it uses libraries such as OpenCV and EmotionRecognizer.

[0381] 4. Virtual Experience Tuning Module

[0382] This module dynamically changes the content of the virtual experience based on the emotion data obtained from the emotion recognition module. For example, if the user is feeling joy, it will provide an experience that amplifies that joy.

[0383] 5. Virtual store optimization module

[0384] This module tailors the virtual store shopping experience based on the user's emotions, for example highlighting new or limited edition products if the user is excited.

[0385] Hardware and Software Used

[0386] Hardware: Webcam, Head Mounted Display

[0387] Software: OpenCV (image processing library), EmotionRecognizer (emotion recognition library), VirtualStore (virtual store management library)

[0388] Processing flow

[0389] 1. User emotion recognition

[0390] It uses a webcam to capture the user's face and analyzes emotions using the EmotionRecognizer library.

[0391] 2. Creating and adjusting virtual experiences

[0392] Based on the emotion data obtained from the emotion recognition module, the virtual experience generation module realizes the user's desire in the virtual space. Furthermore, the virtual experience adjustment module dynamically changes the experience content according to the emotion.

[0393] 3. Virtual store optimization

[0394] The virtual store optimization module tailors the shopping experience based on user emotions, for example highlighting new or limited edition products if the user is excited.

[0395] Examples

[0396] When a user is shopping in a virtual store, if the system detects "excitement" in the user's facial expression, it will highlight new or limited edition products to further increase the user's excitement.

[0397] Examples of prompt statements

[0398] "When a user is shopping in a virtual store, create an application that uses an emotion engine to analyze the user's emotions and provide an optimal shopping experience. If the user is excited, highlight new and limited edition products to further increase the user's excitement."

[0399] The above is a specific embodiment for carrying out the present invention.

[0400] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0401] Step 1:

[0402] The server captures the user's face using a webcam. The input is the real-time video from the webcam, and the output is the captured image data, which is used for subsequent emotion recognition processing.

[0403] Step 2:

[0404] The server inputs the captured image data into the EmotionRecognizer library to analyze the user's emotions. The input is image data, and the output is analyzed emotion data (e.g., happiness, sadness, excitement, etc.). EmotionRecognizer analyzes the user's facial expressions from the image data and identifies the emotion.

[0405] Step 3:

[0406] The server inputs the emotion data into the virtual experience generation module and generates a scenario for realizing the user's desires in a virtual space. The input is the emotion data and the user's desire information, and the output is the generated virtual experience scenario. The virtual experience generation module designs an appropriate virtual experience based on the user's emotions and desires.

[0407] Step 4:

[0408] The server inputs the virtual experience scenario to a virtual experience adjustment module and dynamically changes the experience content according to the emotion. The input is the virtual experience scenario and emotion data, and the output is the adjusted virtual experience scenario. The virtual experience adjustment module adjusts the content and intensity of the experience according to the user's emotion.

[0409] Step 5:

[0410] The server inputs the adjusted virtual experience scenario into a virtual store optimization module to adjust the shopping experience based on the user's emotions. The input is the adjusted virtual experience scenario and the emotion data, and the output is an optimized virtual store experience. The virtual store optimization module changes the way products are displayed and the recommended content according to the user's emotions.

[0411] Step 6:

[0412] The terminal displays the optimized virtual store experience on the head-mounted display. The input is the optimized virtual store experience, and the output is a virtual space that the user visually experiences. The head-mounted display provides the user with a virtual experience in real time.

[0413] The above are the specific processing steps of this system.

[0414] Example 2

[0415] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] In modern society, users are expected to fulfill desires and emotions in the virtual world that cannot be realized in the real world. However, conventional systems have the problem that it is difficult to adjust the experience in real time according to the user's emotions and provide an advanced virtual experience including haptic feedback.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of him, a means for providing the user with the feeling of returning to his hometown, a means for the user to go to a place he imagines, a means for the user to feel the touch of water, a means for analyzing the user's emotions and adjusting the virtual experience according to the emotions, and a means for generating a conversation in real time using a generation AI model. This allows the user to realize his or her desires in the virtual world and obtain the optimal experience according to his or her emotions.

[0418] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[0419] "A means to virtually realize the user's regrets" is a technology that allows users to recreate in a virtual space the wishes and hopes that they were unable to realize in the real world.

[0420] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character or avatar with which the user can converse in a virtual space.

[0421] "Means to give users the feeling of returning to their hometown" refers to technology that allows users to experience the scenery and environment of their hometown within a virtual space.

[0422] "A means for users to go to places they imagine" refers to technology that allows users to recreate specific locations or scenery within a virtual space, providing the experience of being in that place.

[0423] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[0424] "Means of analyzing a user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the emotions of a user from their facial expressions, tone of voice, choice of words, etc., and adjusts the content of the virtual experience based on the results of that analysis.

[0425] "Means for generating conversation in real time using a generative AI model" means technology that uses an artificial intelligence model to generate natural conversation in real time between a user and a virtual character.

[0426] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0427] The present invention is a system for allowing users to virtually realize their desires in the metaverse. The purpose of this system is to allow users to fulfill their desires and emotions that cannot be realized in the real world in the virtual world. Specific embodiments are described below.

[0428] Hardware and Software Used

[0429] Hardware: VR devices (e.g. VR headsets), AR devices (e.g. AR glasses), haptic feedback devices (e.g. haptic gloves)

[0430] Software: Metaverse platforms (e.g. virtual reality engines), sentiment analysis engines (e.g. emotion recognition software), generative AI models (e.g. natural language generation models)

[0431] System Overview

[0432] The user wears a VR or AR device and accesses the system. The user communicates their wishes to the system through voice or text input. For example, the user can input a specific wish such as "I want to talk to my grandchildren."

[0433] The server uses a generative AI model to analyze the user's input. Based on the analysis results, it generates a virtual experience to realize the user's desire. For example, if a user inputs "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates a scenario in which the user can have a conversation in real time.

[0434] The generated virtual experience data is sent from the server to the user's device. The device provides the user with a virtual experience through a VR or AR device based on the received data. A haptic feedback device is used to allow the user to feel touch in the virtual environment.

[0435] In addition, the server uses an emotion analysis engine to analyze the user's emotions from facial expressions, tone of voice, choice of words, etc. Based on the analysis results, the virtual experience can be adjusted. For example, if the user is feeling joy, the server will provide an experience that amplifies that joy.

[0436] Examples of concrete examples and prompts

[0437] Example 1: A user types, "I want to talk to my grandchild." The server uses a generative AI model to generate an avatar of the grandchild and starts a conversation in real time.

[0438] Example prompt: "I want to talk to my grandson. His name is Taro."

[0439] Example 2: A user types, "I want to go back to my hometown." The server uses the metaverse platform to recreate the scenery of the user's hometown.

[0440] Example prompt: "Please recreate the scenery of your hometown. The setting is a small village in Hokkaido."

[0441] Example 3: A user inputs "I want to see a beautiful ocean." The server generates a beautiful ocean scene and uses a haptic feedback device to provide the sensation of water.

[0442] Example prompt: "Show me a beautiful ocean scene. If possible, I'd also like to hear the sound of the waves."

[0443] In this way, the system realizes the user's desires in the virtual world and provides an optimal experience according to their emotions.

[0444] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0445] Step 1:

[0446] The user accesses the system and inputs their wishes. The user wears a VR or AR device and communicates their wishes to the system through voice or text input. For example, the user might say, "I want to talk to my grandchildren." The input data is converted to text format and sent to the server.

[0447] Step 2:

[0448] The server analyzes the user's desires and generates an appropriate virtual experience. The server uses a generative AI model to analyze the user's input text. Based on the analysis results, a virtual experience is generated to realize the user's desire. For example, in response to the input "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates the content of the conversation. The generated virtual experience data is output as 3D model data for the avatar and a conversation script.

[0449] Step 3:

[0450] The server transmits the generated virtual experience to the terminal. The server transmits the generated virtual experience data to the user's terminal. The data includes the avatar's movements, conversation content, and virtual environment settings. The terminal prepares to display the virtual experience based on the received data.

[0451] Step 4:

[0452] The terminal displays the virtual experience and provides haptic feedback as necessary. The terminal provides the user with a virtual experience through a VR device or an AR device based on the received virtual experience data. The user starts a conversation with the grandchild's avatar. The haptic feedback device is used to allow the user to feel haptics in the virtual environment. For example, when the user touches the grandchild, the haptic feedback device vibrates to provide a haptic sensation.

[0453] Step 5:

[0454] The server analyzes the user's emotions and adjusts the virtual experience. Using an emotion analysis engine, the server analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc. The server adjusts the virtual experience based on the analysis results. For example, if the user is feeling happy, the content of the conversation and the avatar's actions are adjusted to amplify that joy. The input data for emotion analysis are the user's facial expressions and voice data, and the output is adjusted virtual experience data.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] In modern society, the means by which users can realize their desires in the virtual world are limited. Although there are systems that adjust virtual experiences according to the user's emotions, there is still a lack of means to analyze the user's emotions in real time and provide an optimal experience in a virtual shopping experience. This makes it difficult for users to have a more satisfying virtual experience.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0459] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to converse with, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the virtual shopping experience based on the user's emotions. This allows the user to realize his / her desires in the virtual world and further obtain an optimal virtual shopping experience according to his / her emotions.

[0460] "Means of linking with the metaverse" refers to technology for communicating with the metaverse, a virtual reality space, and sending and receiving data.

[0461] "A means to virtually realize the user's regrets" is a technology that allows the user to recreate and experience their desires and regrets in a virtual space.

[0462] "A means for generating a person with whom the user can converse in front of the user" refers to technology for generating avatars or characters with which the user can converse in a virtual space.

[0463] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0464] "A means to allow users to go to places they imagine" is a technology that generates a specific place imagined by the user in a virtual space, allowing the user to experience that place.

[0465] "A means for allowing users to feel the sensation of water" refers to technology that recreates the tactile sensation of water in a virtual space, providing users with the sensation of actually touching water.

[0466] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and changes the content of the virtual experience based on the results.

[0467] "Means for optimizing the virtual shopping experience based on user emotions" refers to technology that analyzes user emotions in real time and adjusts the content and delivery method of virtual shopping in response to those emotions.

[0468] The system for implementing the present invention includes a number of means for coordinating a virtual experience according to a user's emotions in cooperation with the metaverse. Specific embodiments are described below.

[0469] System Configuration

[0470] Hardware

[0471] Webcam: Used to capture the user's facial expressions.

[0472] Smartphone: A device that allows users to access the system and engage in the virtual experience.

[0473] Smart glasses: A device for providing an augmented reality (AR) experience.

[0474] Head-mounted display: A device for providing a virtual reality (VR) experience.

[0475] software

[0476] OpenCV: An image processing library used to analyze the user's facial expressions.

[0477] EmotionRecognizer: An emotion recognition library used to analyze user emotions in real time.

[0478] MetaverseAPI: An API for linking with the metaverse and providing virtual experiences.

[0479] Data processing and calculation

[0480] Image capture and emotion recognition

[0481] The server captures the user's face image in real time using a webcam, which is pre-processed using OpenCV and analyzes the user's emotions using the EmotionRecognizer library.

[0482] Coordinating Virtual Experiences

[0483] The server uses the Metaverse API to tailor the content of the virtual experience based on the analyzed emotion data. For example, if the user is recognized as "happy," the system will display special discounts and recommended products to make the virtual shopping experience more enjoyable.

[0484] Examples

[0485] When a user is enjoying shopping in a virtual store: The emotion engine analyzes the user's facial expressions and if it determines that the user is enjoying themselves, the system will display special discounts and recommended products.

[0486] If the user is tired: If the emotion engine analyzes the user's facial expressions and determines that the user is tired, the system will provide relaxing music and interactive guidance.

[0487] Examples of prompt statements

[0488] When a user is enjoying shopping in a virtual store, the emotion engine will analyze the user's facial expressions. If the system determines that the user is enjoying themselves, it will display special discounts and recommended products. If the system determines that the user is tired, it will provide relaxing music and interactive guidance.

[0489] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[0490] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0491] Step 1:

[0492] The server captures the user's facial image in real time using a webcam. The input is the video data from the webcam, and the output is the captured facial image, which is used in subsequent processing steps.

[0493] Step 2:

[0494] The server preprocesses the captured face image using OpenCV. The input is the face image obtained in step 1, and the output is the preprocessed face image. Specific operations include face detection, image resizing, noise removal, etc.

[0495] Step 3:

[0496] The server analyzes the preprocessed face image using the EmotionRecognizer library to recognize the user's emotions. The input is the preprocessed face image obtained in step 2, and the output is the recognized emotion data. Specific operations include extracting facial feature points, inputting them into the emotion model, and classifying emotions.

[0497] Step 4:

[0498] The server adjusts the content of the virtual experience using the Metaverse API based on the recognized emotion data. The input is the emotion data obtained in step 3, and the output is the adjusted virtual experience settings. Specific operations include selecting a scenario for the virtual experience according to the emotion data, placing virtual objects, setting interactions, etc.

[0499] Step 5:

[0500] The terminal receives the adjusted virtual experience settings sent from the server and provides them to the user. The input is the virtual experience setting data obtained in step 4, and the output is the virtual environment experienced by the user. Specific operations include rendering the virtual environment, displaying a user interface, and executing interactions.

[0501] Step 6:

[0502] The user enjoys a virtual experience provided through the device. The input is the virtual environment provided by the device, and the output is the user's experience and changes in emotions. Specific actions include movement within the virtual environment, interaction with objects, and emotional feedback.

[0503] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[0504] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0505] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0506] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0507] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0508] [Second embodiment]

[0509] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0510] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0511] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0512] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0513] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0514] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0515] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0516] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0517] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0518] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0519] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0520] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0521] "Example 1"

[0522] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[0523] "Example 2"

[0524] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[0525] The process flow of each embodiment will be described below.

[0526] "Example 1"

[0527] Step 1: The user starts the system and inputs their desires. For example, they can input their desires, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean," in text or voice.

[0528] Step 2: The system analyzes the input desire and generates a corresponding virtual experience. For example, if the desire is "I want to talk to my grandchild," the system will generate an avatar of the user's grandchild and enable real-time conversation.

[0529] Step 3: The system provides the generated virtual experience to the user. The user can experience the virtual experience through a VR or AR device. For example, if the scenery of the user's hometown is reproduced, the user can feel as if they are there. Step 4: The system monitors the user's response and adjusts the virtual experience as necessary. For example, if the user responds that they would like to feel the touch of water while looking at a seascape, the system utilizes a means of providing haptic feedback to provide the user with the touch of water.

[0530] Example 1

[0531] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0532] In modern society, many users face the challenge of finding it difficult to realize their regrets due to physical and time constraints. For example, it is difficult to talk to far-away family members or experience the feeling of returning to one's hometown. It is also difficult for users to go to places they imagine or experience real sensations through touch. To solve these problems, a means is needed to realize users' wishes using the virtual world.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0534] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for analyzing the user's wishes, a means for generating a virtual experience, and a means for setting haptic feedback. This allows the user to go beyond physical constraints and experience their wishes in a realistic way in the virtual world.

[0535] The "metaverse" is a virtual world built on the Internet, a virtual environment in which users can interact with other users through avatars and engage in various activities.

[0536] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in the virtual world the desires or experiences that they were unable to realize in the real world.

[0537] "Means for generating a person with whom the user can converse in front of the user" refers to a technology or method for generating an avatar of the person with whom the user wants to converse in the virtual world and simulating a conversation in real time.

[0538] "Means of giving the user the feeling of returning to their hometown" refers to technologies or methods that recreate the scenery or environment of the user's hometown in a virtual world, giving the user the feeling of being in that place.

[0539] "A means to go to the place the user imagines" refers to the technology or method that recreates the place the user imagines he or she would like to go to in the virtual world and allows the user to access that place.

[0540] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when interacting with water in the virtual world.

[0541] "Means for analyzing user wishes" refers to techniques or methods for analyzing the wishes entered by the user, understanding their content, and extracting the necessary information.

[0542] A "means for generating a virtual experience" is a technology or method for generating an experience in a virtual world based on a user's desires.

[0543] "Means for configuring haptic feedback" means techniques or methods for configuring and controlling a haptic feedback device to make a user's experience in a virtual world more realistic.

[0544] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0545] The present invention is a system for enabling a user to realize his / her own desires in a virtual world. A specific embodiment of this system will be described below.

[0546] System Configuration

[0547] This system consists of three main elements: a server, a terminal, and a user. The server works in conjunction with the metaverse to analyze users' desires and generate virtual experiences. The terminal provides an interface for users to access the system and enjoy the virtual experience. Users input their desires through the terminal and experience the virtual experience.

[0548] Hardware and Software Used

[0549] Server: A server with high-performance computing power is required, preferably one with the computing resources to run generative AI models (e.g., OpenAI's GPT-4).

[0550] Terminal: The device through which the user accesses the device, which may be a PC, tablet, smartphone, etc. Also includes haptic feedback devices (e.g. HaptX gloves).

[0551] Software: Game engines (e.g. Unity, Unreal Engine) are used to build virtual worlds, and generative AI models are used to perform natural language processing.

[0552] Program Processing

[0553] The server receives the user's desire and analyzes it. It uses a generative AI model to understand the user's input and extract the necessary information. The server then generates the virtual experience. Specifically, it generates an avatar of the person the user wants to talk to and simulates a real-time conversation. It also uses a game engine to build a realistic 3D environment to recreate the user's hometown or the scenery they want to see. It also controls a haptic feedback device and configures it to make the user's experience in the virtual world feel more realistic.

[0554] Examples

[0555] If a user wishes to have a conversation with their grandchild, the specific steps that are taken are as follows:

[0556] 1. User's wish input: The user inputs "I want to talk to my grandson. His grandson's name is Taro, and he is 10 years old" into the input field of the terminal.

[0557] 2. Wish analysis: The server uses a generative AI model to analyze the wish “I want to talk to my grandchild” and extract details such as the grandchild’s name and age.

[0558] 3. Generating the virtual experience: The server uses a generative AI model to generate an avatar of the user's 10-year-old grandson, Taro, and simulates a real-time conversation with him. It also uses Unity to create a 3D recreation of the user's hometown.

[0559] 4. Setting up haptic feedback: The server controls the HaptX gloves and configures them so that the user can feel the sensation of water in the virtual world.

[0560] 5. Providing the virtual experience: The server provides the generated virtual experience to the user. The user can interact with the grandchild’s avatar through the terminal display and feel the sensation of water through the HaptX gloves.

[0561] Examples of prompt statements

[0562] "I want to have a conversation with my grandson. His name is Taro and he is 10 years old."

[0563] "I want to see the scenery of my hometown. It's a small village in the north."

[0564] "I want to see a beautiful ocean. Preferably one like the ocean in the south."

[0565] This allows users to transcend physical constraints and experience their desires in a realistic way in the virtual world.

[0566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0567] Step 1:

[0568] The user uses the terminal to input his / her wish. For example, he / she inputs, "I want to talk to my grandson. His / her name is Taro, and he / she is 10 years old." This input is sent to the server.

[0569] Step 2:

[0570] The server analyzes the desire received from the user. The input text data is passed to the generative AI model, which understands the user's desire and extracts the necessary information. Specifically, it analyzes the desire "I want to talk to my grandchild" and the detailed information "My grandchild's name is Taro and he is 10 years old." The analysis result is passed to the next processing step.

[0571] Step 3:

[0572] The server generates the virtual experience based on the analysis results. First, it uses a generative AI model to generate an avatar of the person the user wants to talk to. Specifically, it generates an avatar of the user's 10-year-old grandson, "Taro," and simulates a real-time conversation. The data for this avatar generation is passed to the next processing step.

[0573] Step 4:

[0574] The server uses a game engine (e.g. Unity) to create a realistic 3D environment to recreate the user's hometown or scenery they want to see. For example, if a user inputs "I want to see the scenery of my hometown," a 3D model is generated based on the scenery data of that place. This 3D environment data is passed to the next processing step.

[0575] Step 5:

[0576] The server controls the haptic feedback device (e.g., HaptX gloves) and configures it to make the user's experience in the virtual world feel more realistic. For example, if the user inputs "I want to see a beautiful ocean," the server configures the haptic feedback device to reproduce the feeling of water. This configuration data is passed to the next processing step.

[0577] Step 6:

[0578] The server provides the generated virtual experience to the user. The user interacts with the generated avatar through the display of the terminal and experiences the 3D environment. The user can also feel the touch of water through a haptic feedback device. The data of this virtual experience is sent to the user, and the user can enjoy the experience in the virtual world.

[0579] (Application example 1)

[0580] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0581] In modern society, users need to have realistic experiences in virtual spaces without having to physically move around. However, current systems have difficulty providing sufficient tactile and visual reality when users try on or use products in virtual spaces. In addition, there is a lack of real-time interaction and haptic feedback when users virtually interact with specific places or people. This limits the user's experience and reduces satisfaction.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0583] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to try on or use products in a virtual store, a means for generating an avatar for the user, a means for generating a scene of the virtual store, and a means for providing haptic feedback. This allows the user to have a real experience in the virtual space.

[0584] The "metaverse" is an interactive digital environment built in a virtual space where users can interact with other users and content through avatars.

[0585] "A means to virtually realize the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, allowing the user to experience them.

[0586] "A means for generating a person with whom the user can converse in front of the user" refers to technology that generates an avatar or character with which the user can converse in real time in a virtual space.

[0587] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[0588] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users in a virtual space and provides the user with the experience of visiting those places.

[0589] "Means allowing the user to feel the sensation of water" refers to technology that allows the user to sense the presence of water in a virtual space through haptic feedback.

[0590] "Means for users to try on or use products in a virtual store" refers to technology that provides users with the experience of trying on or using products in a store within a virtual space.

[0591] "Means for generating a user's avatar" refers to technology for generating an avatar for use in a virtual space based on the user's appearance and characteristics.

[0592] "Means for generating a virtual store scene" refers to technology for reproducing the environment and scenery of a store within a virtual space.

[0593] A "means for providing haptic feedback" is a haptic device or technology that enables a user to physically feel their virtual experience.

[0594] In order to implement this invention, the following hardware and software are used. The hardware requires a smartphone and a haptic feedback device. The software requires the OpenAI API and Virtual Reality SDK.

[0595] The server provides a means to link with the metaverse and uses generative AI models to generate data to virtually realize the user's wishes. For example, if a user wants to talk to a specific person, an avatar of that person can be generated and the user can talk to them in real time.

[0596] The terminal provides an interface for the user to try on or try on products in the virtual store. The user can access the virtual store using a smartphone, select products, and try them on or try them on. For example, if the user wants to try on clothes, the terminal can generate an avatar of the user and dress the avatar in the selected clothes. In addition, by using a haptic feedback device, the user can actually feel the texture of the clothes.

[0597] If a user wants to go to a specific place, the server generates a scene of that place and gives the user the feeling that he or she is there. For example, if a user wants to see the sea, the server generates a scene of the sea that the user imagines and allows the user to go to that place. In this case, a haptic feedback device can be used to provide the feeling of water.

[0598] As an example, the following prompt can be used:

[0599] Avatar generation: "Generate an avatar for the user with the following data: name, height, weight"

[0600] Generate a virtual store with the following data: store name, location

[0601] Try on product: "Apply the following product to the avatar: product type, color, size on user data"

[0602] This allows users to have a realistic experience in a virtual space.

[0603] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0604] Step 1:

[0605] The server receives a user request. As input, it contains the user's desired experience (e.g., talking to a specific person, visiting a specific place). The server analyzes this request and sends the necessary data to the generative AI model. As output, it gets the analysis result.

[0606] Step 2:

[0607] The server uses a generative AI model to generate avatars and scenery based on the user's request. The input includes the user's feature data and details of the desired experience. The generative AI model processes these data and generates avatar and scenery data. The output is the generated avatar and scenery data.

[0608] Step 3:

[0609] The server sends the generated avatar and scenery data to the terminal. The input contains the generated avatar and scenery data. The server converts these data into an appropriate format and sends it to the terminal. The output is the data sent to the terminal.

[0610] Step 4:

[0611] The terminal constructs a virtual space based on the received data. The input includes avatar and scenery data sent from the server. The terminal renders this data and displays it to the user. The output is the virtual space displayed to the user.

[0612] Step 5:

[0613] The user begins to experience the virtual space through the device. The input includes the virtual space displayed on the device. The user manipulates an avatar to talk to specific people or visit specific places. The output is the result of the user's manipulation.

[0614] Step 6:

[0615] The terminal transmits the user's operation to the server. The input includes the user's operation data. The terminal transmits these data to the server to realize real-time interaction. The output is the operation data transmitted to the server.

[0616] Step 7:

[0617] The server updates the virtual space based on the user's operation data. The input includes the user's operation data. The server analyzes these data and updates the state of the virtual space. The output is the updated data of the virtual space.

[0618] Step 8:

[0619] The server sends the updated virtual space data to the terminal. As input, the updated virtual space data is included. The server converts these data into an appropriate format and sends them to the terminal. As output, the updated data sent to the terminal is obtained.

[0620] Step 9:

[0621] The terminal displays the updated virtual space to the user. As input, it contains the updated data sent from the server. The terminal renders these data and displays them to the user. As output, it obtains the updated virtual space displayed to the user.

[0622] Step 10:

[0623] The user physically feels the experience in the virtual space using the haptic feedback device. The input includes haptic feedback data sent from the terminal, which allows the user to feel the experience in the virtual space more realistically. The output is the user's haptic experience.

[0624] Example 2

[0625] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0626] In modern society, many users want to recreate their past memories and regrets in virtual space. However, current systems have difficulty realizing users' specific desires in real time, and are unable to provide sensory experiences such as haptic feedback. For this reason, there is a demand for systems that allow users to experience virtual space in a more realistic way.

[0627] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0628] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the user's regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to their hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for receiving the user's input and generating a prompt sentence using a generative AI model, a means for generating a virtual space using virtual space generation software, and a means for providing haptic feedback using a haptic feedback device. This allows the user to realize their specific desires in the virtual space in real time and further obtain a more realistic experience through haptic feedback.

[0629] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[0630] "A means of virtually realizing the things that users regret" refers to technology that enables users to recreate in a virtual space things or experiences from the past that they feel nostalgic about.

[0631] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character with which the user can converse in a virtual space.

[0632] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0633] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users within a virtual space, allowing the users to visit those places.

[0634] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[0635] "Means for receiving user input and generating a prompt sentence using a generative AI model" refers to technology that analyzes information entered by the user and generates an appropriate prompt sentence using a generative AI model.

[0636] "Means for generating a virtual space using virtual space generation software" refers to a technology that uses software for constructing a virtual space to generate a virtual space in response to a user's request.

[0637] "Means for providing haptic feedback using a haptic feedback device" refers to technology that uses a device to actually reproduce the sensation when a user touches a specific object or environment in a virtual space.

[0638] The present invention is a system that allows users to realize their wishes in a virtual space. This system works in conjunction with the metaverse and provides multiple means for users to virtually realize their wishes.

[0639] Hardware and Software Used

[0640] Hardware: Terminals (PC, smartphone, VR device), servers, haptic feedback devices (e.g. HaptX gloves)

[0641] Software: Generative AI models (e.g. GPT-4), virtual world generation software (e.g. Unity, Unreal Engine)

[0642] System Configuration

[0643] 1. User input of wishes

[0644] The user uses the terminal to input their own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0645] 2. Data collection and analysis

[0646] The server receives user input and uses a generative AI model to generate prompts. For example, if a user wants to talk to their grandchild, it generates an avatar of the grandchild and generates prompts that enable real-time conversation.

[0647] 3. Creation of Virtual Space

[0648] The server uses virtual space generation software to generate a virtual space based on the user's wishes, such as "recreating the scenery of the user's hometown" or "generating the seascape that the user imagines."

[0649] 4. Avatar Creation and Conversation

[0650] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[0651] 5. Providing haptic feedback

[0652] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[0653] Examples

[0654] Example 1: I want to talk to my grandchildren

[0655] The user types into the terminal, "I want to talk to my grandchild."

[0656] The server uses the generative AI model to generate "prompts that generate an avatar of the user's grandchild and enable real-time conversation."

[0657] The server uses virtual space generation software to generate an avatar of the grandchild and conduct the conversation. The grandchild's avatar says, "Hello, Grandpa! What would you like to talk about today?"

[0658] Example 2: I want to go back to my hometown

[0659] The user types "I want to go back to my hometown" into the terminal.

[0660] The server uses a generative AI model to generate "prompt sentences that recreate the scenery of the user's hometown."

[0661] The server uses virtual space generation software to recreate the scenery of the person's hometown, allowing the user to walk around the scenery in the virtual space.

[0662] Example 3: I want to see a beautiful ocean

[0663] The user types "I want to see a beautiful ocean" into the terminal.

[0664] The server uses a generative AI model to generate "prompt sentences that generate the seascape the user imagines and provide haptic feedback."

[0665] The server uses virtual space generation software to generate the ocean scenery and a haptic feedback device to provide the feel of water: when a user places their hand in the virtual ocean, the HaptX gloves make them feel the cold and flowing water.

[0666] The system allows users to realize their specific desires in a virtual space in real time, and provides a more realistic experience through haptic feedback.

[0667] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0668] Step 1:

[0669] User's wish input

[0670] The user uses the terminal to input his / her own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0671] Input: User's wish (in text format)

[0672] Output: User wish data

[0673] Specific actions: The user inputs their wish in text form using the device's keyboard or voice input function. For example, the user inputs "I want to talk to my grandchildren" into the input form displayed on the device screen.

[0674] Step 2:

[0675] Data collection and analysis

[0676] The server receives the user's input and generates a prompt using a generative AI model.

[0677] Input: User's wish data

[0678] Output: Prompt statement

[0679] Specific operation: The server analyzes the user's input data and instructs the generative AI model to generate "a prompt sentence that generates an avatar of the grandchild and enables real-time conversation if the user wants to talk to the grandchild."

[0680] Step 3:

[0681] Creating a Virtual Space

[0682] The server uses virtual space generation software to generate a virtual space based on the user's wishes.

[0683] Input: prompt statement

[0684] Output: Virtual space data

[0685] Specific operation: The server issues instructions to the virtual space generation software, such as "recreate the scenery of the user's hometown" or "generate the seascape that the user imagines," and constructs the virtual space.

[0686] Step 4:

[0687] Avatar creation and conversation

[0688] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[0689] Input: Prompt text, virtual space data

[0690] Output: Avatar, conversation data

[0691] How it works: The server instructs the generative AI model to generate an avatar of the user's grandchild, and then applies a dialogue model to enable real-time conversation. For example, the grandchild's avatar might say, "Hello, Grandpa! What would you like to talk about today?"

[0692] Step 5:

[0693] Providing haptic feedback

[0694] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[0695] Input: Virtual space data, user operation data

[0696] Output: Haptic feedback data

[0697] Specific operation: The server instructs the haptic feedback device to reproduce the feel of water when the user touches the ocean in the virtual space. For example, when the user puts his or her hand in the ocean in the virtual space, the haptic feedback device makes the user feel the coldness and flow of the water.

[0698] Through these steps, users can realize their specific desires in the virtual space in real time, and get a more realistic experience through haptic feedback.

[0699] (Application example 2)

[0700] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0701] In modern society, users are required to have realistic experiences in virtual spaces without having to physically move. However, conventional systems have been unable to fully satisfy users' desires, and in particular, in the shopping experience in virtual stores, it was difficult to have a real-time conversation with an assistant or feel the texture of the products. This resulted in low user satisfaction and made the experience in virtual spaces inferior to reality.

[0702] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0703] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to select a product while talking to an assistant avatar in real time when enjoying shopping in a virtual store, a means for the user to walk around in a virtual store that reproduces a specific place or scenery, and a means for the user to feel the texture of the product through haptic feedback. This allows the user to enjoy a real shopping experience in a virtual space.

[0704] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[0705] "A means of virtually realizing the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, making the user feel as if they are actually experiencing them.

[0706] "A means for generating a person with whom a user can converse in front of the user" refers to a technology that enables a user to generate an avatar of a person with whom the user wants to converse in a virtual space, thereby enabling the user to converse in real time.

[0707] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[0708] "A means to go to places that users imagine" is a technology that generates the places that users imagine they would like to go to in a virtual space and provides the user with the experience of visiting those places.

[0709] "A means for allowing the user to feel the sensation of water" refers to technology that recreates the presence of water in a virtual space and allows the user to feel the sensation of that water through haptic feedback.

[0710] "A means that enables users to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store" refers to technology that enables users to select products while conversing in real time with an assistant avatar when shopping in a virtual store.

[0711] "A means for enabling users to walk around a virtual store that recreates a specific place or scenery" refers to technology that enables users to move freely within a virtual store and experience an environment that recreates a specific place or scenery.

[0712] "Means for allowing a user to feel the texture of a product through haptic feedback" refers to technology that allows a user to feel the texture of a product through a haptic feedback device when picking up the product in a virtual space.

[0713] A system for implementing this invention includes means for linking with the metaverse, means for virtually realizing a user's regrets, means for generating a person for the user to converse with in front of them, means for giving the user the feeling of returning to their hometown, means for allowing the user to go to places the user imagines, means for the user to feel the feel of water, means for a user to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store, means for a user to walk around a virtual store that recreates a specific place or scenery, and means for the user to feel the texture of products through haptic feedback.

[0714] System Program Overview

[0715] The server uses the OpenAI API, the virtual environment library (virtual_environment), and the haptic feedback library (haptic_feedback) to process and calculate data to realize the user's wishes. Specifically, the server generates prompts based on the user's wishes, generates avatars and virtual environments using generative AI models, and provides the user with a realistic sensation through a haptic feedback device.

[0716] Processing flow

[0717] 1. Generate prompts based on user desires

[0718] The server describes the user's wishes in natural language and inputs them into the generative AI model. For example, if the user wishes to "talk to their grandchildren," the server generates a prompt sentence like this:

[0719] The user wants to experience conversation with his / her grandchild.

[0720] 2. Avatar Creation

[0721] The server uses the OpenAI API to generate an avatar based on the user's wishes, and the generated avatar is then used by the user to converse in real time in a virtual space.

[0722] 3. Setting up the virtual environment

[0723] The server uses the virtual environment library to set up a virtual store for the user to experience, and the user can freely walk around the virtual store.

[0724] 4. Haptic feedback settings

[0725] The server uses a haptic feedback library to allow the user to feel the texture of the product, so that when the user picks up the product in the virtual space, they can feel the texture of the product in a realistic way.

[0726] Examples

[0727] For example, if a user has a wish to "talk to his / her grandchild," the server will generate an avatar of the grandchild and enable the user to talk to the grandchild in real time in the virtual store. In addition, when the user picks up a particular product, the user can feel the texture of the product through a haptic feedback device.

[0728] In this way, users can enjoy a real shopping experience in a virtual space.

[0729] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0730] Step 1:

[0731] The server generates a prompt based on the user's desire. If the user has the desire "I want to talk to my grandchildren," the server takes this information as input and generates a prompt like this:

[0732] The user wants to experience conversation with his / her grandchild.

[0733] This prompt is then fed into a generative AI model.

[0734] Step 2:

[0735] The server uses the generative AI model to generate an avatar based on the prompt sentence. Specifically, it calls the OpenAI API and provides the prompt sentence as input. The API analyzes the prompt sentence and generates an avatar of the user's grandchild. This avatar is used by the user to converse in real time in the virtual space. The generated avatar is sent from the server to the user's device.

[0736] Step 3:

[0737] The server uses the virtual environment library to set up a virtual store for the user to experience. If the user has a desire to "enjoy shopping in a virtual store," the server receives this information as input and calls the virtual environment library to generate a virtual store. The generated virtual store is sent to the user's terminal, and the user can freely walk around the virtual store.

[0738] Step 4:

[0739] The server uses the haptic feedback library to allow the user to feel the texture of the product. When the user picks up a particular product, the information is sent to the server. The server calls the haptic feedback library to generate data about the texture of the product. This data is sent to the user's haptic feedback device, allowing the user to feel the texture of the product in a realistic way.

[0740] Step 5:

[0741] Users can walk around the virtual store and select products while talking to the generated avatar in real time. When the user selects a product, the information is sent to the server, and the server sends detailed information about the selected product to the user's device. Based on this information, the user can decide whether to select more products or purchase them.

[0742] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0743] "Example 1"

[0744] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[0745] "Example 2"

[0746] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[0747] The process flow of each embodiment will be described below.

[0748] "Example 1"

[0749] Step 1: The user connects to the system through a VR or AR device.

[0750] Step 2: The user inputs their desired virtual experience into the system.

[0751] Step 3: The system uses an emotion engine to analyze the user's emotions based on facial expressions, tone of voice, choice of words, etc.

[0752] Step 4: The system tailors the virtual experience based on the analysis results and provides it to the user.

[0753] Step 5: The user experiences the provided virtual experience.

[0754] Step 6: The system monitors the user's reactions and adjusts the virtual experience if necessary.

[0755] Example 1

[0756] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0757] In modern society, there is a demand for a way to allow users to realize their regrets in the virtual world. In particular, it is important to realize experiences that are difficult to experience in the real world, such as talking with far-away family members, returning to one's hometown, and experiencing natural environments. There is also a demand to provide a more realistic and satisfying experience by analyzing the user's emotions and adjusting the virtual experience according to those emotions.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place imagined by the user, a means for the user to feel the touch of water, a means for analyzing the user's emotions, a means for adjusting the virtual experience based on the analysis result, and a means for generating a virtual experience using a generation AI model. This allows the user to realize his / her desires in the virtual world and enjoy the optimal virtual experience according to his / her emotions.

[0759] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[0760] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in a virtual space the desires or experiences that they were unable to realize in the real world.

[0761] "Means for generating a person with whom the user can converse in front of the user" refers to a technique or method for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[0762] "Means of giving users the feeling of returning to their hometown" refers to technologies and methods that allow users to recreate the scenery and environment of their hometown in a virtual space, giving them the feeling as if they were there.

[0763] "Means that allow users to go to places they imagine" refers to technologies or methods that allow users to recreate specific locations or environments in a virtual space, providing the experience of being in that place.

[0764] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when touching water in a virtual space.

[0765] "Means of analyzing user emotions" refers to techniques or methods for analyzing emotions from a user's facial expressions, tone of voice, choice of words, etc.

[0766] "Means for adjusting the virtual experience based on the analysis results" refers to technologies and methods for optimizing the experience in the virtual space based on the results of an analysis of the user's emotions, and providing an experience that corresponds to the user's emotions.

[0767] A "means for generating a virtual experience using a generative AI model" is a technology or method that uses an artificial intelligence model to generate experiential scenarios or content in a virtual space based on user input data.

[0768] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0769] This invention is a system for realizing a user's regrets in a virtual world, and provides a technology that works in conjunction with the metaverse to analyze the user's emotions and optimize the virtual experience. A specific embodiment of this system will be described below.

[0770] System Configuration

[0771] The system consists of the following main components:

[0772] 1. User device:

[0773] A device that allows a user to access the system, including PCs, smartphones, tablets, VR devices, AR devices, etc.

[0774] Users access the system through a dedicated application or web interface and enter their wishes.

[0775] 2. Server:

[0776] It takes user input data and uses a generative AI model to generate a virtual experience.

[0777] An emotion engine is used to analyze the user's emotions and tailor the virtual experience based on the analysis results.

[0778] 3. Generative AI model:

[0779] As an example, use a natural language processing model such as OpenAI's GPT-4.

[0780] Generates virtual experience scenarios and content based on user input data.

[0781] 4. Emotion Engine:

[0782] It includes algorithms that analyze emotions from users' facial expressions, tone of voice, choice of words, etc.

[0783] The content of the virtual experience is adjusted based on the analysis results.

[0784] System Operation

[0785] 1. User inputs their wishes:

[0786] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0787] 2. Data analysis and generation:

[0788] The server receives the user's input data and generates the virtual experience using a generative AI model.

[0789] For example, if a user types, "I want to talk to my grandchild," the server will generate an avatar of the user's grandchild and enable a real-time conversation.

[0790] 3. Sentiment analysis using emotion engine:

[0791] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[0792] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[0793] The content of the virtual experience is adjusted based on the analysis results.

[0794] 4. Providing Virtual Experiences:

[0795] The server streams the generated virtual experience data to the user's device.

[0796] Users put on a VR headset or AR glasses and begin the virtual experience.

[0797] For example, a user could put on a VR headset and enjoy a real-time conversation with an avatar of their grandchild.

[0798] Examples and prompts

[0799] As a concrete example, a process will be described in which the user has a desire to "talk to his / her grandchild."

[0800] 1. User input:

[0801] The user types into the terminal, "I want to talk to my grandchild."

[0802] 2. Data analysis and generation:

[0803] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an avatar of the user's grandchild.

[0804] 3. How the Emotion Engine works:

[0805] The emotion engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling joy.

[0806] 4. Providing Virtual Experiences:

[0807] The server then transmits the real-time conversation between the user and the avatar of the grandchild to the user's device, and the user experiences the conversation with the grandchild through a VR device.

[0808] Example prompt:

[0809] If a user types, "I want to talk to my grandchild," generate a program that creates an avatar of the user's grandchild and enables a real-time conversation.

[0810] In this way, a system is constructed that realizes users' wishes in the virtual world.

[0811] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0812] System program processing flow

[0813] Step 1: User inputs their wishes

[0814] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[0815] Input: The user's desire (e.g., "I want to talk to my grandchildren")

[0816] Output: User wish data

[0817] Specific behavior:

[0818] Users enter their wishes into a text box using a dedicated application or a web interface.

[0819] For example, type "I want to talk to my grandchildren."

[0820] Step 2: Analyze and generate data

[0821] The server receives the user's input data and generates the virtual experience using a generative AI model.

[0822] Input: User's wish data

[0823] Output: Virtual experience scenario

[0824] Specific behavior:

[0825] The server sends the user's input to a natural language processing engine, which analyzes the input.

[0826] Based on the analysis results, a generative AI model (e.g., OpenAI's GPT-4) is invoked to generate a virtual experience scenario.

[0827] Example: In response to the input "I want to talk to my grandchild," an avatar of the grandchild is generated and a conversation scenario is created.

[0828] Step 3: Sentiment analysis using the emotion engine

[0829] The server analyzes the user's emotions using an emotion engine, which analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0830] Input: User's camera video and audio data

[0831] Output: Emotion analysis results

[0832] Specific behavior:

[0833] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[0834] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[0835] The content of the virtual experience is adjusted based on the analysis results.

[0836] Example: If the user is smiling, the system will provide conversational content that amplifies the joy.

[0837] Step 4: Create the virtual experience

[0838] The server generates an optimized virtual experience based on the results of the emotion analysis, which is tailored to the user's desires and emotions.

[0839] Input: Sentiment analysis results, virtual experience scenario

[0840] Output: Optimized virtual experience data

[0841] Specific behavior:

[0842] The server integrates the results of the generative AI model and the emotion engine to create an optimal virtual experience scenario.

[0843] Example: If a user types "I want to talk to my grandchild," and sentiment analysis indicates joy, the grandchild's avatar can be set to offer happy topics.

[0844] Step 5: Offer virtual experiences

[0845] The server transmits the generated virtual experience to the user's device, where the user experiences the virtual experience through a VR or AR device.

[0846] Input: Optimized virtual experience data

[0847] Output: The user's virtual experience

[0848] Specific behavior:

[0849] The server streams the generated virtual experience data to the user's device.

[0850] Users put on a VR headset or AR glasses and begin the virtual experience.

[0851] Example: A user puts on a VR headset and enjoys a real-time conversation with an avatar of their grandchild.

[0852] In this way, a system is constructed that realizes users' wishes in the virtual world.

[0853] (Application example 1)

[0854] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0855] In modern society, users are expected to realize their desires in virtual space without moving physically. In addition, it is a challenge to optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience. In particular, it is important to adjust the shopping experience based on the user's emotions in virtual stores.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0857] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the experience of the virtual store based on the user's emotions. This allows the user to realize his / her desires in the virtual space without physical constraints and enjoy the optimal experience according to his / her emotions.

[0858] "Means of linking with the metaverse" refers to technology that exchanges data with the metaverse, a virtual space, and provides an interface that allows users to have virtual experiences.

[0859] "A means to virtually realize the user's regrets" is a technology that enables users to recreate in a virtual space the desires and experiences that they were unable to realize in the real world.

[0860] "Means for generating a person with whom the user can converse in front of the user" refers to a technology for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[0861] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0862] "A means to go to places the user imagines" refers to technology that generates specific places or landscapes imagined by the user in a virtual space, allowing the user to access those places.

[0863] "A means for allowing the user to feel the sensation of water" refers to haptic technology that provides the user with feedback of the actual sensation of water when they touch water in a virtual space.

[0864] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and dynamically changes the content of the virtual experience based on the results.

[0865] The "means for optimizing the virtual store experience based on the user's emotions" is a technology that analyzes the user's emotional state and adjusts the shopping experience in the virtual store according to that state.

[0866] The system for implementing the present invention aims to enable a user to realize his / her desires in a virtual space and enjoy an optimal experience according to his / her emotions. A specific embodiment of the system will be described below.

[0867] System Configuration

[0868] The system consists of the following main components:

[0869] 1. Metaverse Linkage Module

[0870] It exchanges data with the metaverse and provides an interface for users to have virtual experiences. Specifically, it works with the metaverse platform through APIs.

[0871] 2. Virtual experience generation module

[0872] This is a module that allows users to virtually realize their regrets. For example, if a user wants to talk to their grandchild, it will generate an avatar of the grandchild and enable a real-time conversation.

[0873] 3. Emotion Recognition Module

[0874] This is a module for analyzing emotions from the user's facial expressions, tone of voice, etc. Specifically, it uses libraries such as OpenCV and EmotionRecognizer.

[0875] 4. Virtual Experience Tuning Module

[0876] This module dynamically changes the content of the virtual experience based on the emotion data obtained from the emotion recognition module. For example, if the user is feeling joy, it will provide an experience that amplifies that joy.

[0877] 5. Virtual store optimization module

[0878] This module tailors the virtual store shopping experience based on the user's emotions, for example highlighting new or limited edition products if the user is excited.

[0879] Hardware and Software Used

[0880] Hardware: Webcam, Head Mounted Display

[0881] Software: OpenCV (image processing library), EmotionRecognizer (emotion recognition library), VirtualStore (virtual store management library)

[0882] Processing flow

[0883] 1. User emotion recognition

[0884] It uses a webcam to capture the user's face and analyzes emotions using the EmotionRecognizer library.

[0885] 2. Creating and adjusting virtual experiences

[0886] Based on the emotion data obtained from the emotion recognition module, the virtual experience generation module realizes the user's desire in the virtual space. Furthermore, the virtual experience adjustment module dynamically changes the experience content according to the emotion.

[0887] 3. Virtual store optimization

[0888] The virtual store optimization module tailors the shopping experience based on user emotions, for example highlighting new or limited edition products if the user is excited.

[0889] Examples

[0890] When a user is shopping in a virtual store, if the system detects "excitement" in the user's facial expression, it will highlight new or limited edition products to further increase the user's excitement.

[0891] Examples of prompt statements

[0892] "When a user is shopping in a virtual store, create an application that uses an emotion engine to analyze the user's emotions and provide an optimal shopping experience. If the user is excited, highlight new and limited edition products to further increase the user's excitement."

[0893] The above is a specific embodiment for carrying out the present invention.

[0894] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0895] Step 1:

[0896] The server captures the user's face using a webcam. The input is the real-time video from the webcam, and the output is the captured image data, which is used for subsequent emotion recognition processing.

[0897] Step 2:

[0898] The server inputs the captured image data into the EmotionRecognizer library to analyze the user's emotions. The input is image data, and the output is analyzed emotion data (e.g., happiness, sadness, excitement, etc.). EmotionRecognizer analyzes the user's facial expressions from the image data and identifies the emotion.

[0899] Step 3:

[0900] The server inputs the emotion data into the virtual experience generation module and generates a scenario for realizing the user's desires in a virtual space. The input is the emotion data and the user's desire information, and the output is the generated virtual experience scenario. The virtual experience generation module designs an appropriate virtual experience based on the user's emotions and desires.

[0901] Step 4:

[0902] The server inputs the virtual experience scenario to a virtual experience adjustment module and dynamically changes the experience content according to the emotion. The input is the virtual experience scenario and emotion data, and the output is the adjusted virtual experience scenario. The virtual experience adjustment module adjusts the content and intensity of the experience according to the user's emotion.

[0903] Step 5:

[0904] The server inputs the adjusted virtual experience scenario into a virtual store optimization module to adjust the shopping experience based on the user's emotions. The input is the adjusted virtual experience scenario and the emotion data, and the output is an optimized virtual store experience. The virtual store optimization module changes the way products are displayed and the recommended content according to the user's emotions.

[0905] Step 6:

[0906] The terminal displays the optimized virtual store experience on the head-mounted display. The input is the optimized virtual store experience, and the output is a virtual space that the user visually experiences. The head-mounted display provides the user with a virtual experience in real time.

[0907] The above are the specific processing steps of this system.

[0908] Example 2

[0909] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0910] In modern society, users need to be able to fulfill desires and emotions that cannot be realized in the real world in the virtual world. However, conventional systems have the problem that it is difficult to adjust the experience in real time according to the user's emotions and provide an advanced virtual experience that includes haptic feedback.

[0911] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of him, a means for providing the user with the feeling of returning to his hometown, a means for the user to go to a place he imagines, a means for the user to feel the touch of water, a means for analyzing the user's emotions and adjusting the virtual experience according to the emotions, and a means for generating a conversation in real time using a generation AI model. This allows the user to realize his or her desires in the virtual world and obtain the optimal experience according to his or her emotions.

[0912] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[0913] "A means to virtually realize the user's regrets" is a technology that allows users to recreate in a virtual space the wishes and hopes that they were unable to realize in the real world.

[0914] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character or avatar with which the user can converse in a virtual space.

[0915] "Means to give users the feeling of returning to their hometown" refers to technology that allows users to experience the scenery and environment of their hometown within a virtual space.

[0916] "A means for users to go to places they imagine" refers to technology that allows users to recreate specific locations or scenery within a virtual space, providing the experience of being in that place.

[0917] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[0918] "Means of analyzing a user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the emotions of a user from their facial expressions, tone of voice, choice of words, etc., and adjusts the content of the virtual experience based on the results of that analysis.

[0919] "Means for generating conversation in real time using a generative AI model" means technology that uses an artificial intelligence model to generate natural conversation in real time between a user and a virtual character.

[0920] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0921] The present invention is a system for allowing users to virtually realize their desires in the metaverse. The purpose of this system is to allow users to fulfill their desires and emotions that cannot be realized in the real world in the virtual world. Specific embodiments are described below.

[0922] Hardware and Software Used

[0923] Hardware: VR devices (e.g. VR headsets), AR devices (e.g. AR glasses), haptic feedback devices (e.g. haptic gloves)

[0924] Software: Metaverse platforms (e.g. virtual reality engines), sentiment analysis engines (e.g. emotion recognition software), generative AI models (e.g. natural language generation models)

[0925] System Overview

[0926] The user wears a VR or AR device and accesses the system. The user communicates their wishes to the system through voice or text input. For example, the user can input a specific wish such as "I want to talk to my grandchildren."

[0927] The server uses a generative AI model to analyze the user's input. Based on the analysis results, it generates a virtual experience to realize the user's desire. For example, if a user inputs "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates a scenario in which the user can have a conversation in real time.

[0928] The generated virtual experience data is sent from the server to the user's device. The device provides the user with a virtual experience through a VR or AR device based on the received data. A haptic feedback device is used to allow the user to feel touch in the virtual environment.

[0929] In addition, the server uses an emotion analysis engine to analyze the user's emotions from facial expressions, tone of voice, choice of words, etc. Based on the analysis results, the virtual experience can be adjusted. For example, if the user is feeling joy, the server will provide an experience that amplifies that joy.

[0930] Examples of concrete examples and prompts

[0931] Example 1: A user types, "I want to talk to my grandchild." The server uses a generative AI model to generate an avatar of the grandchild and starts a conversation in real time.

[0932] Example prompt: "I want to talk to my grandson. His name is Taro."

[0933] Example 2: A user types, "I want to go back to my hometown." The server uses the metaverse platform to recreate the scenery of the user's hometown.

[0934] Example prompt: "Please recreate the scenery of your hometown. The setting is a small village in Hokkaido."

[0935] Example 3: A user inputs "I want to see a beautiful ocean." The server generates a beautiful ocean scene and uses a haptic feedback device to provide the sensation of water.

[0936] Example prompt: "Show me a beautiful ocean scene. If possible, I'd also like to hear the sound of the waves."

[0937] In this way, the system realizes the user's desires in the virtual world and provides an optimal experience according to their emotions.

[0938] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0939] Step 1:

[0940] The user accesses the system and inputs their wishes. The user wears a VR or AR device and communicates their wishes to the system through voice or text input. For example, the user might say, "I want to talk to my grandchildren." The input data is converted to text format and sent to the server.

[0941] Step 2:

[0942] The server analyzes the user's desires and generates an appropriate virtual experience. The server uses a generative AI model to analyze the user's input text. Based on the analysis results, a virtual experience is generated to realize the user's desire. For example, in response to the input "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates the content of the conversation. The generated virtual experience data is output as 3D model data for the avatar and a conversation script.

[0943] Step 3:

[0944] The server transmits the generated virtual experience to the terminal. The server transmits the generated virtual experience data to the user's terminal. The data includes the avatar's movements, conversation content, and virtual environment settings. The terminal prepares to display the virtual experience based on the received data.

[0945] Step 4:

[0946] The terminal displays the virtual experience and provides haptic feedback as necessary. The terminal provides the user with a virtual experience through a VR device or an AR device based on the received virtual experience data. The user starts a conversation with the grandchild's avatar. The haptic feedback device is used to allow the user to feel haptics in the virtual environment. For example, when the user touches the grandchild, the haptic feedback device vibrates to provide a haptic sensation.

[0947] Step 5:

[0948] The server analyzes the user's emotions and adjusts the virtual experience. Using an emotion analysis engine, the server analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc. The server adjusts the virtual experience based on the analysis results. For example, if the user is feeling happy, the content of the conversation and the avatar's actions are adjusted to amplify that joy. The input data for emotion analysis are the user's facial expressions and voice data, and the output is adjusted virtual experience data.

[0949] (Application example 2)

[0950] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0951] In modern society, the means by which users can realize their desires in the virtual world are limited. Although there are systems that adjust virtual experiences according to the user's emotions, there is still a lack of means to analyze the user's emotions in real time and provide an optimal experience in a virtual shopping experience. This makes it difficult for users to have a more satisfying virtual experience.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0953] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to converse with, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the virtual shopping experience based on the user's emotions. This allows the user to realize his / her desires in the virtual world and further obtain an optimal virtual shopping experience according to his / her emotions.

[0954] "Means of linking with the metaverse" refers to technology for communicating with the metaverse, a virtual reality space, and sending and receiving data.

[0955] "A means to virtually realize the user's regrets" is a technology that allows the user to recreate and experience their desires and regrets in a virtual space.

[0956] "A means for generating a person with whom the user can converse in front of the user" refers to technology for generating avatars or characters with which the user can converse in a virtual space.

[0957] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[0958] "A means to allow users to go to places they imagine" is a technology that generates a specific place imagined by the user in a virtual space, allowing the user to experience that place.

[0959] "A means for allowing users to feel the sensation of water" refers to technology that recreates the tactile sensation of water in a virtual space, providing users with the sensation of actually touching water.

[0960] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and changes the content of the virtual experience based on the results.

[0961] "Means for optimizing the virtual shopping experience based on user emotions" refers to technology that analyzes user emotions in real time and adjusts the content and delivery method of virtual shopping in response to those emotions.

[0962] The system for implementing the present invention includes a number of means for coordinating a virtual experience according to a user's emotions in cooperation with the metaverse. Specific embodiments are described below.

[0963] System Configuration

[0964] Hardware

[0965] Webcam: Used to capture the user's facial expressions.

[0966] Smartphone: A device that allows users to access the system and engage in the virtual experience.

[0967] Smart glasses: A device for providing an augmented reality (AR) experience.

[0968] Head-mounted display: A device for providing a virtual reality (VR) experience.

[0969] software

[0970] OpenCV: An image processing library used to analyze the user's facial expressions.

[0971] EmotionRecognizer: An emotion recognition library used to analyze user emotions in real time.

[0972] MetaverseAPI: An API for linking with the metaverse and providing virtual experiences.

[0973] Data processing and calculation

[0974] Image capture and emotion recognition

[0975] The server captures the user's face image in real time using a webcam, which is pre-processed using OpenCV and analyzes the user's emotions using the EmotionRecognizer library.

[0976] Coordinating Virtual Experiences

[0977] The server uses the Metaverse API to tailor the content of the virtual experience based on the analyzed emotion data. For example, if the user is recognized as "happy," the system will display special discounts and recommended products to make the virtual shopping experience more enjoyable.

[0978] Examples

[0979] When a user is enjoying shopping in a virtual store: The emotion engine analyzes the user's facial expressions and if it determines that the user is enjoying themselves, the system will display special discounts and recommended products.

[0980] If the user is tired: If the emotion engine analyzes the user's facial expressions and determines that the user is tired, the system will provide relaxing music and interactive guidance.

[0981] Examples of prompt statements

[0982] When a user is enjoying shopping in a virtual store, the emotion engine will analyze the user's facial expressions. If the system determines that the user is enjoying themselves, it will display special discounts and recommended products. If the system determines that the user is tired, it will provide relaxing music and interactive guidance.

[0983] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[0984] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0985] Step 1:

[0986] The server captures the user's facial image in real time using a webcam. The input is the video data from the webcam, and the output is the captured facial image, which is used in subsequent processing steps.

[0987] Step 2:

[0988] The server preprocesses the captured face image using OpenCV. The input is the face image obtained in step 1, and the output is the preprocessed face image. Specific operations include face detection, image resizing, noise removal, etc.

[0989] Step 3:

[0990] The server analyzes the preprocessed face image using the EmotionRecognizer library to recognize the user's emotions. The input is the preprocessed face image obtained in step 2, and the output is the recognized emotion data. Specific operations include extracting facial feature points, inputting them into the emotion model, and classifying emotions.

[0991] Step 4:

[0992] The server adjusts the content of the virtual experience using the Metaverse API based on the recognized emotion data. The input is the emotion data obtained in step 3, and the output is the adjusted virtual experience settings. Specific operations include selecting a scenario for the virtual experience according to the emotion data, placing virtual objects, setting interactions, etc.

[0993] Step 5:

[0994] The terminal receives the adjusted virtual experience settings sent from the server and provides them to the user. The input is the virtual experience setting data obtained in step 4, and the output is the virtual environment experienced by the user. Specific operations include rendering the virtual environment, displaying a user interface, and executing interactions.

[0995] Step 6:

[0996] The user enjoys a virtual experience provided through the device. The input is the virtual environment provided by the device, and the output is the user's experience and changes in emotions. Specific actions include movement within the virtual environment, interaction with objects, and emotional feedback.

[0997] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[0998] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0999] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1000] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1001] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1002] [Third embodiment]

[1003] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1004] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1005] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1006] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1007] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1008] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1009] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1010] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1011] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1012] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1013] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1014] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1015] "Example 1"

[1016] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[1017] "Example 2"

[1018] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[1019] The process flow of each embodiment will be described below.

[1020] "Example 1"

[1021] Step 1: The user starts the system and inputs their desires. For example, they can input their desires, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean," in text or voice.

[1022] Step 2: The system analyzes the input desire and generates a corresponding virtual experience. For example, if the desire is "I want to talk to my grandchild," the system will generate an avatar of the user's grandchild and enable real-time conversation.

[1023] Step 3: The system provides the generated virtual experience to the user. The user can experience the virtual experience through a VR or AR device. For example, if the scenery of the user's hometown is reproduced, the user can feel as if they are there. Step 4: The system monitors the user's response and adjusts the virtual experience as necessary. For example, if the user responds that they would like to feel the touch of water while looking at a seascape, the system utilizes a means of providing haptic feedback to provide the user with the touch of water.

[1024] Example 1

[1025] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1026] In modern society, many users face the challenge of finding it difficult to realize their regrets due to physical and time constraints. For example, it is difficult to talk to far-away family members or experience the feeling of returning to one's hometown. It is also difficult for users to go to places they imagine or experience real sensations through touch. To solve these problems, a means is needed to realize users' wishes using the virtual world.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1028] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for analyzing the user's wishes, a means for generating a virtual experience, and a means for setting haptic feedback. This allows the user to go beyond physical constraints and experience their wishes in a realistic way in the virtual world.

[1029] The "metaverse" is a virtual world built on the Internet, a virtual environment in which users can interact with other users through avatars and engage in various activities.

[1030] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in the virtual world the desires or experiences that they were unable to realize in the real world.

[1031] "Means for generating a person with whom the user can converse in front of the user" refers to a technology or method for generating an avatar of the person with whom the user wants to converse in the virtual world and simulating a conversation in real time.

[1032] "Means of giving the user the feeling of returning to their hometown" refers to technologies or methods that recreate the scenery or environment of the user's hometown in a virtual world, giving the user the feeling of being in that place.

[1033] "Means that allow users to go to places they imagine" refers to technologies and methods that recreate the places that users imagine they want to go to in the virtual world and allow users to access those places.

[1034] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when interacting with water in the virtual world.

[1035] "Means for analyzing user wishes" refers to techniques or methods for analyzing the wishes entered by the user, understanding their content, and extracting the necessary information.

[1036] A "means for generating a virtual experience" is a technology or method for generating an experience in a virtual world based on a user's desires.

[1037] "Means for configuring haptic feedback" means techniques or methods for configuring and controlling a haptic feedback device to make a user's experience in a virtual world more realistic.

[1038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1039] The present invention is a system for enabling a user to realize his / her own desires in a virtual world. A specific embodiment of this system will be described below.

[1040] System Configuration

[1041] This system consists of three main elements: a server, a terminal, and a user. The server works in conjunction with the metaverse to analyze users' desires and generate virtual experiences. The terminal provides an interface for users to access the system and enjoy the virtual experience. Users input their desires through the terminal and experience the virtual experience.

[1042] Hardware and Software Used

[1043] Server: A server with high-performance computing power is required, preferably one with the computing resources to run generative AI models (e.g., OpenAI's GPT-4).

[1044] Terminal: The device through which the user accesses the device, which may be a PC, tablet, smartphone, etc. Also includes haptic feedback devices (e.g. HaptX gloves).

[1045] Software: Game engines (e.g. Unity, Unreal Engine) are used to build virtual worlds, and generative AI models are used to perform natural language processing.

[1046] Program Processing

[1047] The server receives the user's desire and analyzes it. It uses a generative AI model to understand the user's input and extract the necessary information. The server then generates the virtual experience. Specifically, it generates an avatar of the person the user wants to talk to and simulates a real-time conversation. It also uses a game engine to build a realistic 3D environment to recreate the user's hometown or the scenery they want to see. It also controls a haptic feedback device and configures it to make the user's experience in the virtual world feel more realistic.

[1048] Examples

[1049] If a user wishes to have a conversation with their grandchild, the specific steps that are taken are as follows:

[1050] 1. User's wish input: The user inputs "I want to talk to my grandson. His grandson's name is Taro, and he is 10 years old" into the input field of the terminal.

[1051] 2. Wish analysis: The server uses a generative AI model to analyze the wish “I want to talk to my grandchild” and extract details such as the grandchild’s name and age.

[1052] 3. Generating the virtual experience: The server uses a generative AI model to generate an avatar of the user's 10-year-old grandson, Taro, and simulates a real-time conversation with him. It also uses Unity to create a 3D recreation of the user's hometown.

[1053] 4. Setting up haptic feedback: The server controls the HaptX gloves and configures them so that the user can feel the sensation of water in the virtual world.

[1054] 5. Providing the virtual experience: The server provides the generated virtual experience to the user. The user can interact with the grandchild’s avatar through the terminal display and feel the sensation of water through the HaptX gloves.

[1055] Examples of prompt statements

[1056] "I want to have a conversation with my grandson. His name is Taro and he is 10 years old."

[1057] "I want to see the scenery of my hometown. It's a small village in the north."

[1058] "I want to see a beautiful ocean. Preferably one like the ocean in the south."

[1059] This allows users to transcend physical constraints and experience their desires in a realistic way in the virtual world.

[1060] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1061] Step 1:

[1062] The user uses the terminal to input his / her wish. For example, he / she inputs, "I want to talk to my grandson. His / her name is Taro, and he / she is 10 years old." This input is sent to the server.

[1063] Step 2:

[1064] The server analyzes the desire received from the user. The input text data is passed to the generative AI model, which understands the user's desire and extracts the necessary information. Specifically, it analyzes the desire "I want to talk to my grandchild" and the detailed information "My grandchild's name is Taro and he is 10 years old." The analysis result is passed to the next processing step.

[1065] Step 3:

[1066] The server generates the virtual experience based on the analysis results. First, it uses a generative AI model to generate an avatar of the person the user wants to talk to. Specifically, it generates an avatar of the user's 10-year-old grandson, "Taro," and simulates a real-time conversation. The data for this avatar generation is passed to the next processing step.

[1067] Step 4:

[1068] The server uses a game engine (e.g. Unity) to create a realistic 3D environment to recreate the user's hometown or scenery they want to see. For example, if a user inputs "I want to see the scenery of my hometown," a 3D model is generated based on the scenery data of that place. This 3D environment data is passed to the next processing step.

[1069] Step 5:

[1070] The server controls the haptic feedback device (e.g., HaptX gloves) and configures it to make the user's experience in the virtual world feel more realistic. For example, if the user inputs "I want to see a beautiful ocean," the server configures the haptic feedback device to reproduce the feeling of water. This configuration data is passed to the next processing step.

[1071] Step 6:

[1072] The server provides the generated virtual experience to the user. The user interacts with the generated avatar through the display of the terminal and experiences the 3D environment. The user can also feel the touch of water through a haptic feedback device. The data of this virtual experience is sent to the user, and the user can enjoy the experience in the virtual world.

[1073] (Application example 1)

[1074] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1075] In modern society, users need to have realistic experiences in virtual spaces without having to physically move around. However, current systems have difficulty providing sufficient tactile and visual reality when users try on or use products in virtual spaces. In addition, there is a lack of real-time interaction and haptic feedback when users virtually interact with specific places or people. This limits the user's experience and reduces satisfaction.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1077] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to try on or use products in a virtual store, a means for generating an avatar for the user, a means for generating a scene of the virtual store, and a means for providing haptic feedback. This allows the user to have a real experience in the virtual space.

[1078] The "metaverse" is an interactive digital environment built in a virtual space where users can interact with other users and content through avatars.

[1079] "A means to virtually realize the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, allowing the user to experience them.

[1080] "A means for generating a person with whom the user can converse in front of the user" refers to technology that generates an avatar or character with which the user can converse in real time in a virtual space.

[1081] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[1082] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users in a virtual space and provides the user with the experience of visiting those places.

[1083] "Means allowing the user to feel the sensation of water" refers to technology that allows the user to sense the presence of water in a virtual space through haptic feedback.

[1084] "Means for users to try on or use products in a virtual store" refers to technology that provides users with the experience of trying on or using products in a store within a virtual space.

[1085] "Means for generating a user's avatar" refers to technology for generating an avatar for use in a virtual space based on the user's appearance and characteristics.

[1086] "Means for generating a virtual store scene" refers to technology for reproducing the environment and scenery of a store within a virtual space.

[1087] A "means for providing haptic feedback" is a haptic device or technology that enables a user to physically feel their virtual experience.

[1088] In order to implement this invention, the following hardware and software are used. The hardware requires a smartphone and a haptic feedback device. The software requires the OpenAI API and Virtual Reality SDK.

[1089] The server provides a means to link with the metaverse and uses generative AI models to generate data to virtually realize the user's wishes. For example, if a user wants to talk to a specific person, an avatar of that person can be generated and the user can talk to them in real time.

[1090] The terminal provides an interface for the user to try on or try on products in the virtual store. The user can access the virtual store using a smartphone, select products, and try them on or try them on. For example, if the user wants to try on clothes, the terminal can generate an avatar of the user and dress the avatar in the selected clothes. In addition, by using a haptic feedback device, the user can actually feel the texture of the clothes.

[1091] If a user wants to go to a specific place, the server generates a scene of that place and gives the user the feeling that he or she is there. For example, if a user wants to see the sea, the server generates a scene of the sea that the user imagines and allows the user to go to that place. In this case, a haptic feedback device can be used to provide the feeling of water.

[1092] As an example, the following prompt can be used:

[1093] Avatar generation: "Generate an avatar for the user with the following data: name, height, weight"

[1094] Generate a virtual store with the following data: store name, location

[1095] Try on product: "Apply the following product to the avatar: product type, color, size on user data"

[1096] This allows users to have a realistic experience in a virtual space.

[1097] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1098] Step 1:

[1099] The server receives a user request. As input, it contains the user's desired experience (e.g., talking to a specific person, visiting a specific place). The server analyzes this request and sends the necessary data to the generative AI model. As output, it gets the analysis result.

[1100] Step 2:

[1101] The server uses a generative AI model to generate avatars and scenery based on the user's request. The input includes the user's feature data and details of the desired experience. The generative AI model processes these data and generates avatar and scenery data. The output is the generated avatar and scenery data.

[1102] Step 3:

[1103] The server sends the generated avatar and scenery data to the terminal. The input contains the generated avatar and scenery data. The server converts these data into an appropriate format and sends it to the terminal. The output is the data sent to the terminal.

[1104] Step 4:

[1105] The terminal constructs a virtual space based on the received data. The input includes avatar and scenery data sent from the server. The terminal renders this data and displays it to the user. The output is the virtual space displayed to the user.

[1106] Step 5:

[1107] The user begins to experience the virtual space through the device. The input includes the virtual space displayed on the device. The user manipulates an avatar to talk to specific people or visit specific places. The output is the result of the user's manipulation.

[1108] Step 6:

[1109] The terminal transmits the user's operation to the server. The input includes the user's operation data. The terminal transmits these data to the server to realize real-time interaction. The output is the operation data transmitted to the server.

[1110] Step 7:

[1111] The server updates the virtual space based on the user's operation data. The input includes the user's operation data. The server analyzes these data and updates the state of the virtual space. The output is the updated data of the virtual space.

[1112] Step 8:

[1113] The server sends the updated virtual space data to the terminal. As input, the updated virtual space data is included. The server converts these data into an appropriate format and sends them to the terminal. As output, the updated data sent to the terminal is obtained.

[1114] Step 9:

[1115] The terminal displays the updated virtual space to the user. As input, it contains the updated data sent from the server. The terminal renders these data and displays them to the user. As output, it obtains the updated virtual space displayed to the user.

[1116] Step 10:

[1117] The user physically feels the experience in the virtual space using the haptic feedback device. The input includes haptic feedback data sent from the terminal, which allows the user to feel the experience in the virtual space more realistically. The output is the user's haptic experience.

[1118] Example 2

[1119] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1120] In modern society, many users want to recreate their past memories and regrets in virtual space. However, current systems have difficulty realizing users' specific desires in real time, and are unable to provide sensory experiences such as haptic feedback. For this reason, there is a demand for systems that allow users to experience virtual space in a more realistic way.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1122] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the user's regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to their hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for receiving the user's input and generating a prompt sentence using a generative AI model, a means for generating a virtual space using virtual space generation software, and a means for providing haptic feedback using a haptic feedback device. This allows the user to realize their specific desires in the virtual space in real time and further obtain a more realistic experience through haptic feedback.

[1123] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[1124] "A means of virtually realizing the things that users regret" refers to technology that enables users to recreate in a virtual space things or experiences from the past that they feel nostalgic about.

[1125] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character with which the user can converse in a virtual space.

[1126] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[1127] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users within a virtual space, allowing the users to visit those places.

[1128] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[1129] "Means for receiving user input and generating a prompt sentence using a generative AI model" refers to technology that analyzes information entered by the user and generates an appropriate prompt sentence using a generative AI model.

[1130] "Means for generating a virtual space using virtual space generation software" refers to a technology that uses software for constructing a virtual space to generate a virtual space in response to a user's request.

[1131] "Means for providing haptic feedback using a haptic feedback device" refers to technology that uses a device to actually reproduce the sensation when a user touches a specific object or environment in a virtual space.

[1132] The present invention is a system that allows users to realize their wishes in a virtual space. This system works in conjunction with the metaverse and provides multiple means for users to virtually realize their wishes.

[1133] Hardware and Software Used

[1134] Hardware: Terminals (PC, smartphone, VR device), servers, haptic feedback devices (e.g. HaptX gloves)

[1135] Software: Generative AI models (e.g. GPT-4), virtual world generation software (e.g. Unity, Unreal Engine)

[1136] System Configuration

[1137] 1. User input of wishes

[1138] The user uses the terminal to input their own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[1139] 2. Data collection and analysis

[1140] The server receives user input and uses a generative AI model to generate prompts. For example, if a user wants to talk to their grandchild, it generates an avatar of the grandchild and generates prompts that enable real-time conversation.

[1141] 3. Creation of Virtual Space

[1142] The server uses virtual space generation software to generate a virtual space based on the user's wishes, such as "recreating the scenery of the user's hometown" or "generating the seascape that the user imagines."

[1143] 4. Avatar Creation and Conversation

[1144] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[1145] 5. Providing haptic feedback

[1146] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[1147] Examples

[1148] Example 1: I want to talk to my grandchildren

[1149] The user types into the terminal, "I want to talk to my grandchild."

[1150] The server uses the generative AI model to generate "prompts that generate an avatar of the user's grandchild and enable real-time conversation."

[1151] The server uses virtual space generation software to generate an avatar of the grandchild and conduct the conversation. The grandchild's avatar says, "Hello, Grandpa! What would you like to talk about today?"

[1152] Example 2: I want to go back to my hometown

[1153] The user types "I want to go back to my hometown" into the terminal.

[1154] The server uses a generative AI model to generate "prompt sentences that recreate the scenery of the user's hometown."

[1155] The server uses virtual space generation software to recreate the scenery of the person's hometown, allowing the user to walk around the scenery in the virtual space.

[1156] Example 3: I want to see a beautiful ocean

[1157] The user types "I want to see a beautiful ocean" into the terminal.

[1158] The server uses a generative AI model to generate "prompt sentences that generate the seascape the user imagines and provide haptic feedback."

[1159] The server uses virtual space generation software to generate the ocean scenery and a haptic feedback device to provide the feel of water: when a user places their hand in the virtual ocean, the HaptX gloves make them feel the cold and flowing water.

[1160] The system allows users to realize their specific desires in a virtual space in real time, and provides a more realistic experience through haptic feedback.

[1161] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1162] Step 1:

[1163] User's wish input

[1164] The user uses the terminal to input his / her own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[1165] Input: User's wish (in text format)

[1166] Output: User wish data

[1167] Specific actions: The user inputs their wish in text form using the device's keyboard or voice input function. For example, the user inputs "I want to talk to my grandchildren" into the input form displayed on the device screen.

[1168] Step 2:

[1169] Data collection and analysis

[1170] The server receives the user's input and generates a prompt using a generative AI model.

[1171] Input: User's wish data

[1172] Output: Prompt statement

[1173] Specific operation: The server analyzes the user's input data and instructs the generative AI model to generate "a prompt sentence that generates an avatar of the grandchild and enables real-time conversation if the user wants to talk to the grandchild."

[1174] Step 3:

[1175] Creating a Virtual Space

[1176] The server uses virtual space generation software to generate a virtual space based on the user's wishes.

[1177] Input: prompt statement

[1178] Output: Virtual space data

[1179] Specific operation: The server issues instructions to the virtual space generation software, such as "recreate the scenery of the user's hometown" or "generate the seascape that the user imagines," and constructs the virtual space.

[1180] Step 4:

[1181] Avatar creation and conversation

[1182] The server generates an avatar of the user's grandchild and enables real-time conversation, using generative AI models to make the conversation natural.

[1183] Input: Prompt text, virtual space data

[1184] Output: Avatar, conversation data

[1185] How it works: The server instructs the generative AI model to generate an avatar of the user's grandchild, and then applies a dialogue model to enable real-time conversation. For example, the grandchild's avatar might say, "Hello, Grandpa! What would you like to talk about today?"

[1186] Step 5:

[1187] Providing haptic feedback

[1188] If the user has a desire to see a beautiful ocean, the server uses a haptic feedback device to provide the user with the feel of water.

[1189] Input: Virtual space data, user operation data

[1190] Output: Haptic feedback data

[1191] Specific operation: The server instructs the haptic feedback device to reproduce the feel of water when the user touches the ocean in the virtual space. For example, when the user puts his or her hand in the ocean in the virtual space, the haptic feedback device makes the user feel the coldness and flow of the water.

[1192] Through these steps, users can realize their specific desires in the virtual space in real time, and get a more realistic experience through haptic feedback.

[1193] (Application example 2)

[1194] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1195] In modern society, users are required to have realistic experiences in virtual spaces without having to physically move. However, conventional systems have been unable to fully satisfy users' desires, and in particular, in the shopping experience in virtual stores, it was difficult to have a real-time conversation with an assistant or feel the texture of the products. This resulted in low user satisfaction and made the experience in virtual spaces inferior to reality.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1197] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to select a product while talking to an assistant avatar in real time when enjoying shopping in a virtual store, a means for the user to walk around in a virtual store that reproduces a specific place or scenery, and a means for the user to feel the texture of the product through haptic feedback. This allows the user to enjoy a real shopping experience in a virtual space.

[1198] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[1199] "A means of virtually realizing the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, making the user feel as if they are actually experiencing them.

[1200] "A means for generating a person with whom a user can converse in front of the user" refers to a technology that enables a user to generate an avatar of a person with whom the user wants to converse in a virtual space, thereby enabling the user to converse in real time.

[1201] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[1202] "A means to go to places that users imagine" is a technology that generates the places that users imagine they would like to go to in a virtual space and provides the user with the experience of visiting those places.

[1203] "A means for allowing the user to feel the sensation of water" refers to technology that recreates the presence of water in a virtual space and allows the user to feel the sensation of that water through haptic feedback.

[1204] "A means that enables users to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store" refers to technology that enables users to select products while conversing in real time with an assistant avatar when shopping in a virtual store.

[1205] "A means for enabling users to walk around a virtual store that recreates a specific place or scenery" refers to technology that enables users to move freely within a virtual store and experience an environment that recreates a specific place or scenery.

[1206] "Means for allowing a user to feel the texture of a product through haptic feedback" refers to technology that allows a user to feel the texture of a product through a haptic feedback device when picking up the product in a virtual space.

[1207] A system for implementing this invention includes means for linking with the metaverse, means for virtually realizing a user's regrets, means for generating a person for the user to converse with in front of them, means for giving the user the feeling of returning to their hometown, means for allowing the user to go to places the user imagines, means for the user to feel the feel of water, means for a user to select products while conversing in real time with an assistant avatar when enjoying shopping in a virtual store, means for a user to walk around a virtual store that recreates a specific place or scenery, and means for the user to feel the texture of products through haptic feedback.

[1208] System Program Overview

[1209] The server uses the OpenAI API, the virtual environment library (virtual_environment), and the haptic feedback library (haptic_feedback) to process and calculate data to realize the user's wishes. Specifically, the server generates prompts based on the user's wishes, generates avatars and virtual environments using generative AI models, and provides the user with a realistic sensation through a haptic feedback device.

[1210] Processing flow

[1211] 1. Generate prompts based on user desires

[1212] The server describes the user's wishes in natural language and inputs them into the generative AI model. For example, if the user wishes to "talk to their grandchildren," the server generates a prompt sentence like this:

[1213] The user wants to experience conversation with his / her grandchild.

[1214] 2. Avatar Creation

[1215] The server uses the OpenAI API to generate an avatar based on the user's wishes, and the generated avatar is then used by the user to converse in real time in a virtual space.

[1216] 3. Setting up the virtual environment

[1217] The server uses the virtual environment library to set up a virtual store for the user to experience, and the user can freely walk around the virtual store.

[1218] 4. Haptic feedback settings

[1219] The server uses a haptic feedback library to allow the user to feel the texture of the product, so that when the user picks up the product in the virtual space, they can feel the texture of the product in a realistic way.

[1220] Examples

[1221] For example, if a user has a wish to "talk to his / her grandchild," the server will generate an avatar of the grandchild and enable the user to talk to the grandchild in real time in the virtual store. In addition, when the user picks up a particular product, the user can feel the texture of the product through a haptic feedback device.

[1222] In this way, users can enjoy a real shopping experience in a virtual space.

[1223] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1224] Step 1:

[1225] The server generates a prompt based on the user's desire. If the user has the desire "I want to talk to my grandchildren," the server takes this information as input and generates a prompt like this:

[1226] The user wants to experience conversation with his / her grandchild.

[1227] This prompt is then fed into a generative AI model.

[1228] Step 2:

[1229] The server uses the generative AI model to generate an avatar based on the prompt sentence. Specifically, it calls the OpenAI API and provides the prompt sentence as input. The API analyzes the prompt sentence and generates an avatar of the user's grandchild. This avatar is used by the user to converse in real time in the virtual space. The generated avatar is sent from the server to the user's device.

[1230] Step 3:

[1231] The server uses the virtual environment library to set up a virtual store for the user to experience. If the user has a desire to "enjoy shopping in a virtual store," the server receives this information as input and calls the virtual environment library to generate a virtual store. The generated virtual store is sent to the user's terminal, and the user can freely walk around the virtual store.

[1232] Step 4:

[1233] The server uses the haptic feedback library to allow the user to feel the texture of the product. When the user picks up a particular product, the information is sent to the server. The server calls the haptic feedback library to generate data about the texture of the product. This data is sent to the user's haptic feedback device, allowing the user to feel the texture of the product in a realistic way.

[1234] Step 5:

[1235] Users can walk around the virtual store and select products while talking to the generated avatar in real time. When the user selects a product, the information is sent to the server, and the server sends detailed information about the selected product to the user's device. Based on this information, the user can decide whether to select more products or purchase them.

[1236] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1237] "Example 1"

[1238] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[1239] "Example 2"

[1240] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system has the ability to recognize the user's emotions and adjust the virtual experience according to the emotions. Specifically, while the user is experiencing a virtual experience through a VR device or an AR device, the emotion engine analyzes the user's emotions from the user's facial expressions, tone of voice, choice of words, etc. Then, the virtual experience is adjusted based on the analysis results. For example, if the user feels joy, the system provides a virtual experience that amplifies the joy. Conversely, if the user feels sadness, the system provides a virtual experience that alleviates the sadness. In this way, the system incorporating an emotion engine can optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience.

[1241] The process flow of each embodiment will be described below.

[1242] "Example 1"

[1243] Step 1: The user connects to the system through a VR or AR device.

[1244] Step 2: The user inputs their desired virtual experience into the system.

[1245] Step 3: The system uses an emotion engine to analyze the user's emotions based on facial expressions, tone of voice, choice of words, etc.

[1246] Step 4: The system tailors the virtual experience based on the analysis results and provides it to the user.

[1247] Step 5: The user experiences the provided virtual experience.

[1248] Step 6: The system monitors the user's reactions and adjusts the virtual experience if necessary.

[1249] Example 1

[1250] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1251] In modern society, there is a demand for a way to allow users to realize their regrets in the virtual world. In particular, it is important to realize experiences that are difficult to experience in the real world, such as talking with far-away family members, returning to one's hometown, and experiencing natural environments. There is also a demand to provide a more realistic and satisfying experience by analyzing the user's emotions and adjusting the virtual experience according to those emotions.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place imagined by the user, a means for the user to feel the touch of water, a means for analyzing the user's emotions, a means for adjusting the virtual experience based on the analysis result, and a means for generating a virtual experience using a generation AI model. This allows the user to realize his / her desires in the virtual world and enjoy the optimal virtual experience according to his / her emotions.

[1253] The "metaverse" is a virtual space built on the Internet, which is a virtual environment in which users can interact with other users through avatars and carry out various activities.

[1254] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in a virtual space the desires or experiences that they were unable to realize in the real world.

[1255] "Means for generating a person with whom the user can converse in front of the user" refers to a technique or method for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[1256] "Means of giving users the feeling of returning to their hometown" refers to technologies and methods that allow users to recreate the scenery and environment of their hometown in a virtual space, giving them the feeling as if they were there.

[1257] "Means that allow users to go to places they imagine" refers to technologies or methods that allow users to recreate specific locations or environments in a virtual space, providing the experience of being in that place.

[1258] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when touching water in a virtual space.

[1259] "Means of analyzing user emotions" refers to techniques or methods for analyzing emotions from a user's facial expressions, tone of voice, choice of words, etc.

[1260] "Means for adjusting the virtual experience based on the analysis results" refers to technologies and methods for optimizing the experience in the virtual space based on the results of an analysis of the user's emotions, and providing an experience that corresponds to the user's emotions.

[1261] A "means for generating a virtual experience using a generative AI model" is a technology or method that uses an artificial intelligence model to generate experiential scenarios or content in a virtual space based on user input data.

[1262] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1263] This invention is a system for realizing a user's regrets in a virtual world, and provides a technology that works in conjunction with the metaverse to analyze the user's emotions and optimize the virtual experience. A specific embodiment of this system will be described below.

[1264] System Configuration

[1265] The system consists of the following main components:

[1266] 1. User device:

[1267] A device that allows a user to access the system, including PCs, smartphones, tablets, VR devices, AR devices, etc.

[1268] Users access the system through a dedicated application or web interface and enter their wishes.

[1269] 2. Server:

[1270] It takes user input data and uses a generative AI model to generate a virtual experience.

[1271] An emotion engine is used to analyze the user's emotions and tailor the virtual experience based on the analysis results.

[1272] 3. Generative AI model:

[1273] As an example, use a natural language processing model such as OpenAI's GPT-4.

[1274] Generates virtual experience scenarios and content based on user input data.

[1275] 4. Emotion Engine:

[1276] It includes algorithms that analyze emotions from users' facial expressions, tone of voice, choice of words, etc.

[1277] The content of the virtual experience is adjusted based on the analysis results.

[1278] System Operation

[1279] 1. User inputs their wishes:

[1280] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[1281] 2. Data analysis and generation:

[1282] The server receives the user's input data and generates the virtual experience using a generative AI model.

[1283] For example, if a user types, "I want to talk to my grandchild," the server will generate an avatar of the user's grandchild and enable a real-time conversation.

[1284] 3. Sentiment analysis using emotion engine:

[1285] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[1286] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[1287] The content of the virtual experience is adjusted based on the analysis results.

[1288] 4. Providing Virtual Experiences:

[1289] The server streams the generated virtual experience data to the user's device.

[1290] Users put on a VR headset or AR glasses and begin the virtual experience.

[1291] For example, a user could put on a VR headset and enjoy a real-time conversation with an avatar of their grandchild.

[1292] Examples and prompts

[1293] As a concrete example, a process will be described in which the user has a desire to "talk to his / her grandchild."

[1294] 1. User input:

[1295] The user types into the terminal, "I want to talk to my grandchild."

[1296] 2. Data analysis and generation:

[1297] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an avatar of the user's grandchild.

[1298] 3. How the Emotion Engine works:

[1299] The emotion engine analyzes the user's facial expressions and tone of voice to determine whether the user is feeling joy.

[1300] 4. Providing Virtual Experiences:

[1301] The server then transmits the real-time conversation between the user and the avatar of the grandchild to the user's device, and the user experiences the conversation with the grandchild through a VR device.

[1302] Example prompt:

[1303] If a user types, "I want to talk to my grandchild," generate a program that creates an avatar of the user's grandchild and enables a real-time conversation.

[1304] In this way, a system is constructed that realizes users' wishes in the virtual world.

[1305] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1306] System program processing flow

[1307] Step 1: User inputs their wishes

[1308] Users input their wishes into the system through a terminal, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[1309] Input: The user's desire (e.g., "I want to talk to my grandchildren")

[1310] Output: User wish data

[1311] Specific behavior:

[1312] Users enter their wishes into a text box using a dedicated application or a web interface.

[1313] For example, type "I want to talk to my grandchildren."

[1314] Step 2: Analyze and generate data

[1315] The server receives the user's input data and generates the virtual experience using a generative AI model.

[1316] Input: User's wish data

[1317] Output: Virtual experience scenario

[1318] Specific behavior:

[1319] The server sends the user's input to a natural language processing engine, which analyzes the input.

[1320] Based on the analysis results, a generative AI model (e.g., OpenAI's GPT-4) is invoked to generate a virtual experience scenario.

[1321] Example: In response to the input "I want to talk to my grandchild," an avatar of the grandchild is generated and a conversation scenario is created.

[1322] Step 3: Sentiment analysis using the emotion engine

[1323] The server analyzes the user's emotions using an emotion engine, which analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1324] Input: User's camera video and audio data

[1325] Output: Emotion analysis results

[1326] Specific behavior:

[1327] The server collects the user's camera footage and audio data in real time and transmits it to the emotion engine.

[1328] The emotion engine uses facial expression recognition and voice analysis algorithms to analyze the user's emotions.

[1329] The content of the virtual experience is adjusted based on the analysis results.

[1330] Example: If the user is smiling, the system will provide conversational content that amplifies the joy.

[1331] Step 4: Generate the virtual experience

[1332] The server generates an optimized virtual experience based on the results of the emotion analysis, which is tailored to the user's desires and emotions.

[1333] Input: Sentiment analysis results, virtual experience scenario

[1334] Output: Optimized virtual experience data

[1335] Specific behavior:

[1336] The server integrates the results of the generative AI model and the emotion engine to create an optimal virtual experience scenario.

[1337] Example: If a user types "I want to talk to my grandchild," and sentiment analysis indicates joy, the grandchild's avatar can be set to offer happy topics.

[1338] Step 5: Offer virtual experiences

[1339] The server transmits the generated virtual experience to the user's device, where the user experiences the virtual experience through a VR or AR device.

[1340] Input: Optimized virtual experience data

[1341] Output: The user's virtual experience

[1342] Specific behavior:

[1343] The server streams the generated virtual experience data to the user's device.

[1344] Users put on a VR headset or AR glasses and begin the virtual experience.

[1345] Example: A user puts on a VR headset and enjoys a real-time conversation with an avatar of their grandchild.

[1346] In this way, a system is constructed that realizes users' wishes in the virtual world.

[1347] (Application example 1)

[1348] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1349] In modern society, users are expected to realize their desires in virtual space without moving physically. In addition, it is a challenge to optimize the virtual experience according to the user's emotions and provide a more realistic and satisfying experience. In particular, it is important to adjust the shopping experience based on the user's emotions in virtual stores.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1351] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the experience of the virtual store based on the user's emotions. This allows the user to realize his / her desires in the virtual space without physical constraints and enjoy the optimal experience according to his / her emotions.

[1352] "Means of linking with the metaverse" refers to technology that exchanges data with the metaverse, a virtual space, and provides an interface that allows users to have virtual experiences.

[1353] "A means to virtually realize the user's regrets" is a technology that enables users to recreate in a virtual space the desires and experiences that they were unable to realize in the real world.

[1354] "Means for generating a person with whom the user can converse in front of the user" refers to a technology for generating an avatar of a specific person or character so that the user can converse in a virtual space.

[1355] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[1356] "A means to go to places the user imagines" refers to technology that generates specific places or landscapes imagined by the user in a virtual space, allowing the user to access those places.

[1357] "A means for allowing the user to feel the sensation of water" refers to haptic technology that provides the user with feedback on the actual sensation of water when they touch water in a virtual space.

[1358] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and dynamically changes the content of the virtual experience based on the results.

[1359] The "means for optimizing the virtual store experience based on the user's emotions" is a technology that analyzes the user's emotional state and adjusts the shopping experience in the virtual store according to that state.

[1360] The system for implementing the present invention aims to enable a user to realize his / her desires in a virtual space and enjoy an optimal experience according to his / her emotions. A specific embodiment of the system is described below.

[1361] System Configuration

[1362] The system consists of the following main components:

[1363] 1. Metaverse Linkage Module

[1364] It exchanges data with the metaverse and provides an interface for users to have virtual experiences. Specifically, it works with the metaverse platform through APIs.

[1365] 2. Virtual experience generation module

[1366] This is a module that allows users to virtually realize their regrets. For example, if a user wants to talk to their grandchild, it will generate an avatar of the grandchild and enable a real-time conversation.

[1367] 3. Emotion Recognition Module

[1368] This is a module for analyzing emotions from the user's facial expressions, tone of voice, etc. Specifically, it uses libraries such as OpenCV and EmotionRecognizer.

[1369] 4. Virtual Experience Tuning Module

[1370] This module dynamically changes the content of the virtual experience based on the emotion data obtained from the emotion recognition module. For example, if the user is feeling joy, it will provide an experience that amplifies that joy.

[1371] 5. Virtual store optimization module

[1372] This module tailors the virtual store shopping experience based on the user's emotions, for example highlighting new or limited edition products if the user is excited.

[1373] Hardware and Software Used

[1374] Hardware: Webcam, Head Mounted Display

[1375] Software: OpenCV (image processing library), EmotionRecognizer (emotion recognition library), VirtualStore (virtual store management library)

[1376] Processing flow

[1377] 1. User emotion recognition

[1378] It uses a webcam to capture the user's face and analyzes emotions using the EmotionRecognizer library.

[1379] 2. Creating and adjusting virtual experiences

[1380] Based on the emotion data obtained from the emotion recognition module, the virtual experience generation module realizes the user's desire in the virtual space. Furthermore, the virtual experience adjustment module dynamically changes the experience content according to the emotion.

[1381] 3. Virtual store optimization

[1382] The virtual store optimization module tailors the shopping experience based on user emotions, for example highlighting new or limited edition products if the user is excited.

[1383] Examples

[1384] When a user is shopping in a virtual store, if the system detects "excitement" in the user's facial expression, it will highlight new or limited edition products to further increase the user's excitement.

[1385] Examples of prompt statements

[1386] "When a user is shopping in a virtual store, create an application that uses an emotion engine to analyze the user's emotions and provide an optimal shopping experience. If the user is excited, highlight new and limited edition products to further increase the user's excitement."

[1387] The above is a specific embodiment for carrying out the present invention.

[1388] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1389] Step 1:

[1390] The server captures the user's face using a webcam. The input is the real-time video from the webcam, and the output is the captured image data, which is used for subsequent emotion recognition processing.

[1391] Step 2:

[1392] The server inputs the captured image data into the EmotionRecognizer library to analyze the user's emotions. The input is image data, and the output is analyzed emotion data (e.g., happiness, sadness, excitement, etc.). EmotionRecognizer analyzes the user's facial expressions from the image data and identifies the emotion.

[1393] Step 3:

[1394] The server inputs the emotion data into the virtual experience generation module and generates a scenario for realizing the user's desires in a virtual space. The input is the emotion data and the user's desire information, and the output is the generated virtual experience scenario. The virtual experience generation module designs an appropriate virtual experience based on the user's emotions and desires.

[1395] Step 4:

[1396] The server inputs the virtual experience scenario to a virtual experience adjustment module and dynamically changes the experience content according to the emotion. The input is the virtual experience scenario and emotion data, and the output is the adjusted virtual experience scenario. The virtual experience adjustment module adjusts the content and intensity of the experience according to the user's emotion.

[1397] Step 5:

[1398] The server inputs the adjusted virtual experience scenario into a virtual store optimization module to adjust the shopping experience based on the user's emotions. The input is the adjusted virtual experience scenario and the emotion data, and the output is an optimized virtual store experience. The virtual store optimization module changes the way products are displayed and the recommended content according to the user's emotions.

[1399] Step 6:

[1400] The terminal displays the optimized virtual store experience on the head-mounted display. The input is the optimized virtual store experience, and the output is a virtual space that the user visually experiences. The head-mounted display provides the user with a virtual experience in real time.

[1401] The above are the specific processing steps of this system.

[1402] Example 2

[1403] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1404] In modern society, users need to be able to fulfill desires and emotions that cannot be realized in the real world in the virtual world. However, conventional systems have the problem that it is difficult to adjust the experience in real time according to the user's emotions and provide an advanced virtual experience that includes haptic feedback.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the things the user regrets, a means for generating a person for the user to talk to in front of him, a means for providing the user with the feeling of returning to his hometown, a means for the user to go to a place he imagines, a means for the user to feel the touch of water, a means for analyzing the user's emotions and adjusting the virtual experience according to the emotions, and a means for generating a conversation in real time using a generation AI model. This allows the user to realize his or her desires in the virtual world and obtain the optimal experience according to his or her emotions.

[1406] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[1407] "A means to virtually realize the user's regrets" is a technology that allows users to recreate in a virtual space the wishes and hopes that they were unable to realize in the real world.

[1408] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character or avatar with which the user can converse in a virtual space.

[1409] "Means to give users the feeling of returning to their hometown" refers to technology that allows users to experience the scenery and environment of their hometown within a virtual space.

[1410] "A means for users to go to places they imagine" refers to technology that allows users to recreate specific locations or scenery within a virtual space, providing the experience of being in that place.

[1411] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[1412] "Means of analyzing a user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the emotions of a user from their facial expressions, tone of voice, choice of words, etc., and adjusts the content of the virtual experience based on the results of that analysis.

[1413] "Means for generating conversation in real time using a generative AI model" means technology that uses an artificial intelligence model to generate natural conversation in real time between a user and a virtual character.

[1414] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1415] The present invention is a system for allowing users to virtually realize their desires in the metaverse. The purpose of this system is to allow users to fulfill their desires and emotions that cannot be realized in the real world in the virtual world. Specific embodiments are described below.

[1416] Hardware and Software Used

[1417] Hardware: VR devices (e.g. VR headsets), AR devices (e.g. AR glasses), haptic feedback devices (e.g. haptic gloves)

[1418] Software: Metaverse platforms (e.g. virtual reality engines), sentiment analysis engines (e.g. emotion recognition software), generative AI models (e.g. natural language generation models)

[1419] System Overview

[1420] The user wears a VR or AR device and accesses the system. The user communicates their wishes to the system through voice or text input. For example, the user can input a specific wish such as "I want to talk to my grandchildren."

[1421] The server uses a generative AI model to analyze the user's input. Based on the analysis results, it generates a virtual experience to realize the user's desire. For example, if a user inputs "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates a scenario in which the user can have a conversation in real time.

[1422] The generated virtual experience data is sent from the server to the user's device. The device provides the user with a virtual experience through a VR or AR device based on the received data. A haptic feedback device is used to allow the user to feel touch in the virtual environment.

[1423] In addition, the server uses an emotion analysis engine to analyze the user's emotions from facial expressions, tone of voice, choice of words, etc. Based on the analysis results, the virtual experience can be adjusted. For example, if the user is feeling joy, the server will provide an experience that amplifies that joy.

[1424] Examples of concrete examples and prompts

[1425] Example 1: A user types, "I want to talk to my grandchild." The server uses a generative AI model to generate an avatar of the grandchild and starts a conversation in real time.

[1426] Example prompt: "I want to talk to my grandson. His name is Taro."

[1427] Example 2: A user types, "I want to go back to my hometown." The server uses the metaverse platform to recreate the scenery of the user's hometown.

[1428] Example prompt: "Please recreate the scenery of your hometown. The setting is a small village in Hokkaido."

[1429] Example 3: A user inputs "I want to see a beautiful ocean." The server generates a beautiful ocean scene and uses a haptic feedback device to provide the sensation of water.

[1430] Example prompt: "Show me a beautiful ocean scene. If possible, I'd also like to hear the sound of the waves."

[1431] In this way, the system realizes the user's desires in the virtual world and provides an optimal experience according to their emotions.

[1432] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1433] Step 1:

[1434] The user accesses the system and inputs their wishes. The user wears a VR or AR device and communicates their wishes to the system through voice or text input. For example, the user might say, "I want to talk to my grandchildren." The input data is converted to text format and sent to the server.

[1435] Step 2:

[1436] The server analyzes the user's desires and generates an appropriate virtual experience. The server uses a generative AI model to analyze the user's input text. Based on the analysis results, a virtual experience is generated to realize the user's desire. For example, in response to the input "I want to talk to my grandchild," the server generates an avatar of the grandchild and creates the content of the conversation. The generated virtual experience data is output as 3D model data for the avatar and a conversation script.

[1437] Step 3:

[1438] The server transmits the generated virtual experience to the terminal. The server transmits the generated virtual experience data to the user's terminal. The data includes the avatar's movements, conversation content, and virtual environment settings. The terminal prepares to display the virtual experience based on the received data.

[1439] Step 4:

[1440] The terminal displays the virtual experience and provides haptic feedback as necessary. The terminal provides the user with a virtual experience through a VR device or an AR device based on the received virtual experience data. The user starts a conversation with the grandchild's avatar. The haptic feedback device is used to allow the user to feel haptics in the virtual environment. For example, when the user touches the grandchild, the haptic feedback device vibrates to provide a haptic sensation.

[1441] Step 5:

[1442] The server analyzes the user's emotions and adjusts the virtual experience. Using an emotion analysis engine, the server analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc. The server adjusts the virtual experience based on the analysis results. For example, if the user is feeling happy, the content of the conversation and the avatar's actions are adjusted to amplify that joy. The input data for emotion analysis are the user's facial expressions and voice data, and the output is adjusted virtual experience data.

[1443] (Application example 2)

[1444] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1445] In modern society, the means by which users can realize their desires in the virtual world are limited. Although there are systems that adjust virtual experiences according to the user's emotions, there is still a lack of means to analyze the user's emotions in real time and provide an optimal experience in a virtual shopping experience. This makes it difficult for users to have a more satisfying virtual experience.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1447] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to converse with, a means for providing the user with the feeling of returning to his / her hometown, a means for allowing the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for recognizing the user's emotions and adjusting the virtual experience according to the emotions, and a means for optimizing the virtual shopping experience based on the user's emotions. This allows the user to realize his / her desires in the virtual world and further obtain an optimal virtual shopping experience according to his / her emotions.

[1448] "Means of linking with the metaverse" refers to technology for communicating with the metaverse, a virtual reality space, and sending and receiving data.

[1449] "A means to virtually realize the user's regrets" is a technology that allows the user to recreate and experience their desires and regrets in a virtual space.

[1450] "A means for generating a person with whom the user can converse in front of the user" refers to technology for generating avatars or characters with which the user can converse in a virtual space.

[1451] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[1452] "A means to allow users to go to places they imagine" is a technology that generates a specific place imagined by the user in a virtual space, allowing the user to experience that place.

[1453] "A means for allowing users to feel the sensation of water" refers to technology that recreates the tactile sensation of water in a virtual space, providing users with the sensation of actually touching water.

[1454] "Means of recognizing the user's emotions and adjusting the virtual experience in accordance with those emotions" refers to technology that analyzes the user's emotions from their facial expressions, tone of voice, etc., and changes the content of the virtual experience based on the results.

[1455] "Means for optimizing the virtual shopping experience based on user emotions" refers to technology that analyzes user emotions in real time and adjusts the content and delivery method of virtual shopping in response to those emotions.

[1456] The system for implementing the present invention includes a number of means for coordinating a virtual experience according to a user's emotions in cooperation with the metaverse. Specific embodiments are described below.

[1457] System Configuration

[1458] Hardware

[1459] Webcam: Used to capture the user's facial expressions.

[1460] Smartphone: A device that allows users to access the system and engage in the virtual experience.

[1461] Smart glasses: A device for providing an augmented reality (AR) experience.

[1462] Head-mounted display: A device for providing a virtual reality (VR) experience.

[1463] software

[1464] OpenCV: An image processing library used to analyze the user's facial expressions.

[1465] EmotionRecognizer: An emotion recognition library used to analyze user emotions in real time.

[1466] MetaverseAPI: An API for linking with the metaverse and providing virtual experiences.

[1467] Data processing and calculation

[1468] Image capture and emotion recognition

[1469] The server captures the user's face image in real time using a webcam, which is pre-processed using OpenCV and analyzes the user's emotions using the EmotionRecognizer library.

[1470] Coordinating Virtual Experiences

[1471] The server uses the Metaverse API to tailor the content of the virtual experience based on the analyzed emotion data. For example, if the user is recognized as "happy," the system will display special discounts and recommended products to make the virtual shopping experience more enjoyable.

[1472] Examples

[1473] When a user is enjoying shopping in a virtual store: The emotion engine analyzes the user's facial expressions and if it determines that the user is enjoying themselves, the system will display special discounts and recommended products.

[1474] If the user is tired: If the emotion engine analyzes the user's facial expressions and determines that the user is tired, the system will provide relaxing music and interactive guidance.

[1475] Examples of prompt statements

[1476] When a user is enjoying shopping in a virtual store, the emotion engine will analyze the user's facial expressions. If the system determines that the user is enjoying themselves, it will display special discounts and recommended products. If the system determines that the user is tired, it will provide relaxing music and interactive guidance.

[1477] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[1478] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1479] Step 1:

[1480] The server captures the user's facial image in real time using a webcam. The input is the video data from the webcam, and the output is the captured facial image, which is used in subsequent processing steps.

[1481] Step 2:

[1482] The server preprocesses the captured face image using OpenCV. The input is the face image obtained in step 1, and the output is the preprocessed face image. Specific operations include face detection, image resizing, noise removal, etc.

[1483] Step 3:

[1484] The server analyzes the preprocessed face image using the EmotionRecognizer library to recognize the user's emotions. The input is the preprocessed face image obtained in step 2, and the output is the recognized emotion data. Specific operations include extracting facial feature points, inputting them into the emotion model, and classifying emotions.

[1485] Step 4:

[1486] The server adjusts the content of the virtual experience using the Metaverse API based on the recognized emotion data. The input is the emotion data obtained in step 3, and the output is the adjusted virtual experience settings. Specific operations include selecting a scenario for the virtual experience according to the emotion data, placing virtual objects, setting interactions, etc.

[1487] Step 5:

[1488] The terminal receives the adjusted virtual experience settings sent from the server and provides them to the user. The input is the virtual experience setting data obtained in step 4, and the output is the virtual environment experienced by the user. Specific operations include rendering the virtual environment, displaying a user interface, and executing interactions.

[1489] Step 6:

[1490] The user enjoys a virtual experience provided through the device. The input is the virtual environment provided by the device, and the output is the user's experience and changes in emotions. Specific actions include movement within the virtual environment, interaction with objects, and emotional feedback.

[1491] In this way, users can realize their desires in the virtual world and have the most optimal virtual shopping experience according to their emotions.

[1492] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1493] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1494] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1495] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1496] [Fourth embodiment]

[1497] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1498] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1500] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1502] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1504] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1505] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1506] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1507] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1508] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1509] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1510] "Example 1"

[1511] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[1512] "Example 2"

[1513] The embodiment of the present invention is a system having a means for linking with the metaverse. Specifically, the system provides a means for a user to realize his / her regrets in a virtual world. For example, if a user has a desire to talk with his / her grandchild, the system generates an avatar of the user's grandchild and enables the user to talk in real time. If a user has a desire to return to his / her hometown, the system reproduces the scenery of the user's hometown and provides the user with a feeling as if he / she is there. Furthermore, if a user has a desire to see a beautiful ocean, the system generates the ocean scenery that the user imagines and enables the user to go to that place. At this time, the system also includes a means for providing haptic feedback so that the user can feel the touch of water. These means enable the user to realize his / her desires in the virtual world.

[1514] The process flow of each embodiment will be described below.

[1515] "Example 1"

[1516] Step 1: The user starts the system and inputs their desires. For example, they can input their desires, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean," in text or voice.

[1517] Step 2: The system analyzes the input desire and generates a corresponding virtual experience. For example, if the desire is "I want to talk to my grandchild," the system will generate an avatar of the user's grandchild and enable real-time conversation.

[1518] Step 3: The system provides the generated virtual experience to the user. The user can experience the virtual experience through a VR or AR device. For example, if the scenery of the user's hometown is reproduced, the user can feel as if they are there. Step 4: The system monitors the user's response and adjusts the virtual experience as necessary. For example, if the user responds that they would like to feel the touch of water while looking at a seascape, the system utilizes a means of providing haptic feedback to provide the user with the touch of water.

[1519] Example 1

[1520] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1521] In modern society, many users face the challenge of finding it difficult to realize their regrets due to physical and time constraints. For example, it is difficult to talk to far-away family members or experience the feeling of returning to one's hometown. It is also difficult for users to go to places they imagine or experience real sensations through touch. To solve these problems, a means is needed to realize users' wishes using the virtual world.

[1522] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1523] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for analyzing the user's wishes, a means for generating a virtual experience, and a means for setting haptic feedback. This allows the user to go beyond physical constraints and experience their wishes in a realistic way in the virtual world.

[1524] The "metaverse" is a virtual world built on the Internet, a virtual environment in which users can interact with other users through avatars and engage in various activities.

[1525] "Means of virtually realizing the user's regrets" refers to technologies and methods that allow a user to recreate in the virtual world the desires or experiences that they were unable to realize in the real world.

[1526] "Means for generating a person with whom the user can converse in front of the user" refers to a technology or method for generating an avatar of the person with whom the user wants to converse in the virtual world and simulating a conversation in real time.

[1527] "Means of giving the user the feeling of returning to their hometown" refers to technologies or methods that recreate the scenery or environment of the user's hometown in a virtual world, giving the user the feeling of being in that place.

[1528] "A means to go to the place the user imagines" refers to the technology or method that recreates the place the user imagines he or she would like to go to in the virtual world and allows the user to access that place.

[1529] "Means for enabling a user to feel the sensation of water" refers to haptic feedback techniques or methods that enable a user to actually feel the sensation of water when interacting with water in the virtual world.

[1530] "Means for analyzing user wishes" refers to techniques or methods for analyzing the wishes entered by the user, understanding their content, and extracting the necessary information.

[1531] A "means for generating a virtual experience" is a technology or method for generating an experience in a virtual world based on a user's desires.

[1532] "Means for configuring haptic feedback" means techniques or methods for configuring and controlling a haptic feedback device to make a user's experience in a virtual world more realistic.

[1533] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1534] The present invention is a system for enabling a user to realize his / her own desires in a virtual world. A specific embodiment of this system will be described below.

[1535] System Configuration

[1536] This system consists of three main elements: a server, a terminal, and a user. The server works in conjunction with the metaverse to analyze users' desires and generate virtual experiences. The terminal provides an interface for users to access the system and enjoy the virtual experience. Users input their desires through the terminal and experience the virtual experience.

[1537] Hardware and Software Used

[1538] Server: A server with high-performance computing power is required, preferably one with the computing resources to run generative AI models (e.g., OpenAI's GPT-4).

[1539] Terminal: The device through which the user accesses the device, which may be a PC, tablet, smartphone, etc. Also includes haptic feedback devices (e.g. HaptX gloves).

[1540] Software: Game engines (e.g. Unity, Unreal Engine) are used to build virtual worlds, and generative AI models are used to perform natural language processing.

[1541] Program Processing

[1542] The server receives the user's desire and analyzes it. It uses a generative AI model to understand the user's input and extract the necessary information. The server then generates the virtual experience. Specifically, it generates an avatar of the person the user wants to talk to and simulates a real-time conversation. It also uses a game engine to build a realistic 3D environment to recreate the user's hometown or the scenery they want to see. It also controls a haptic feedback device and configures it to make the user's experience in the virtual world feel more realistic.

[1543] Examples

[1544] If a user wishes to have a conversation with their grandchild, the specific steps that are taken are as follows:

[1545] 1. User's wish input: The user inputs "I want to talk to my grandson. His grandson's name is Taro, and he is 10 years old" into the input field of the terminal.

[1546] 2. Wish analysis: The server uses a generative AI model to analyze the wish “I want to talk to my grandchild” and extract details such as the grandchild’s name and age.

[1547] 3. Generating the virtual experience: The server uses a generative AI model to generate an avatar of the user's 10-year-old grandson, Taro, and simulates a real-time conversation with him. It also uses Unity to create a 3D recreation of the user's hometown.

[1548] 4. Setting up haptic feedback: The server controls the HaptX gloves and configures them so that the user can feel the sensation of water in the virtual world.

[1549] 5. Providing the virtual experience: The server provides the generated virtual experience to the user. The user can interact with the grandchild’s avatar through the terminal display and feel the sensation of water through the HaptX gloves.

[1550] Examples of prompt statements

[1551] "I want to have a conversation with my grandson. His name is Taro and he is 10 years old."

[1552] "I want to see the scenery of my hometown. It's a small village in the north."

[1553] "I want to see a beautiful ocean. Preferably one like the ocean in the south."

[1554] This allows users to transcend physical constraints and experience their desires in a realistic way in the virtual world.

[1555] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1556] Step 1:

[1557] The user uses the terminal to input his / her wish. For example, he / she inputs, "I want to talk to my grandson. His / her name is Taro, and he / she is 10 years old." This input is sent to the server.

[1558] Step 2:

[1559] The server analyzes the desire received from the user. The input text data is passed to the generative AI model, which understands the user's desire and extracts the necessary information. Specifically, it analyzes the desire "I want to talk to my grandchild" and the detailed information "My grandchild's name is Taro and he is 10 years old." The analysis result is passed to the next processing step.

[1560] Step 3:

[1561] The server generates the virtual experience based on the analysis results. First, it uses a generative AI model to generate an avatar of the person the user wants to talk to. Specifically, it generates an avatar of the user's 10-year-old grandson, "Taro," and simulates a real-time conversation. The data for this avatar generation is passed to the next processing step.

[1562] Step 4:

[1563] The server uses a game engine (e.g. Unity) to create a realistic 3D environment to recreate the user's hometown or scenery they want to see. For example, if a user inputs "I want to see the scenery of my hometown," a 3D model is generated based on the scenery data of that place. This 3D environment data is passed to the next processing step.

[1564] Step 5:

[1565] The server controls the haptic feedback device (e.g., HaptX gloves) and configures it to make the user's experience in the virtual world feel more realistic. For example, if the user inputs "I want to see a beautiful ocean," the server configures the haptic feedback device to reproduce the feeling of water. This configuration data is passed to the next processing step.

[1566] Step 6:

[1567] The server provides the generated virtual experience to the user. The user interacts with the generated avatar through the display of the terminal and experiences the 3D environment. The user can also feel the touch of water through a haptic feedback device. The data of this virtual experience is sent to the user, and the user can enjoy the experience in the virtual world.

[1568] (Application example 1)

[1569] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1570] In modern society, users need to have realistic experiences in virtual spaces without having to physically move around. However, current systems have difficulty providing sufficient tactile and visual reality when users try on or use products in virtual spaces. In addition, there is a lack of real-time interaction and haptic feedback when users virtually interact with specific places or people. This limits the user's experience and reduces satisfaction.

[1571] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1572] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the regrets of the user, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to his / her hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for the user to try on or use products in a virtual store, a means for generating an avatar for the user, a means for generating a scene of the virtual store, and a means for providing haptic feedback. This allows the user to have a real experience in the virtual space.

[1573] The "metaverse" is an interactive digital environment built in a virtual space where users can interact with other users and content through avatars.

[1574] "A means to virtually realize the user's regrets" is a technology that recreates the user's desires and regrets in a virtual space, allowing the user to experience them.

[1575] "A means for generating a person with whom the user can converse in front of the user" refers to technology that generates an avatar or character with which the user can converse in real time in a virtual space.

[1576] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling that they are in that place.

[1577] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users in a virtual space and provides the user with the experience of visiting those places.

[1578] "Means allowing the user to feel the sensation of water" refers to technology that allows the user to sense the presence of water in a virtual space through haptic feedback.

[1579] "Means for users to try on or use products in a virtual store" refers to technology that provides users with the experience of trying on or using products in a store within a virtual space.

[1580] "Means for generating a user's avatar" refers to technology for generating an avatar for use in a virtual space based on the user's appearance and characteristics.

[1581] "Means for generating a virtual store scene" refers to technology for reproducing the environment and scenery of a store within a virtual space.

[1582] A "means for providing haptic feedback" is a haptic device or technology that enables a user to physically feel their virtual experience.

[1583] In order to implement this invention, the following hardware and software are used. The hardware requires a smartphone and a haptic feedback device. The software requires the OpenAI API and Virtual Reality SDK.

[1584] The server provides a means to link with the metaverse and uses generative AI models to generate data to virtually realize the user's wishes. For example, if a user wants to talk to a specific person, an avatar of that person can be generated and the user can talk to them in real time.

[1585] The terminal provides an interface for the user to try on or try on products in the virtual store. The user can access the virtual store using a smartphone, select products, and try them on or try them on. For example, if the user wants to try on clothes, the terminal can generate an avatar of the user and dress the avatar in the selected clothes. In addition, by using a haptic feedback device, the user can actually feel the texture of the clothes.

[1586] If a user wants to go to a specific place, the server generates a scene of that place and gives the user the feeling that he or she is there. For example, if a user wants to see the sea, the server generates a scene of the sea that the user imagines and allows the user to go to that place. In this case, a haptic feedback device can be used to provide the feeling of water.

[1587] As an example, the following prompt can be used:

[1588] Avatar generation: "Generate an avatar for the user with the following data: name, height, weight"

[1589] Generate a virtual store with the following data: store name, location

[1590] Try on product: "Apply the following product to the avatar: product type, color, size on user data"

[1591] This allows users to have a realistic experience in a virtual space.

[1592] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1593] Step 1:

[1594] The server receives a user request. As input, it contains the user's desired experience (e.g., talking to a specific person, visiting a specific place). The server analyzes this request and sends the necessary data to the generative AI model. As output, it gets the analysis result.

[1595] Step 2:

[1596] The server uses a generative AI model to generate avatars and scenery based on the user's request. The input includes the user's feature data and details of the desired experience. The generative AI model processes these data and generates avatar and scenery data. The output is the generated avatar and scenery data.

[1597] Step 3:

[1598] The server sends the generated avatar and scenery data to the terminal. The input contains the generated avatar and scenery data. The server converts these data into an appropriate format and sends it to the terminal. The output is the data sent to the terminal.

[1599] Step 4:

[1600] The terminal constructs a virtual space based on the received data. The input includes avatar and scenery data sent from the server. The terminal renders this data and displays it to the user. The output is the virtual space displayed to the user.

[1601] Step 5:

[1602] The user begins to experience the virtual space through the device. The input includes the virtual space displayed on the device. The user manipulates an avatar to talk to specific people or visit specific places. The output is the result of the user's manipulation.

[1603] Step 6:

[1604] The terminal transmits the user's operation to the server. The input includes the user's operation data. The terminal transmits these data to the server to realize real-time interaction. The output is the operation data transmitted to the server.

[1605] Step 7:

[1606] The server updates the virtual space based on the user's operation data. The input includes the user's operation data. The server analyzes these data and updates the state of the virtual space. The output is the updated data of the virtual space.

[1607] Step 8:

[1608] The server sends the updated virtual space data to the terminal. As input, the updated virtual space data is included. The server converts these data into an appropriate format and sends them to the terminal. As output, the updated data sent to the terminal is obtained.

[1609] Step 9:

[1610] The terminal displays the updated virtual space to the user. As input, it contains the updated data sent from the server. The terminal renders these data and displays them to the user. As output, it obtains the updated virtual space displayed to the user.

[1611] Step 10:

[1612] The user physically feels the experience in the virtual space using the haptic feedback device. The input includes haptic feedback data sent from the terminal, which allows the user to feel the experience in the virtual space more realistically. The output is the user's haptic experience.

[1613] Example 2

[1614] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1615] In modern society, many users want to recreate their past memories and regrets in virtual space. However, current systems have difficulty realizing users' specific desires in real time, and are unable to provide sensory experiences such as haptic feedback. For this reason, there is a demand for systems that allow users to experience virtual space in a more realistic way.

[1616] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1617] In this invention, the server includes a means for linking with the metaverse, a means for virtually realizing the user's regrets, a means for generating a person for the user to talk to in front of the user, a means for providing the user with the feeling of returning to their hometown, a means for the user to go to a place that the user imagines, a means for the user to feel the touch of water, a means for receiving the user's input and generating a prompt sentence using a generative AI model, a means for generating a virtual space using virtual space generation software, and a means for providing haptic feedback using a haptic feedback device. This allows the user to realize their specific desires in the virtual space in real time and further obtain a more realistic experience through haptic feedback.

[1618] The "metaverse" is a three-dimensional virtual space built using virtual reality technology that users can access via the Internet.

[1619] "A means of virtually realizing the things that users regret" refers to technology that enables users to recreate in a virtual space things or experiences from the past that they feel nostalgic about.

[1620] "Means for generating a person with whom the user can converse in front of the user" refers to technology for generating a virtual character with which the user can converse in a virtual space.

[1621] "A means of giving users the feeling of returning to their hometown" refers to technology that recreates the scenery and environment of the user's hometown in a virtual space, giving the user the feeling of being in that place.

[1622] "A means to allow users to go to places they imagine" is a technology that generates specific places imagined by users within a virtual space, allowing the users to visit those places.

[1623] "A means for allowing a user to feel the sensation of water" refers to haptic feedback technology that allows a user to actually feel the sensation of water when they touch water in a virtual space.

[1624] "Means for receiving user input and generating a prompt sentence using a generative AI model" refers to technology that analyzes information entered by the user and generates an appropriate prompt sentence using a generative AI model.

[1625] "Means for generating a virtual space using virtual space generation software" refers to a technology that uses software for constructing a virtual space to generate a virtual space in response to a user's request.

[1626] "Means for providing haptic feedback using a haptic feedback device" refers to technology that uses a device to actually reproduce the sensation when a user touches a specific object or environment in a virtual space.

[1627] The present invention is a system that allows users to realize their wishes in a virtual space. This system works in conjunction with the metaverse and provides multiple means for users to virtually realize their wishes.

[1628] Hardware and Software Used

[1629] Hardware: Terminals (PC, smartphone, VR device), servers, haptic feedback devices (e.g. HaptX gloves)

[1630] Software: Generative AI models (e.g. GPT-4), virtual world generation software (e.g. Unity, Unreal Engine)

[1631] System Configuration

[1632] 1. User input of wishes

[1633] The user uses the terminal to input their own wishes, such as "I want to talk to my grandchildren," "I want to go back to my hometown," or "I want to see the beautiful ocean."

[1634] 2. Data collection and analysis ...

Claims

1. A means for inputting the desires the user wishes to realize; A means for extracting information necessary to realize the user's wish by analyzing the input user's wish; means for generating a prompt sentence instructing generation of the extracted information; A means for generating information required to realize a user's desire using the generated prompt sentence and the generative AI model; A means for recreating the user's desire in a virtual space using the generated information and providing the user with the recreated virtual space; A system including:

2. The virtual space provided to the user includes any one of a virtual space in which the user can have a conversation with a person of his / her choice, a virtual space that reproduces a specific place that the user wants to go to, and a virtual space in which the user can experience something that he / she wants to experience; The system of claim 1 .

3. A means of analyzing the user's emotional state using an emotion engine; A means for adjusting a virtual space provided to a user based on the analyzed emotional state of the user; The system of claim 1 further comprising:

Images