System

Abstract

To provide a system allowing AI and a human to naturally communicate with each other in a metaverse space.SOLUTION: A system includes means for: generating a metaverse space; generating a generative AI avatar with generative AI as intelligence; generating a human avatar; allowing the generative AI avatar and the human avatar to coexist in the metaverse space; communicating with the generative AI avatar through text or voice; collectively receiving benefits that the generative AI generates for humanity by chatting, consulting on daily life problems or generating data; distinguishing the generative AI avatar from the human avatar in cooperation with identification information; collecting user behavior and utterance data; pre-processing the collected data; using the pre-processed data to train the generative AI model; evaluating a performance of the trained model; and deploying the evaluated model as the generative AI avatar.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Advances in AI technology have made it possible for AI to converse like humans. However, in spaces where AI and humans coexist, particularly in the metaverse, systems that allow AI and humans to communicate naturally have not yet been fully developed. Furthermore, there is no system that allows users to enjoy the various benefits provided by AI all at once. Furthermore, there are insufficient means to distinguish between AI and humans. [Means for solving the problem]

[0005] This invention provides a system that generates a metaverse space where "generated AI avatars" with generative AI intelligence coexist with human avatars. In this system, humans can converse with the "generated AI avatars" via text or a smart speaker, enjoying the benefits that generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation. Furthermore, humans can be identified through linkage with their My Number cards, making it easy to distinguish them from generated AI avatars. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, the means for generating the metaverse space uses 3D graphics technology or VR technology to generate a virtual space. This metaverse space is designed to allow users to move and act freely.

[0029] "Example 2"

[0030] As a means of generating "generative AI avatars" with generative AI intelligence, AI technologies such as deep learning and reinforcement learning are used to generate AI that can converse like a human. This generative AI avatar can learn from the user's actions and comments and generate dialogue and actions according to the user's needs.

[0031] "Example 3"

[0032] As a means of generating human avatars, users will be able to customize their own avatars, specifically by freely setting their own face, body shape, clothing, etc.

[0033] "Example 4"

[0034] As a means of identifying people through linking with My Number cards, the system reads the information on the My Number card and uses it to verify the user's identity, ensuring user safety and making it easy to distinguish between generated AI avatars and human avatars.

[0035] The processing flow of each embodiment will be described below.

[0036] "Example 1"

[0037] Step 1: Using 3D graphics and VR technology, a metaverse space is generated. This metaverse space is designed to allow users to move and act freely.

[0038] Step 2: Design a user interface that allows users to move and act freely within the metaverse space.

[0039] "Example 2"

[0040] Step 1: Use AI techniques such as deep learning and reinforcement learning to create an AI that can converse like a human.

[0041] Step 2: Place the generated AI in the metaverse space as a "generated AI avatar."

[0042] Step 3: Generate the AI ​​avatar so that it learns from the user's actions and utterances and generates dialogue and actions according to the user's needs.

[0043] "Example 3"

[0044] Step 1: Allow users to customize their own avatars. Specifically, users can freely set their own face, body shape, clothing, etc.

[0045] Step 2: Place the configured avatar in the metaverse space and allow the user to freely operate it.

[0046] "Example 4"

[0047] Step 1: Read the information on your My Number card and use it to verify the user's identity.

[0048] Step 2: The verified identity information is used to distinguish between the user's avatar and the generated AI avatar, ensuring user safety and making it easy to distinguish between the generated AI avatar and a human avatar.

[0049] Example 1

[0050] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0051] In conventional metaverse spaces, it was difficult for users to move freely and interact within the virtual space. Furthermore, in an environment where generated AI avatars and human avatars coexist, it was difficult for users to communicate smoothly with the generated AI avatars. Furthermore, realizing a metaverse space that dynamically changes based on the user's actions and comments was also a challenge.

[0052] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0053] In this invention, the server includes a means for transmitting a request for a user to access the metaverse space, a means for the server to receive the request and generate data for the metaverse space, a means for the server to transmit the generated data to the terminal, a means for the terminal to render the metaverse space based on the received data, and a means for the user to move and interact within the metaverse space through the terminal. This allows the user to move freely within the virtual space and smoothly communicate with the generated AI avatar. It also makes it possible to realize a metaverse space that dynamically changes based on the user's actions and comments.

[0054] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[0055] A "generative AI avatar" is a virtual person or character that has generative AI as its intelligence and is generated to converse and interact with users.

[0056] A "human avatar" is a digital alter ego created to represent a real person in a virtual space.

[0057] The "means for sending a request" is a device or software that has the function of sending the information necessary for a user to access the metaverse space to the server.

[0058] "Means for generating data" refers to devices or software that have the functionality to allow the server to create data for the metaverse space based on a user request.

[0059] "Means for transmitting data" refers to devices or software that have the function of transmitting data from the metaverse space generated by the server to the user's terminal.

[0060] A "means for rendering data" is a device or software that has the function of visually displaying the metaverse space based on the data that the terminal receives from the server.

[0061] An "interaction means" is a device or software that has the functionality to allow a user to interact with objects or other avatars within the Metaverse space.

[0062] "Identification information" refers to information used to identify a user, and includes, for example, a My Number card.

[0063] An "audio input device" is a device that allows a user to input voice, and includes, for example, a smart speaker or a microphone.

[0064] The present invention is a system that generates a metaverse space and allows users to freely move and interact within it. A specific embodiment of this system will be described below.

[0065] First, a user launches a dedicated application and sends a request to access the metaverse space. This request includes the user's authentication information and the type of virtual space they want. The device used by the user can be a PC or a VR headset.

[0066] The server receives requests from users and generates data for the Metaverse space using 3D graphics engines such as Unity or Unreal Engine. Specifically, it generates virtual terrain, buildings, and objects, allowing users to move and act freely. It also utilizes VR technology to provide a more immersive experience by allowing users to wear a VR headset.

[0067] The generated data for the Metaverse space is sent from the server to the user's device. This data includes 3D models, textures, lighting information, and more. The device uses Unity or Unreal Engine to render the Metaverse space based on the received data. Users can visually experience this virtual space through a VR headset or display.

[0068] Users can use a keyboard, mouse, or VR controller to move freely within the Metaverse and interact with objects. For example, they can walk around a virtual museum and view the artworks on display. They can also interact with specific objects to display additional information.

[0069] Furthermore, this system includes a generative AI avatar, with which users can converse via text or voice input. The generative AI avatar learns from the user's actions and comments, generating dialogue and actions according to the user's needs. This allows users to enjoy the benefits of generative AI, such as casual conversation, advice on daily life issues, and various data generation, all in one place.

[0070] As a concrete example, consider a scenario in which a user visits a virtual museum. The user wears a VR headset and accesses the metaverse space through their device. The server generates museum data using Unity and sends it to the device. The user can then freely walk around the museum using the VR controller and view the artworks on display.

[0071] An example of a prompt sentence might be:

[0072] "Generate a scenario in which a user visits a virtual museum. The user puts on a VR headset and uses the VR controllers to move freely around the museum and view the artworks on display."

[0073] In this way, users can move freely within the metaverse space and communicate smoothly with the generated AI avatar. It is also possible to realize a metaverse space that changes dynamically based on the user's actions and comments.

[0074] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0075] Step 1:

[0076] A user submits a request to access the metaverse space.

[0077] The user launches a dedicated application and enters their username and password on the login screen to authenticate. If authentication is successful, a request to access the metaverse space is sent to the server. This request includes the user's authentication information and the desired type of virtual space. The input is the user's authentication information and the desired type of virtual space, and the output is the request sent to the server.

[0078] Step 2:

[0079] The server receives the request and generates data for the metaverse space.

[0080] When the server receives a request from a user, it generates data for the metaverse space using a 3D graphics engine such as Unity or Unreal Engine. For example, if the user requests a "virtual city," the server generates the city's terrain data, 3D models of buildings, and objects such as roads and parks. The input is the user's request information, and the output is the data for the generated metaverse space.

[0081] Step 3:

[0082] The server transmits the generated data to the terminal.

[0083] The server compresses the data of the generated metaverse space and sends it to the user's device. The data includes the city's terrain, 3D building models, textures, and lighting information. The input is the data of the generated metaverse space, and the output is the data sent to the device.

[0084] Step 4:

[0085] The device renders the metaverse space based on the data received.

[0086] The device decompresses the data received from the server and uses Unity's rendering engine to render the virtual space. The user wears a VR headset and visually experiences the virtual space displayed on the display. The input is the data received from the server, and the output is the rendered Metaverse space.

[0087] Step 5:

[0088] Users move around the metaverse space and interact with it through their devices.

[0089] Users can freely move around the virtual space using VR controllers. For example, they can walk around a virtual museum and get up close and personal with the artworks on display. They can also interact with specific objects to reveal additional information. The input is the user's manipulation information, and the output is the user's actions and interactions within the virtual space.

[0090] Step 6:

[0091] The user converses with the generated AI avatar.

[0092] The user converses with the generated AI avatar using text or a voice input device. The generated AI avatar learns from the user's actions and comments, and generates dialogue and actions according to the user's needs. For example, if the user consults the generated AI avatar about a problem in their daily life, the generated AI avatar will provide appropriate advice. The input is the user's comments and behavioral information, and the output is the generated AI avatar's response.

[0093] (Application example 1)

[0094] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] In conventional metaverse spaces, users can move and act freely within the virtual space, but it is difficult to recreate a real-life shopping experience. Furthermore, it is not possible to obtain product information or check details in real time, which reduces user convenience. Furthermore, it is difficult to distinguish between generated AI avatars and human avatars, which can sometimes impair the user experience. There is a need to solve these issues and provide a more realistic and convenient virtual shopping experience.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0097] In this invention, the server includes: means for generating a metaverse space; means for generating a "generated AI avatar" whose intelligence is a generative AI; means for generating a human avatar; means for the "generated AI avatar" and the human avatar to coexist in the metaverse space; means for communicating with the "generated AI avatar" via text or a smart speaker; means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, all at once; means for identifying humans in conjunction with identification information; means for easily distinguishing humans from the generated AI avatar; means for generating a virtual store, allowing users to select products in the virtual space and obtain detailed information; and means for obtaining product information using a generative AI model and displaying it in the virtual space. This allows users to obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0098] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[0099] A "generative AI avatar" is a virtual entity that has generative AI as its intelligence and can interact and act with the user.

[0100] A "human avatar" is a digital alter ego that allows a user to represent themselves in a virtual space.

[0101] A "virtual store" is a virtual store set up within the metaverse space where users can select products and check detailed information.

[0102] A "generative AI model" is a model of artificial intelligence that has been trained to perform a specific task using generative AI techniques.

[0103] "Identification information" refers to information for identifying a user, and includes personal identification information such as a My Number card.

[0104] A "prompt" is an instruction entered into a generative AI model to make it perform a specific task.

[0105] The following system configuration will be described as an embodiment of the present invention.

[0106] The server first generates a metaverse space. The metaverse space is created using 3D graphics and VR technology, allowing users to move and act freely. Specifically, the virtual space is generated using a game engine such as Unreal Engine.

[0107] The server then generates a "generative AI avatar" that uses the generative AI as its intelligence. This generative AI avatar is a virtual entity that can interact with and act on the user, and is constructed using a generative AI model such as the OpenAI® API.

[0108] The server also generates a human avatar, a digital alter ego that represents the user in the virtual space and changes dynamically according to the user's actions and words.

[0109] The server configures the "generated AI avatar" and the human avatar to coexist in the metaverse space, allowing the user to interact with and coexist with the generated AI avatar in the virtual space.

[0110] The server also provides the "generated AI avatar" with a means of conversation via text or a smart speaker. Users can communicate with the generated AI avatar through text input or voice input.

[0111] Furthermore, the server will provide a means for users to receive all the benefits that generative AI brings to humanity, such as chatting, advice on daily life issues, and various data generation, allowing users to utilize the diverse functions of generative AI.

[0112] The server provides a means to identify a person in association with their identification information. For example, the server identifies a user using personal identification information such as a My Number card.

[0113] The server also provides a means to easily distinguish between generated AI avatars and human avatars, allowing users to easily distinguish between generated AI avatars and human avatars.

[0114] Furthermore, the server generates a virtual store and provides a means for the user to select a product in the virtual space and obtain detailed information. The user can select a product in the virtual store in the virtual space and check detailed information.

[0115] The server uses the generative AI model to obtain product information and provides a means to display it in the virtual space. Specifically, it obtains product information using the OpenAI API and displays it in the virtual space using Unreal Engine.

[0116] As a concrete example, consider a scenario in which a user puts on a head-mounted display and enters a virtual store. The user can move freely within the virtual space and select a product they are interested in. Detailed information about the selected product is displayed, and the user can try it on or purchase it.

[0117] Examples of prompts for generative AI models include:

[0118] Please tell me more information about product ID 12345.

[0119] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0120] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0121] Step 1:

[0122] The server generates the metaverse space.

[0123] Input: 3D graphics technology and VR technology setting data

[0124] Data Processing: Building a Virtual World with Unreal Engine

[0125] Output: A virtual space where users can move and act freely

[0126] Step 2:

[0127] The server generates a "generated AI avatar" whose intelligence is the generated AI.

[0128] Input: Configuration data for the generative AI model

[0129] Data Processing: Building generative AI avatars using the OpenAI API

[0130] Output: A virtual entity that can interact with and act on the user.

[0131] Step 3:

[0132] The server generates a human avatar.

[0133] Input: User profile information

[0134] Data processing: Creating a digital avatar that changes dynamically based on the user's actions and words

[0135] Output: An avatar that represents the user in a virtual space

[0136] Step 4:

[0137] The server sets up the "generated AI avatar" and the human avatar to coexist in the metaverse space.

[0138] Input: Position information of the generated AI avatar and the human avatar

[0139] Data processing: Arrange them so that they can coexist in the virtual space

[0140] Output: An environment where users can interact and coexist with generated AI avatars in a virtual space.

[0141] Step 5:

[0142] The server provides the "generated AI avatar" with a means to converse with it via text or a smart speaker.

[0143] Input: Text and voice input from the user

[0144] Data processing: Using generative AI models to analyze input and generate appropriate responses

[0145] Output: Response message from the generated AI avatar

[0146] Step 6:

[0147] The server will provide a means for people to receive all the benefits that generative AI brings to humanity, such as chatting, advice on everyday life issues, and generating various data.

[0148] Input: User consultation details and data generation requests

[0149] Data processing: Using generative AI models to generate appropriate responses and data

[0150] Output: Response messages to the user and generated data

[0151] Step 7:

[0152] The server provides a means for identifying a person in association with the identity information.

[0153] Input: User identification information (e.g., My Number card)

[0154] Data processing: Identifying users based on identifying information

[0155] Output: User identification result

[0156] Step 8:

[0157] The server provides a means to easily distinguish between generated AI avatars.

[0158] Input: Identification information of the generated AI avatar and the human avatar

[0159] Data processing: Displaying information to distinguish between the two based on identification information

[0160] Output: A display that allows users to easily distinguish between generated AI avatars and human avatars.

[0161] Step 9:

[0162] The server generates a virtual store and provides a means for users to select products in the virtual space and obtain detailed information.

[0163] Input: Virtual store setting data

[0164] Data processing: Building a virtual store in a virtual space using Unreal Engine

[0165] Output: A virtual store where the user can select products and view more information.

[0166] Step 10:

[0167] The server uses the generative AI model to obtain product information and provides a means to display it within the virtual space.

[0168] Input: The ID of the product selected by the user

[0169] Data processing: Obtain product information using OpenAI API and display it in a virtual space using Unreal Engine

[0170] Output: User-visible product details

[0171] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0172] Example 2

[0173] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] In conventional metaverse spaces, it was difficult for users to have natural conversations with generated AI avatars, and the avatars were unable to respond appropriately based on the user's actions and statements. Furthermore, the metaverse space could not dynamically change according to the user's real-time actions and situations, limiting the user experience.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0176] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" through text or voice input, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life problems, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing between humans and the generated AI avatar, means for collecting user behavior and speech data, means for preprocessing the collected data, means for training a generative AI model using the preprocessed data, means for evaluating the performance of the trained model, means for deploying the evaluated model as a generative AI avatar, and means for the user to interact with the generative AI avatar. This allows the generative AI avatar to respond appropriately based on the user's behavior and speech, and enables the metaverse space to dynamically change according to the user's real-time behavior and situation.

[0177] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0178] A "generative AI avatar" is a virtual character created using generative AI technology that can converse and act like a human.

[0179] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[0180] "Means for communicating through text or voice input" refers to an interface that allows users to converse with the generated AI avatar using text or voice.

[0181] "Chat, advice on everyday life issues, various data generation, etc." is a general term for the various services provided by the generated AI avatar, and is a function to support users' daily lives.

[0182] "Means of identification in conjunction with identification information" refers to technology for recognizing a user using specific identification information (e.g., ID card or biometrics).

[0183] "Means to make it easier to distinguish from generated AI avatars" refers to methods or technologies that allow users to distinguish between generated AI avatars and human avatars.

[0184] "Means for collecting user behavior and comment data" refers to a system for recording the actions and comments that users make within the virtual space.

[0185] "Means for preprocessing collected data" refers to the process used to convert collected data into a form suitable for analysis and model training.

[0186] A "means for training a generative AI model using preprocessed data" is a method for training a generative AI model using preprocessed data.

[0187] "Means for evaluating the performance of a trained model" refers to a method for measuring the accuracy and effectiveness of a generative AI model once training is complete.

[0188] "Means for deploying a fully evaluated model as a generative AI avatar" refers to the process by which a fully evaluated generative AI model can be used as an actual avatar.

[0189] "Means for users to interact with generated AI avatars" refers to interfaces or systems that allow users to have natural interactions with generated AI avatars.

[0190] This invention provides a system that generates a metaverse space in which generated AI avatars and human avatars coexist. Specific embodiments of this system are described below.

[0191] System configuration

[0192] Hardware

[0193] The server uses a high-performance GPU (e.g., NVIDIA Tesla V100) to train and infer generative AI models, and also has a large amount of storage to store user behavior and speech data.

[0194] software

[0195] The server uses the following software to build the system.

[0196] TENSORFLOW(R): Used for training and inference of deep learning models.

[0197] PyTorch: Used for training and inference of deep learning models.

[0198] OpenAI GPT-3 (registered trademark): Used as a natural language processing model.

[0199] Google® Speech-to-Text API: Used to convert voice data into text.

[0200] Data collection and preprocessing

[0201] The server collects user behavior and comment data, including chat logs, voice data, and user operation history. The collected data is preprocessed using the following steps:

[0202] 1. Converting voice data to text: The server converts the voice data to text using the Google Speech-to-Text API.

[0203] 2. Tokenization of text data: The server tokenizes the text data and converts it into a format suitable for the deep learning model.

[0204] 3. Noise removal: The server removes unwanted noise from the data to create a clean dataset.

[0205] Training and evaluating the model

[0206] The server trains a generative AI model using the preprocessed data. Specifically, it uses TensorFlow or PyTorch to input the collected data into the model and update the model's parameters every epoch. Once the model is fully trained, its performance is evaluated using a test dataset. Evaluation metrics include the naturalness of the dialogue and user satisfaction.

[0207] Model Deployment

[0208] Once the evaluation is complete, the model is deployed as a generative AI avatar. The server deploys the model to a cloud environment (e.g., AWS (registered trademark), Google Cloud) and exposes an API endpoint, allowing users to start interacting with the generated AI avatar.

[0209] User interaction

[0210] The user interacts with the generated AI avatar. Based on the user's input, the avatar generates an appropriate response. Here is an example:

[0211] Example: A user asks a generative AI avatar, "What's the weather like today?" and the server uses a generative AI model to respond, "The weather is sunny today."

[0212] Example prompt: "When a user asks a generated AI avatar about today's weather, the avatar will provide up-to-date weather information."

[0213] In this way, the generative AI avatar can generate dialogue and behaviors according to the user's needs.

[0214] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0215] Step 1: Data collection

[0216] The server collects user behavior and speech data. Input includes chat logs, voice data, and user operation history. The server records this data in real time and saves it in storage. Specifically, it captures messages and voice commands sent by users in chat applications.

[0217] Step 2: Data Preprocessing

[0218] The server preprocesses the collected data. The input includes the raw data collected in step 1. The server converts the audio data to text, tokenizes the text data, and removes noise. Specifically, it transcribes the audio data using the Google Speech-to-Text API and then tokenizes the text. The output is a preprocessed, clean dataset.

[0219] Step 3: Model training

[0220] The server trains a generative AI model using the preprocessed data. The input includes the preprocessed data from step 2. The server uses TensorFlow or PyTorch to input the data into the model and update the model parameters every epoch. Specifically, the server trains the model using a deep learning algorithm (e.g., Transformer). The output is a trained generative AI model.

[0221] Step 4: Model evaluation

[0222] The server evaluates the performance of the trained model. The inputs include the model trained in step 3 and a test dataset. The server measures the model's predictive accuracy using the test data and collects and evaluates feedback from users. Specific operations include using the naturalness of the dialogue and user satisfaction as evaluation metrics. The output is the evaluation result.

[0223] Step 5: Model deployment

[0224] The server deploys the evaluated model as a generative AI avatar. The input includes the model evaluated in step 4. The server deploys the model to a cloud environment (e.g., AWS, Google Cloud) and exposes an API endpoint. Specific operations include uploading the model to the cloud server and configuring the API. The output makes the generative AI avatar available.

[0225] Step 6: Interact with the user

[0226] The user interacts with a generative AI avatar. Input includes the user's text and voice input. The server uses the generative AI model to generate a response based on the user's input. Specifically, when the user types "What's the weather like today?", the server responds with "It's sunny today" based on the latest weather information. The output is an appropriate response for the user.

[0227] (Application example 2)

[0228] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] Conventional metaverse spaces and generative AI avatars are limited in their ability to interact with users and generate behavior, and lack concrete means to improve the shopping experience in physical stores. Another issue is the difficulty of learning from users' actions and comments and responding appropriately in real time. Furthermore, effective means are needed to recommend products, provide in-store guidance, and provide coupons to users in physical stores.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generated AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans in conjunction with identification information, means for easily distinguishing between the generated AI avatar and the human avatar, means for engaging in natural conversation with the user in a physical store, recommending products, providing store guidance, and offering coupons, and means for the user to interact with the generated AI avatar through an application installed on a smartphone. This allows users to improve their shopping experience in physical stores.

[0231] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0232] A "generative AI avatar" is a virtual character that is generated using artificial intelligence technologies such as deep learning and reinforcement learning and can converse and act like a human.

[0233] A "human avatar" is a digital character that represents the user in a virtual space and reflects the user's actions and statements.

[0234] "Means of coexistence" is a mechanism that allows generated AI avatars and human avatars to exist simultaneously within the metaverse space and influence each other.

[0235] "Means of communicating through text or smart speakers" refers to a method by which users can interact with the generated AI avatar using text input or voice input.

[0236] "Chat, advice on everyday life issues, various data generation, etc." refers to the various services and information that the generated AI avatar provides through dialogue with the user.

[0237] "Means of identification in association with identification information" refers to a method of recognizing a user by associating them with specific identification information (e.g., My Number card).

[0238] "Measures to make it easier to distinguish" are methods to make it easier to distinguish between generated AI avatars and human avatars.

[0239] "A means of having natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons" refers to a method in which a generated AI avatar interacts with users in physical stores and provides appropriate product information and services.

[0240] "Means for a user to interact with the generated AI avatar through an application installed on a smartphone" refers to a method for a user to communicate with a generated AI avatar through an application on a smartphone.

[0241] This invention is a system for engaging in natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons. This system includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for collectively receiving the benefits that generative AI brings to humanity, such as casual conversation, advice on daily life concerns, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing them from the generated AI avatar, means for engaging in natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons, and means for the user to interact with the generated AI avatar through an application installed on a smartphone.

[0242] The server generates a metaverse space where generated AI avatars and human avatars coexist. Users can interact with the generated AI avatars via text and voice through an application installed on their smartphones. This interaction is generated using the OpenAI API. The generated AI avatar learns from the user's actions and statements and generates interactions and actions according to the user's needs.

[0243] Specifically, if a user asks, "What are the recommended products in this store?", the generated AI avatar will respond, "This week's recommendation is a new smartphone. There are also special discounts, so be sure to check it out." In this way, users can improve their shopping experience in physical stores.

[0244] The hardware used is a smartphone, and the software used is Python and the OpenAI API. Data processing and calculations are performed by taking user input, sending it as a prompt to a generative AI model, and returning the generated response to the user.

[0245] An example of a prompt is as follows:

[0246] User: What products do you recommend at this store?

[0247] AI Avatar: This week's picks are new smartphones, and we've got special deals for you to check out.

[0248] In this way, a system can be implemented to enhance the shopping experience in physical stores.

[0249] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0250] Step 1:

[0251] The user launches a smartphone application and begins interacting with the generated AI avatar. The user then asks questions or makes requests via text input or voice input. The input data is sent to the application in text format.

[0252] Step 2:

[0253] The device (smartphone) receives user input data and sends it to the generative AI model as a prompt. Specifically, it generates a prompt using the OpenAI API and makes an API request. The input data is the user's question or request, and the output data is the prompt sent to the generative AI model.

[0254] Step 3:

[0255] The server uses a generative AI model (OpenAI API) to generate a response based on the prompt. The generative AI model uses deep learning and reinforcement learning to generate an appropriate response to the user's input. The input data is the prompt, and the output data is the generated response.

[0256] Step 4:

[0257] The server returns the generated response to the terminal (smartphone). Specifically, it sends the generated response to the smartphone as an API response. The input data is the generated response, and the output data is the response sent to the smartphone.

[0258] Step 5:

[0259] The device (smartphone) displays the received response to the user. Specifically, it displays the response in text format on the screen or plays the response aloud using speech synthesis technology. The input data is the response received from the server, and the output data is the response that is displayed or played to the user.

[0260] Step 6:

[0261] The user checks the response from the generated AI avatar and asks further questions or requests as necessary, thus continuing the dialogue. The input data is the user's new question or request, and the output data is sent again to the generative AI model as a prompt.

[0262] In this way, users can interact with the generated AI avatar to enhance their shopping experience in physical stores.

[0263] Example 3

[0264] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] In conventional metaverse spaces, it was difficult for users to freely customize their own avatars, and there was a lack of a way to easily distinguish between generated AI avatars and human avatars. This limited the user experience and could lead to security and privacy issues. Furthermore, the lack of identity verification through linkage with personal identification cards meant that user safety was not adequately ensured.

[0266] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a means for generating a metaverse space, a means for generating a "generated AI avatar" whose intelligence is a generated AI, a means for generating a human avatar, a means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, a means for communicating with the "generated AI avatar" via a text or voice input device, a means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, a means for identifying humans in conjunction with a personal identification card, a means for easily distinguishing between humans and generated AI avatars, a means for users to customize their own avatars, and a means for saving the customized avatars. This allows users to freely customize their own avatars and easily distinguish between generated AI avatars and human avatars. Furthermore, identity verification using a personal identification card ensures user safety.

[0267] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0268] A "generative AI avatar" is a virtual character that has been given intelligence using generative AI technology, and is an avatar that can interact and take action with the user.

[0269] A "human avatar" is a digital character that represents the user in a virtual space and is a user-customizable avatar.

[0270] "Personal identification card" means an official means of identification for verifying a user's identity, and includes, for example, a My Number card.

[0271] "Customization" refers to the ability of users to freely change the appearance and attributes of their avatar.

[0272] An "audio input device" is a device that converts a user's voice into a digital signal and inputs it into the system.

[0273] A "session" refers to a unit of a series of operations and communications from the time a user logs in to the time they log out of the system.

[0274] A "database" is a digital storage device for storing and managing user information, avatar customization information, etc.

[0275] "Real-time" refers to immediate response to user operations and input.

[0276] "Identity verification" is the process of officially verifying a user's identity using a personal identification card or similar.

[0277] This invention provides a system that allows users to freely customize generated AI avatars and human avatars that coexist in the metaverse space. Specific embodiments of this system are described below.

[0278] System configuration

[0279] The server generates the metaverse space and makes it accessible to users. The server uses high-performance computing hardware (e.g., cloud servers) to build the metaverse space using 3D modeling tools (e.g., Blender) and user interfaces (e.g., React.js).

[0280] The terminal provides an interface for users to access the metaverse space and customize their avatars. The terminal transmits user input to the server and displays responses from the server to the user.

[0281] Users access the system through a web browser and log in. They are authenticated by entering their email address and password. If authentication is successful, the user can proceed to the avatar customization screen.

[0282] Avatar Customization

[0283] The customization screen allows the user to select options for face shape, body type, clothing, etc. For example, to change hairstyle, the user clicks on the "Hairstyle" option and selects a new hairstyle from the list.

[0284] The terminal transmits the user's selections to the server in real time.

[0285] The server updates the avatar's 3D model based on the received selection information and sends the updated 3D model data to the device.

[0286] The device displays the updated avatar to the user.

[0287] Save your avatar

[0288] Once the user has completed the customization, he or she clicks the "Save" button.

[0289] The terminal sends a save request to the server.

[0290] The server saves the user's customization information in the database and sends a confirmation message to the device when the saving is complete.

[0291] The terminal displays a confirmation message to the user.

[0292] Identification through linkage with My Number Card

[0293] The user holds their My Number card over the card reader.

[0294] The terminal reads the information on the My Number card through a card reader and sends the read information to the server.

[0295] The server verifies the user's identity based on the information on the My Number card it receives, by checking the information on the My Number card against a database to see if there is a match.

[0296] If the identity verification is successful, the server creates or updates the user's account and stores the account information in a database.

[0297] The terminal will display a successful account creation or update message to the user.

[0298] Specific examples

[0299] For example, if a user wants to change the hairstyle of their avatar, they would follow these steps:

[0300] 1. The user clicks on the "Hairstyle" option on the customization screen.

[0301] 2. The device sends the user's selection to the server.

[0302] 3. The server retrieves the 3D model data of the selected hairstyle and applies it to the avatar.

[0303] 4. The device displays the updated avatar to the user.

[0304] Additionally, when a user creates a new account, they follow these steps:

[0305] 1. The user selects the "Authenticate with My Number Card" option on the account creation screen.

[0306] 2. The terminal instructs the user to hold their My Number card over the card reader.

[0307] 3. The user holds their My Number card over the card reader.

[0308] 4. The terminal sends the information read from the My Number card to the server.

[0309] 5. The server verifies the user's identity based on the information received and creates an account.

[0310] 6. The terminal displays a success message to the user.

[0311] Prompt Sentence Examples

[0312] Examples of prompts to input to a generative AI model might include:

[0313] "Please explain what steps a user would take if they wanted to change their hairstyle."

[0314] "Please explain how to verify a user's identity using the My Number card."

[0315] This makes it easier for the generative AI model to understand specific procedures and processing contents. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0316] Step 1: User Login

[0317] The user opens a web browser and accesses the system's login page. The user enters their email address and password and clicks the login button. The terminal receives the user's input and sends it to the server. The server compares the received email address and password with a database and performs authentication. If authentication is successful, the user's session is started, and data for the customization screen is generated and sent to the terminal. The terminal displays the customization screen to the user.

[0318] Input: User's email address and password

[0319] Output: Authentication results and customization screen data

[0320] Step 2: Display the avatar customization screen

[0321] The server generates an avatar customization screen based on the user's session information. The device displays the customization screen received from the server to the user. The user selects options such as face shape, body type, and clothing on the customization screen.

[0322] Input: User session information

[0323] Output: Customization screen data

[0324] Step 3: Customize your avatar

[0325] The user selects options such as face shape, body type, and clothing on the customization screen. For example, to change their hairstyle, the user clicks the "Hairstyle" option and selects a new hairstyle from the list. The device sends the user's selection to the server in real time. The server updates the avatar's 3D model based on the received selection information. The updated 3D model data is sent to the device. The device then displays the updated avatar to the user.

[0326] Input: User's customized selection

[0327] Output: Updated avatar 3D model data

[0328] Step 4: Save your avatar

[0329] When the user has completed customization, he / she clicks the "Save" button. The terminal sends a save request to the server. The server saves the user's customization information in a database. When saving is complete, a confirmation message is sent to the terminal. The terminal displays the confirmation message to the user.

[0330] Input: Save request

[0331] Output: Save completion confirmation message

[0332] Step 5: Scan your My Number card

[0333] The user holds their My Number card over the card reader. The terminal reads the My Number card information through the card reader and sends the read information to the server.

[0334] Input: My Number card information

[0335] Output: Information from the scanned My Number card

[0336] Step 6: Identity Verification

[0337] The server verifies the user's identity based on the received My Number card information. Specifically, it compares the My Number card information with the database to see if there is a match. If identity verification is successful, the server creates or updates the user's account. The account information is saved in the database.

[0338] Input: My Number card information

[0339] Output: Identity verification results and account information

[0340] Step 7: Account Creation or Update Notification

[0341] The terminal will display a successful account creation or update message to the user.

[0342] Input: Account creation or update success message

[0343] Output: User notification

[0344] (Application example 3)

[0345] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] In today's virtual world, when users customize their own avatars and try on clothes and select products in virtual stores, there is a lack of means to verify the user's identity and ensure their safety. There is also a need for a way to easily distinguish between AI-generated avatars and human avatars. This will improve the user experience while providing a safe and reliable virtual shopping environment.

[0347] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generating AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generating AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans through linkage with My Number cards, means for easily distinguishing between humans and generated AI avatars, means for users to customize their own avatars, means for using the avatar to try on clothes and select products in a virtual store, and means for verifying identity using My Number cards. This allows users to customize their own avatars and try on clothes and select products in a virtual store in a safe and reliable virtual shopping environment.

[0348] A "metaverse space" is a three-dimensional virtual space generated using virtual reality technology, in which users can interact through avatars.

[0349] A "generative AI avatar" is a virtual character that has been given intelligence using generative AI technology and is capable of interacting with and acting in ways that interact with users.

[0350] A "human avatar" is a customizable virtual character that allows users to represent themselves in a virtual space.

[0351] The "My Number Card" is a personal identification number card used in Japan to verify personal identity and for various administrative procedures.

[0352] A "virtual store" is an online shopping environment created using virtual reality technology, allowing users to browse and purchase products in a virtual space.

[0353] "Customization" refers to the ability of users to freely set and change the appearance and clothing of their avatar.

[0354] "Identity verification" is the process of verifying a user's personal information and proving that the person is a legitimate user.

[0355] "Trying on" refers to a user's avatar virtually trying on clothes and accessories in a virtual store.

[0356] "Generative AI" is a system that uses artificial intelligence technology to generate data and interact with and take action on behalf of users.

[0357] "Coexistence" refers to a state in which multiple avatars exist simultaneously within the metaverse space and influence each other.

[0358] A system for carrying out this invention has the following configuration: The server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for receiving all the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with an Individual Number card, means for making it easy to distinguish from a generated AI avatar, means for users to customize their own avatar, means for using an avatar to try on clothes and select products in a virtual store, and means for verifying identity using an Individual Number card.

[0359] Hardware and software used

[0360] Hardware: Smartphones, smart glasses, servers

[0361] Software: OpenCV (image processing library), AvatarCustomizer (avatar customization library), MyNumberVerifier (My Number Card verification library)

[0362] Data processing and calculation

[0363] 1. Avatar Customization:

[0364] Users use their smartphones or smart glasses to take a photo of themselves and customize their avatar.

[0365] Use the AvatarCustomizer library to configure the user's face, body type, and clothing.

[0366] Use OpenCV to load a user's face image and apply it to an avatar.

[0367] 2. Check your My Number Card:

[0368] Users scan their My Number card with their smartphone camera to verify their identity.

[0369] Use the MyNumberVerifier library to verify the My Number card image and number.

[0370] Use OpenCV to read an image of the My Number card, extract the number, and confirm it.

[0371] 3. Try-on and product selection in the virtual store:

[0372] Users use their customized avatars to try on clothes and select products in a virtual store.

[0373] Software is used to simulate avatar movement and fitting within a virtual store.

[0374] Specific examples

[0375] Users open the app on their smartphone, take a photo of themselves, and customize their avatar. They then scan their My Number card with the smartphone camera to verify their identity. Once successfully verified, they can use the avatar to try on clothes and select products in a virtual store.

[0376] Prompt Sentence Examples

[0377] Users take a photo of themselves with their smartphone to customize their avatar, then scan their My Number card to verify their identity. Once successfully verified, they can use their avatar to try on clothes and select products in a virtual store.

[0378] In this way, users can customize their avatars, try on clothes and select products in virtual stores in a safe and reliable virtual shopping environment.

[0379] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0380] Step 1:

[0381] The user launches the app on their smartphone and takes a photo of their face. The input is the user's face photo, and the output is face photo data. The face photo is taken using the smartphone's camera function and saved as image data.

[0382] Step 2:

[0383] The server receives the facial photo data and customizes the avatar using the AvatarCustomizer library. The input is the facial photo data and the output is the customized avatar. The server analyzes the facial photo data, extracts the user's facial features, and applies them to the avatar.

[0384] Step 3:

[0385] The user scans their My Number card with their smartphone camera. The input is an image of the My Number card, and the output is the scanned card image data. The smartphone's camera function is used to take a photo of the My Number card and save it as image data.

[0386] Step 4:

[0387] The server receives the image data of the My Number card and performs identity verification using the MyNumberVerifier library. The input is the image data of the My Number card, and the output is the result of identity verification. The server analyzes the image data, extracts the card number, and compares it with the database.

[0388] Step 5:

[0389] The server notifies the user of the result of identity verification. The input is the result of identity verification, and the output is a notification message. If the verification is successful, the server sends the user a success message; if it fails, it sends a message encouraging the user to try again.

[0390] Step 6:

[0391] A user accesses a virtual store using a customized avatar. The input is the customized avatar, and the output is access rights to the virtual store. The user logs in to the virtual store through the app and moves around the store using the avatar.

[0392] Step 7:

[0393] Users use an avatar in a virtual store to try on clothes and select products. The input is the user's operation data, and the output is the results of the try-on and product selection. The user operates the avatar to select products and simulates trying on clothes.

[0394] Step 8:

[0395] The server saves the results of the user's try-on and product selection, and performs the purchase process as necessary. The input is the results of the try-on and product selection, and the output is a notification that the purchase process has been completed. The server records the user's selection, and if the purchase is confirmed, it processes the payment.

[0396] Through these steps, users can customize their own avatars in a safe and reliable virtual shopping environment, and try on clothes and select products in a virtual store.

[0397] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0398] "Example 1"

[0399] One form of invention combining an emotion engine is a system including an emotion engine that recognizes emotions from the user's tone of voice, facial expressions, behavior, etc. In this system, the emotion engine recognizes the user's emotions in real time and adjusts the dialogue and behavior of the generated AI avatar according to those emotions. For example, if the user expresses joy, the generated AI avatar will speak in a bright tone and generate cheerful behavior. Conversely, if the user expresses sadness, the generated AI avatar will speak in a calm tone and generate comforting behavior.

[0400] "Example 2"

[0401] Another possible system is one in which the emotion engine detects changes in a user's emotions in real time and dynamically changes the environment of the metaverse space in response to those changes. For example, if the user is expressing joy, the metaverse space will change to a bright and vibrant environment. Conversely, if the user is expressing sadness, the metaverse space will change to a quiet and calm environment. This will make it possible to provide the optimal metaverse space according to the user's emotions.

[0402] "Example 3"

[0403] Furthermore, it is conceivable that a system could use an emotion engine to recognize the user's emotions and adjust the dialogue and behavior of the generated AI avatar according to those emotions, as well as change the facial expression and tone of voice of the generated AI avatar itself according to the user's emotions. For example, if the user expresses happiness, the generated AI avatar could create a smiling facial expression and speak in a bright tone of voice. Conversely, if the user expresses sadness, the generated AI avatar could create a sad facial expression and speak in a calm tone of voice. This would make it possible to provide dialogue that is more adapted to the user's emotions.

[0404] "Example 4"

[0405] In addition, a system could be envisioned that not only uses an emotion engine to recognize the user's emotions and adjust the generated AI avatar's dialogue and behavior according to those emotions, but also allows the generated AI avatar to learn and evolve its own behavioral patterns according to the user's emotions. For example, if the generated AI avatar performs a specific action when the user expresses joy and the user's reaction is positive, the generated AI avatar will learn that behavior and reproduce it when a similar situation arises. This allows the generated AI avatar to learn and evolve behaviors that are more adapted to the user's emotions.

[0406] The processing flow of each embodiment will be described below.

[0407] "Example 1"

[0408] Step 1: Activate the emotion engine that recognizes emotions from the user's tone of voice, facial expressions, behavior, etc.

[0409] Step 2: The emotion engine recognizes the user's emotions in real time.

[0410] Step 3: Adjust the dialogue and behavior of the generated AI avatar based on the recognized emotions.

[0411] "Example 2"

[0412] Step 1: Launch an emotion engine that detects changes in user emotions in real time.

[0413] Step 2: The emotion engine detects changes in the user's emotions in real time.

[0414] Step 3: Dynamically change the environment of the metaverse space according to changes in detected emotions.

[0415] "Example 3"

[0416] Step 1: Activate the emotion engine that recognizes the user's emotions.

[0417] Step 2: The emotion engine recognizes the user's emotions in real time.

[0418] Step 3: Adjust the dialogue, behavior, facial expressions, and tone of voice of the generated AI avatar based on the recognized emotions.

[0419] "Example 4"

[0420] Step 1: Activate the emotion engine that recognizes the user's emotions.

[0421] Step 2: The emotion engine recognizes the user's emotions in real time.

[0422] Step 3: Adjust the dialogue and behavior of the generated AI avatar based on the recognized emotions.

[0423] Step 4: Learn and evolve the behavioral patterns of the generated AI avatar based on user responses.

[0424] Example 1

[0425] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0426] In conventional metaverse spaces, it was difficult to generate dialogue and behavior based on the user's emotions and behavior, limiting the user experience. Furthermore, there was a lack of technology to recognize the user's emotions in real time and adjust the avatar's dialogue and behavior accordingly. Furthermore, it was difficult to distinguish between the user and the generated AI avatar, making it difficult to identify the user.

[0427] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a means for generating a metaverse space, a means for generating a "generated AI avatar" whose intelligence is a generated AI, a means for generating a human avatar, a means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, a means for communicating with the "generated AI avatar" through text or voice, a means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, a means for identifying humans, a means for easily distinguishing between the generated AI avatar and the human avatar, a means for collecting the user's tone of voice, facial expressions, and behavior in real time, a means for analyzing the collected data and recognizing the user's emotions, and a means for adjusting the dialogue and behavior of the generated AI avatar based on the recognized emotions. This enables the generation of dialogue and behavior in accordance with the user's emotions and behavior, providing a richer user experience. It also makes it easier to distinguish between the user and the generated AI avatar, ensuring reliable user identification.

[0428] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, and is an environment in which users can move and act freely.

[0429] A "generative AI avatar" is a virtual character that has generative AI as its intelligence and can interact and act with the user.

[0430] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[0431] "Means of coexistence" is a mechanism that allows generated AI avatars and human avatars to exist simultaneously within the metaverse space and influence each other.

[0432] "Means of communicating through text or voice" refers to a method by which a user can communicate with a generated AI avatar using text input or voice input.

[0433] "Chat, advice on everyday life issues, various data generation, etc." are various services provided by the generation AI, and are functions to support users' daily lives.

[0434] "Means of human identification" refers to a method for identifying a user and distinguishing them from other users and generated AI avatars.

[0435] "Means to make it easy to distinguish from generated AI avatars" refers to a mechanism that makes it easy to distinguish between users and generated AI avatars.

[0436] "Means for collecting tone of voice, facial expressions, and behavior in real time" refers to a method for acquiring and analyzing a user's audio and video data in real time.

[0437] "Means for analyzing collected data and recognizing the user's emotions" refers to technology for analyzing acquired audio and video data to determine the user's emotional state.

[0438] "Means for adjusting the dialogue and behavior of the generated AI avatar based on the recognized emotions" refers to a method for changing the dialogue content and behavior of the generated AI avatar according to the user's emotional state.

[0439] MODE FOR CARRYING OUT THE INVENTION

[0440] This invention is a system that generates a metaverse space and adjusts the dialogue and behavior of the generated AI avatar according to the user's emotions. Specific embodiments of this system are described below.

[0441] Creating a Metaverse Space

[0442] The server uses 3D graphics technology (e.g., Unity or Unreal Engine) to generate the Metaverse space, a virtual environment in which users can move and act freely. The server loads 3D models, textures, and a physics engine to create virtual cities and natural environments that users can explore.

[0443] Generate AI avatars

[0444] The server generates a "generative AI avatar" whose intelligence is the generative AI. This avatar is a virtual character that interacts with and behaves with the user. The server uses a generative AI model (e.g., OpenAI GPT-4 (registered trademark)) to generate the avatar's dialogue and behavior.

[0445] Collecting user sentiment data

[0446] The device collects the user's tone of voice, facial expressions, and behavior in real time. Specifically, the device uses a microphone and a camera to capture the user's audio and video data. For example, the microphone collects the user's voice while the camera captures the user's facial expressions.

[0447] Emotional Data Analysis

[0448] The device sends the collected audio and video data to an emotion engine (e.g., Microsoft® Azure® Cognitive Services or Google Cloud Vision API) to analyze the user's emotions. The emotion engine analyzes changes in voice tone and facial expressions to determine whether the user is happy or sad.

[0449] Avatar dialogue and behavior generation

[0450] The server uses the generative AI model to generate the avatar's dialogue and behavior based on the emotion data received from the emotion engine. For example, if the user expresses joy, the server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate joyful behavior."

[0451] User feedback

[0452] The server reflects the generated dialogue and actions in the avatar and provides feedback to the user. Specifically, the avatar speaks to the user in a cheerful tone and makes fun movements. For example, the avatar might say in a cheerful tone, "Hello! What would you like to do for fun today?"

[0453] Specific examples

[0454] The user puts on a VR headset and enters the metaverse space. The device collects the user's voice and facial expressions and sends them to the emotion engine. The emotion engine analyzes the user's emotions and detects the emotion of "happiness." The server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate fun actions." The avatar then speaks in a bright tone, saying, "Hello! What would you like to do for fun today?"

[0455] Prompt Sentence Examples

[0456] 1. If the user is expressing happiness:

[0457] "Your users are happy, so interact with them in a light-hearted tone and generate pleasurable behavior."

[0458] 2. If the user expresses sadness:

[0459] "The user is sad, so speak in a calm tone and generate comforting actions."

[0460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0461] Step 1: Initialize the Metaverse Space

[0462] The server initializes the metaverse space using 3D graphics technology (e.g., Unity or Unreal Engine). As input, it loads 3D models, textures, and physics engine data. This generates a virtual environment in which the user can move and act freely. Specifically, the server builds a virtual city or natural environment, allowing the user to explore it.

[0463] Step 2: Collecting user sentiment data

[0464] The device collects the user's tone of voice, facial expressions, and behavior in real time. As input, it uses a microphone and a camera to capture the user's audio and video data, which provides data to understand the user's emotional state. Specifically, the device collects the user's speaking voice with a microphone and captures the user's facial expressions with a camera.

[0465] Step 3: Analyze the emotion data

[0466] The device sends the collected audio and video data to an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API) to analyze the user's emotions. The collected audio and video data is used as input. This provides emotional data to determine whether the user is happy or sad. Specifically, the emotion engine analyzes changes in voice tone and facial expressions to detect the user's emotional state.

[0467] Step 4: Generating Avatar Dialogue and Behavior

[0468] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate the avatar's dialogue and behavior based on the emotion data received from the emotion engine. The emotion data and prompt text are used as input. This generates dialogue content and behavior that corresponds to the user's emotions. Specifically, the server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate fun behavior."

[0469] Step 5: User feedback

[0470] The server reflects the generated dialogue and actions in the avatar and provides feedback to the user. The generated dialogue content and behavior data are used as input. This allows the avatar to engage in appropriate dialogue and actions with the user. A specific example of this behavior is the avatar speaking in a cheerful tone, saying, "Hello! What would you like to do for fun today?"

[0471] (Application example 1)

[0472] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0473] In conventional metaverse spaces, the user experience was limited due to a lack of functionality to recognize users' emotions in real time and adjust the avatar's dialogue and behavior accordingly. Furthermore, virtual stores lacked the functionality to generate dialogue and behavior according to emotions when users browse and purchase products, making it difficult to increase user satisfaction.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generated AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans in conjunction with identification information, means for easily distinguishing between humans and generated AI avatars, means for recognizing a user's emotions in real time and adjusting the dialogue and behavior of the generated AI avatar according to those emotions, and means for generating dialogue and behavior according to emotions when the user browses or purchases products in a virtual store. This enables dialogue and behavior according to the user's emotions to be generated in real time, improving the user experience in the virtual store.

[0475] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[0476] A "generative AI avatar" is a virtual character that has generative AI as its intelligence and can interact and act with the user.

[0477] A "human avatar" is a character that represents the user in a virtual space.

[0478] The "emotion engine" is a system that recognizes emotions from the user's tone of voice, facial expressions, and behavior, and adjusts the dialogue and behavior of the generated AI avatar according to those emotions.

[0479] A "virtual store" is a store set up in a virtual space where users can browse and purchase products.

[0480] "Identification information" refers to information for identifying a user, and includes personal identification information such as a My Number card.

[0481] "Real-time" refers to the system instantly recognizing the user's actions and emotions and responding accordingly.

[0482] "Dialogue" means that the user and the generated AI avatar exchange words and communicate.

[0483] "Behavior" refers to the actions and reactions that the generated AI avatar takes in response to the user's emotions and actions.

[0484] "Chat" is when the user casually discusses everyday topics with the generated AI avatar.

[0485] "Problem consultation" means that users consult with a generated AI avatar about worries and problems they face in their daily lives.

[0486] "Various data generation" means that the generation AI generates various data in response to user requests.

[0487] The system for implementing this invention generates a metaverse space, recognizes the user's emotions in real time, and adjusts the dialogue and behavior of the generated AI avatar. Specific embodiments are described below.

[0488] System configuration

[0489] The system uses the following hardware and software:

[0490] Hardware: Smartphone, head-mounted display, camera

[0491] Software: OpenCV, dlib, MetaverseSDK, EmotionRecognitionModel

[0492] Program processing explanation

[0493] Creating a Metaverse Space

[0494] The server uses 3D graphics and VR technology to generate the metaverse space, which is designed to allow users to move and act freely.

[0495] emotion recognition

[0496] The user's face is captured using a camera on the device (smartphone or head-mounted display). The face is detected using OpenCV and dlib, and facial feature points are extracted. The extracted feature points are input into the EmotionRecognitionModel, and the user's emotions are recognized in real time.

[0497] Coordinating the dialogue and behavior of generative AI avatars

[0498] The server adjusts the generated AI avatar's dialogue and behavior according to the recognized emotion. For example, if the user expresses joy, the generated AI avatar will speak in a bright tone and generate joyful behavior. Conversely, if the user expresses sadness, the generated AI avatar will speak in a calm tone and generate comforting behavior.

[0499] Application in virtual stores

[0500] When a user browses and purchases products in a virtual store, the system generates dialogue and behaviors based on their emotions. For example, if a user shows interest in a product, the generated AI avatar will respond in a cheerful tone, saying, "This product is very popular!". If the user appears depressed, the generated AI avatar will respond in a gentle tone, saying, "Is there something I can help you with? Please let me know if there's anything I can help you with."

[0501] Examples of concrete examples and prompts

[0502] Specific examples

[0503] As users browse products in a virtual store, an avatar cheerfully tells them, "This product is very popular!"

[0504] If the user appears depressed, the avatar will speak to them in a gentle tone, asking, "Is there something I can help you with? Please let me know if there's anything I can help you with."

[0505] Prompt Sentence Examples

[0506] "Generate a scenario where when the user smiles, the avatar speaks in a cheerful tone."

[0507] "Generate a scenario where, when the user makes a sad expression, the avatar will have a comforting conversation."

[0508] The above is a specific embodiment for carrying out the present invention.

[0509] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0510] Step 1:

[0511] The server generates the Metaverse space using 3D graphics and VR technology. It receives the design data of the virtual space as input and provides the virtual space in which users can move and act freely as output. Specifically, the server analyzes the design data and renders a 3D model to construct the virtual space.

[0512] Step 2:

[0513] The device uses a camera to capture the user's face. It receives real-time video data as input and generates facial image data as output. Specifically, the device acquires video data from the camera and cuts out the facial area.

[0514] Step 3:

[0515] The device uses OpenCV and dlib to detect faces and extract facial feature points. It receives facial image data as input and generates facial feature point data as output. Specifically, the device applies a face detection algorithm to identify facial feature points.

[0516] Step 4:

[0517] The device inputs the extracted feature points into the EmotionRecognitionModel to recognize the user's emotions in real time. It receives facial feature point data as input and generates emotion data as output. Specifically, the device inputs the feature point data into the model and applies an emotion recognition algorithm to identify the emotion.

[0518] Step 5:

[0519] The server adjusts the dialogue and behavior of the generated AI avatar according to the recognized emotions. It receives emotion data as input and generates dialogue and behavior data for the avatar as output. Specifically, the server analyzes the emotion data and inputs prompts to the generative AI model to generate appropriate dialogue and behavior.

[0520] Step 6:

[0521] The server reflects the generated dialogue and behavioral data to the generated AI avatar in the metaverse space. It receives the dialogue and behavioral data as input and controls the avatar's behavior as output. Specifically, the server sends the dialogue and behavioral data to the avatar's control system and updates the avatar's behavior.

[0522] Step 7:

[0523] When browsing and purchasing products in a virtual store, users experience emotionally appropriate interactions and behaviors. The input is the avatar's interactions and behaviors, and the output is increased user satisfaction. Specifically, users interact with the avatar to obtain product information and complete the purchase process.

[0524] Example 2

[0525] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0526] Conventional metaverse spaces and generative AI avatar systems have not adequately generated dynamic environmental changes and dialogue in response to the user's emotions and behavior. Furthermore, it is difficult to distinguish between generative AI avatars and human avatars, resulting in a poor user experience. Furthermore, there are issues with inefficient training and evaluation of generative AI models, resulting in suboptimal system performance.

[0527] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generating AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" through a text or voice input device, means for receiving all the benefits that the generating AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with identification information, means for making it easy to distinguish between the human and the generated AI avatar, means for detecting the user's emotions in real time, means for dynamically changing the environment of the metaverse space in response to changes in the emotions, means for training a generating AI model using a high-performance computing device, and means for evaluating the generating AI model and adjusting parameters as necessary. This enables the generation of dynamic environmental changes and dialogue in response to the user's emotions and behavior, makes it easier to distinguish between generated AI avatars and human avatars, and optimizes the performance of the entire system.

[0528] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0529] A "generative AI avatar" is a virtual character created using generative AI technology that has the intelligence to engage in natural conversation and dialogue with the user.

[0530] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[0531] "Text or voice input device" refers to a device that allows a user to input text or voice, including a keyboard, microphone, smart speaker, etc.

[0532] An "emotion engine" is software or hardware that detects a user's emotions in real time and changes the system's behavior in response to those emotions.

[0533] A "high-performance computing device" is a high-performance computer for efficiently training and evaluating generative AI models, including GPUs and dedicated computing units.

[0534] A "generative AI model" is an artificial intelligence model trained using techniques such as deep learning and reinforcement learning, which interacts with users and generates data.

[0535] "Identification information" refers to information used to identify a user, including ID cards and biometric data.

[0536] "Means of dynamic change" refers to technology that allows the state of a system or environment to be changed in real time in response to the user's behavior or emotions.

[0537] "Means for adjusting parameters" refers to methods for changing parameters such as model settings and learning rates in order to optimize the performance of a generative AI model.

[0538] MODE FOR CARRYING OUT THE INVENTION

[0539] This invention is a system that generates a metaverse space and uses the generated AI avatar to interact with users. Specific embodiments of this system are described below.

[0540] System configuration

[0541] The server system is configured using the following hardware and software.

[0542] Hardware: A computing device equipped with a high-performance GPU (e.g., NVIDIA Tesla V100)

[0543] Software: Deep learning frameworks (e.g. TensorFlow, PyTorch)

[0544] Data collection and preprocessing

[0545] The server collects user behavior and speech data. This includes text and voice data entered by the user through the device, as well as sensor data such as the user's facial expressions and movements. The collected data is cleaned and converted into a format suitable for the model. For example, text data is tokenized and voice data is converted into a spectrogram.

[0546] Model training and evaluation

[0547] The server uses the preprocessed data to train a generative AI model. It efficiently trains using a deep learning framework, leveraging a high-performance GPU. After training is complete, the server evaluates the model's performance and adjusts parameters as needed.

[0548] Emotion detection and changes in the metaverse space

[0549] The server uses an emotion engine to detect the user's emotions in real time, and dynamically changes the environment of the metaverse space depending on the user's emotions. For example, if the user expresses joy, the metaverse space will change to a bright and colorful environment.

[0550] User interaction

[0551] The generated AI avatar will converse with the user. It will generate an appropriate response based on what the user says and respond to the user via the device. For example, if the user asks, "What's the weather like today?" the generated AI avatar will respond, "It's sunny today. The temperature is about 25 degrees."

[0552] Examples and prompts

[0553] Below is a specific example of a user initiating a dialogue with a generated AI avatar.

[0554] User: "What's the weather like today?"

[0555] Generated AI avatar: "It's sunny today. The temperature is about 25 degrees."

[0556] Below is a concrete example of how the emotion engine detects the user's emotions in real time and dynamically changes the environment in the metaverse space.

[0557] User: "I'm so happy today!"

[0558] Metaverse space: Transforms into a bright and vibrant environment.

[0559] An example of a prompt sentence to be input to the generative AI model is shown below.

[0560] "How do you change the metaverse space when a user is expressing joy?"

[0561] "How does a generative AI avatar respond when a user displays the emotion of sadness?"

[0562] This will enable the provision of optimal dialogue and metaverse spaces that respond to the user's emotions and behavior.

[0563] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0564] Program processing flow

[0565] Step 1: Data collection

[0566] The server collects data on user behavior and speech, including text and voice data entered by the user through the device, as well as sensor data such as the user's facial expressions and movements.

[0567] Input: Text and voice data that you enter into your device

[0568] Output: raw data collected

[0569] Specific operation: When a user types "What's the weather like today?" into the device, the server collects this text data.

[0570] Step 2: Data Preprocessing

[0571] The server cleans the collected data and converts it into a format suitable for the model, for example, by tokenizing text data or converting audio data into a spectrogram.

[0572] Input: Raw data collected

[0573] Output: Preprocessed data

[0574] Specific operation: The server breaks down the text "What's the weather like today?" into words and converts them into numerical data.

[0575] Step 3: Model training

[0576] The server uses the preprocessed data to train the generative AI model, leveraging a high-performance GPU and a deep learning framework for efficient training.

[0577] Input: Preprocessed data

[0578] Output: A trained generative AI model

[0579] Specific operation: The server trains the neural network using tokenized text data as input.

[0580] Step 4: Model evaluation

[0581] The server evaluates the performance of the trained model using metrics such as precision and recall, and adjusts parameters as needed to optimize the model's performance.

[0582] Input: A trained generative AI model

[0583] Output: Evaluation results and adjusted model

[0584] Specific operation: The server evaluates the model's response accuracy using the test data and logs the results.

[0585] Step 5: Emotion detection

[0586] The server uses an emotion engine to detect the user's emotions in real time, and dynamically changes the environment of the metaverse space according to the user's emotions.

[0587] Input: Real-time user data (text, voice, facial expressions, etc.)

[0588] Output: Detected emotion data

[0589] Specific behavior: When a user types "I'm so happy today!", the server uses the emotion engine to detect the emotion "joy."

[0590] Step 6: Changing the Metaverse Space

[0591] The server dynamically changes the environment of the metaverse space based on the detected emotion data.

[0592] Input: Detected emotion data

[0593] Output: Transformed Metaverse space

[0594] Specific operation: After detecting the emotion of "joy," the server transforms the metaverse space into a bright and colorful environment.

[0595] Step 7: Interact with the user

[0596] The generated AI avatar will converse with the user, generate appropriate responses based on what the user says, and respond to the user via the device.

[0597] Input: User utterance data

[0598] Output: The generated response

[0599] Specific behavior: When the user asks, "What's the weather like today?", the generated AI avatar replies, "It's sunny today. The temperature is about 25 degrees."

[0600] (Application example 2)

[0601] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0602] In conventional metaverse spaces, natural interactions with users and dynamic changes in the environment based on their emotions have not been fully realized. Furthermore, there is no system that can detect users' emotions in real time and change the atmosphere of the metaverse space based on that. This limits the user experience and hinders the full appeal of the metaverse space.

[0603] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0604] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing them from the generative AI avatar, means for detecting user emotions in real time, means for dynamically changing the environment of the metaverse space in response to the user's emotions, means for generating natural conversational responses based on user input, and means for changing the atmosphere of the store in response to the user's emotions. This enables natural dialogue with the user, realizes dynamic changes in the metaverse space in response to the user's emotions, and provides a more engaging user experience.

[0605] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0606] A "generative AI avatar" is a virtual character created using generative AI technology, and is an intelligent avatar that can converse and act like a human.

[0607] A "human avatar" is a character that represents the user in a virtual space and moves according to the user's actions.

[0608] "Emotion detection" is a technology that analyzes a user's emotions in real time from their facial expressions, voice, etc., and identifies their emotional state.

[0609] "Natural conversational response" is a function that allows the generative AI to generate replies to user input in a natural, human-like manner.

[0610] "Dynamic environmental changes" is a function that changes elements of the virtual space, such as lighting, music, and background, in real time according to the user's emotions and actions.

[0611] "Identification information" refers to information used to identify a user, including, for example, an ID card or biometric data.

[0612] "Store atmosphere" refers to environmental elements such as lighting, music, and decorations in a store within a virtual space, and the atmosphere is adjusted by changing these elements according to the user's emotions.

[0613] The following system configuration will be described as an embodiment of the present invention.

[0614] System Configuration

[0615] The system includes a server that generates the metaverse space, a server that generates generative AI avatars, a terminal that detects user emotions in real time, a server that generates natural conversational responses based on user input, and a server that changes the atmosphere of the store according to the user's emotions.

[0616] Hardware and software used

[0617] Hardware: webcam (to capture the user's facial expressions), microphone (for voice input), display (to display the metaverse space)

[0618] Software: OpenAI API (generative AI model), OpenCV (image processing), EmotionDetector (emotion detection), MetaverseEnvironment (metaverse environment management)

[0619] Data processing and calculation

[0620] 1. User facial expression capture: Capture user facial expressions in real time using the device's webcam.

[0621] 2. Emotion detection: The captured facial expression data is analyzed using EmotionDetector to detect the user's emotions.

[0622] 3. AI response generation: The server generates natural conversational responses to user input using OpenAI's generative AI model.

[0623] 4. Dynamic Environment Change: Update the MetaverseEnvironment based on the detected emotions to change the store's atmosphere.

[0624] Specific examples

[0625] For example, if a user is looking for a new smartphone, the following dialogue might take place:

[0626] Prompt Sentence Examples

[0627] User: I'm looking for a new smartphone.

[0628] Assistant: What features are important to you? For example, camera performance, battery life, etc.

[0629] During this interaction, if the user's facial expression indicates happiness, the system will brighten the store's lights and change the background music to an upbeat tone. Conversely, if the user indicates sadness, the system will change the lighting to a more subdued color and quieten the background music.

[0630] Processing flow

[0631] The server receives user input and generates natural conversational responses using a generative AI model. The device detects the user's emotions in real time using a webcam and Emotion Detector and sends the information to the server. The server updates the Metaverse Environment based on the received emotional information, dynamically changing the store's atmosphere.

[0632] In this way, natural dialogue with the user becomes possible, and the metaverse space changes dynamically in response to the user's emotions, providing a more engaging user experience.

[0633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0634] Step 1:

[0635] The user captures facial expressions through the webcam on the device. The input is the user's real-time facial expression data, and the output is the captured image data. The device sends this image data to EmotionDetector.

[0636] Step 2:

[0637] The device uses EmotionDetector to analyze the captured facial expression data and detect the user's emotions. The input is the captured image data, and the output is the detected emotional information (e.g., joy, sadness, surprise, etc.). The device sends this emotional information to the server.

[0638] Step 3:

[0639] The user inputs text or voice through the terminal. The input is the user's text or voice data, and the output is the user input in text format. The terminal sends this user input to the server.

[0640] Step 4:

[0641] The server uses OpenAI's generative AI model to generate natural-sounding conversational responses based on the user's input. The input is the user's textual input data, and the output is the generated conversational response. The server then sends this conversational response to the device.

[0642] Step 5:

[0643] The terminal displays or plays audibly the conversational response received from the server to the user. The input is the generated conversational response, and the output is a display or audio output to the user. The user can then provide further input.

[0644] Step 6:

[0645] The server updates the MetaverseEnvironment based on the emotion information sent from the device, dynamically changing the store's atmosphere. The input is the detected emotion information, and the output is the updated Metaverse environment settings. The server sends this setting to the device.

[0646] Step 7:

[0647] The terminal changes the store atmosphere based on the updated metaverse environment settings received from the server. The input is the updated metaverse environment settings, and the output is the changed store atmosphere (e.g., lighting color, background music, etc.). The user can continue to interact with this changed environment.

[0648] Example 3

[0649] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0650] In conventional metaverse spaces, users have limited means to freely customize their own avatars, and there is a lack of means to easily distinguish between generated AI avatars and human avatars. Furthermore, it is difficult to generate dialogue and behavior that reflects the user's emotions, and there are insufficient means to improve the user experience. This leads to issues such as reduced user satisfaction and limited use of the metaverse space.

[0651] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating an intelligent generated AI avatar, means for generating a human avatar, means for the generated AI avatar and the human avatar to coexist in the metaverse space, means for communicating with the generated AI avatar through text or voice, means for receiving all the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with a personal identification card, means for making it easy to distinguish between humans and generated AI avatars, means for a user to upload a photo of their face and generate an avatar using image processing technology, means for analyzing the user's emotions and adjusting the dialogue and behavior of the generated AI avatar according to the emotions, and means for recording the user's reactions and learning the behavior patterns of the generated AI avatar using a reinforcement learning algorithm. This allows users to freely customize their own avatars, making it easy to distinguish between generated AI avatars and human avatars, and also makes it possible to provide dialogue and behavior that responds to the user's emotions.

[0652] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0653] A "generated AI avatar" is a virtual character generated using artificial intelligence technology, which has intelligence and can interact and act with the user.

[0654] A "human avatar" is a character that represents the user in a virtual space, and is a virtual alter ego that can be customized by the user.

[0655] A "personal identification card" is a card used to verify a user's identity, such as an IC card that contains personal information, such as a My Number card.

[0656] "Image processing technology" is a technology for analyzing and processing digital images, and is a technology for performing processes such as facial recognition and feature extraction.

[0657] "Emotion analysis" is a technology that analyzes input data such as a user's facial expressions and voice to identify the user's emotional state.

[0658] A "reinforcement learning algorithm" is a machine learning algorithm that allows an agent to learn optimal behavior through interaction with the environment.

[0659] "Dialogue adjustment" is the process of appropriately changing the dialogue content and behavior of the generated AI avatar according to the user's emotions and situation.

[0660] "Behavioral pattern learning" is the process by which a generative AI avatar learns optimal behavior based on the user's reactions and reproduces appropriate behavior in similar situations in the future.

[0661] This invention is a system in which a user uses a generated AI avatar and a human avatar in a metaverse space to interact and act according to their emotions. Specific embodiments of this system are described below.

[0662] Avatar Customization

[0663] Users customize their avatars using a dedicated application. Users launch the application and upload a photo of their face. The device analyzes the photo using image processing libraries such as OpenCV and generates an avatar that resembles the user. Users can also customize the avatar's body shape and clothing.

[0664] Linking with My Number Card

[0665] The user holds their personal identification card (e.g., My Number card) over a dedicated reader. The device reads the card information through the NFC reader, and the server uses this information to verify the user's identity. This ensures the user's safety and guarantees that the generated avatar is the user.

[0666] Use of emotion engine

[0667] Users input their facial expressions and voice using a camera or microphone. The device analyzes the user's emotions using Microsoft Azure's Emotion API or Google Cloud's Natural Language API. For example, if the user smiles, the emotion engine will identify the emotion "joy."

[0668] Coordinating avatar dialogue and behavior

[0669] The server sends prompts to a generative AI model (e.g., GPT-4) based on the analysis results. The generative AI model generates dialogue content based on the user's emotions. For example, it generates a message like, "Hello, how are you feeling today?" The device displays the generated dialogue content to the user and adjusts the avatar's facial expression and tone of voice.

[0670] Learning avatar behavior patterns

[0671] The server records the user's reactions and uses a reinforcement learning algorithm to learn the avatar's behavioral patterns. For example, if the AI ​​avatar performs a specific action when the user expresses joy, and the user's reaction is positive, the AI ​​avatar will learn that behavior and reproduce it when a similar situation occurs.

[0672] Examples of concrete examples and prompts

[0673] A user launches the application and uploads a photo of their face. The application uses OpenCV to analyze the photo and generate an avatar that resembles the user. The user then holds their personal identification card over an NFC reader, and the server verifies the user's identity. When the user smiles into the camera, the emotion engine analyzes this, and the generated AI avatar creates a smiling expression and begins a conversation with a cheerful voice, asking, "Hello, how are you feeling today?"

[0674] Prompt Sentence Examples

[0675] "When a user smiles at the camera, the generative AI avatar should adopt a smiling facial expression and begin the dialogue with a cheerful tone of voice."

[0676] In this way, a system can be realized that provides dialogue and actions according to the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0677] Step 1:

[0678] The user launches a dedicated application. The user clicks the "Upload Face Photo" button, selects and uploads a photo of their face. The device receives the uploaded photo and analyzes it using the OpenCV library. Specifically, it extracts facial features (such as the position of the eyes, nose, and mouth) and uses this to generate an avatar that resembles the user. The input is the user's face photo, and the output is an avatar that resembles the user.

[0679] Step 2:

[0680] The user holds their personal identification card (e.g., My Number card) over an NFC reader. The device reads the card's information through the NFC reader. The read information is sent to a server, which then uses this information to verify the user's identity. Specifically, the server obtains personal information from the IC chip in the card and compares it with a government database. The input is the information on the personal identification card, and the output is the user's identity verification result.

[0681] Step 3:

[0682] Users input their facial expressions and voice through a camera or microphone. The device receives this input data and analyzes the user's emotions using Microsoft Azure's Emotion API or Google Cloud's Natural Language API. Specifically, it performs facial expression recognition and voice analysis to identify the user's emotional state. The input is the user's facial expressions and voice, and the output is the user's emotional state.

[0683] Step 4:

[0684] The server sends a prompt to a generative AI model (e.g., GPT-4) based on the analysis results. The prompt contains the user's emotional information. The generative AI model generates dialogue content based on the prompt according to the user's emotion. Specifically, if the user expresses joy, it generates the message "Hello, how are you feeling today?" The input is the user's emotional information, and the output is the generated dialogue content.

[0685] Step 5:

[0686] The device displays the generated dialogue to the user and adjusts the avatar's facial expression and tone of voice. Specifically, based on the dialogue received from the generative AI model, the device configures the avatar to speak with a smile and a bright tone of voice. The input is the generated dialogue, and the output is the adjusted avatar's facial expression and tone of voice.

[0687] Step 6:

[0688] The server records the user's reactions, specifically whether the user smiled and how they responded to the interaction. This data is used to learn the avatar's behavioral patterns using a reinforcement learning algorithm. The input is the user's reaction data, and the output is the learned behavioral patterns.

[0689] Step 7:

[0690] The server evolves the avatar's behavior based on the learning results. Specifically, the avatar is configured to perform a specific behavior when the user expresses joy. This allows the avatar to reproduce that behavior when a similar situation occurs. The input is the learning results, and the output is the evolved avatar's behavior.

[0691] (Application example 3)

[0692] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0693] In conventional metaverse spaces, users had limited means to customize their avatars, and the avatars' facial expressions and behaviors could not be changed in response to the user's emotions. This made it difficult to provide a more personalized experience for users. Furthermore, user identity verification was insufficient, creating security issues.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0695] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via a text or voice input device, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life problems, and various data generation, means for identifying humans in conjunction with a personal identification card, means for easily distinguishing between the human and the generated AI avatar, means for recognizing a user's emotions and changing the facial expression and tone of voice of the generated AI avatar according to the emotions, and means for learning and evolving the behavioral patterns of the generated AI avatar according to the user's emotions. This enables a personalized experience according to the user's emotions, ensuring security and enabling more natural conversations and actions.

[0696] A "metaverse space" is a virtual space generated using virtual reality technology in which users can interact through digital avatars.

[0697] A "generated AI avatar" is an intelligent digital avatar generated using artificial intelligence technology that can interact with and take action with the user.

[0698] A "human avatar" is a digital character that represents the user in a virtual space, and allows the user to customize their face, body shape, clothing, etc.

[0699] A "personal identification card" is a card used to verify a user's identity, such as a My Number card.

[0700] An "emotion engine" is a technology that recognizes the user's emotions and adjusts the system's behavior according to those emotions.

[0701] A "voice input device" is a device that receives a user's voice as input and transmits it to the system.

[0702] A "behavioral pattern" is a series of actions and reactions that the generated AI avatar takes based on learning the user's actions and reactions.

[0703] A "personalized experience" is an experience that is customized based on the individual characteristics and emotions of each user, providing more individualized services and interactions.

[0704] "Security" refers to the technologies and methods used to ensure the safety of systems and data and to prevent unauthorized access and information leaks.

[0705] A system for implementing the present invention has the following configuration.

[0706] System Configuration

[0707] 1. Creating the Metaverse Space:

[0708] The server uses virtual reality technology to generate the Metaverse space, a virtual space where users can interact through digital avatars.

[0709] 2. Generate AI avatars:

[0710] The server generates a "generated AI avatar" that uses the generated AI as its intelligence. This avatar can interact with and act on behalf of the user.

[0711] 3. Human avatar generation:

[0712] Users can create a human avatar by customizing their face, body shape, clothing, etc. This avatar is a digital character that represents the user in a virtual space.

[0713] 4. Avatar Coexistence:

[0714] The server provides a means for "generated AI avatars" and human avatars to coexist in the metaverse space, allowing users and generated AI avatars to interact in the same space.

[0715] 5. Means of communication:

[0716] Users can converse with the "generated AI avatar" through text or voice input devices, such as smartphones or smart speakers.

[0717] 6. Receiving the benefits all at once:

[0718] The server will provide a means for people to enjoy all the benefits that generative AI brings to humanity, including casual conversation, advice on everyday life issues, and various data generation.

[0719] 7. Linking with personal identification cards:

[0720] The server provides a means to identify a person in association with a personal identification card (e.g., My Number Card), thereby verifying the user's identity.

[0721] 8. Emotion Recognition and Avatar Adjustment:

[0722] The server uses an emotion engine to recognize the user's emotions and provides a means to change the facial expressions and tone of voice of the generated AI avatar according to those emotions.

[0723] 9. Learning Behavioral Patterns:

[0724] The server provides a means for learning and evolving the behavioral patterns of the generated AI avatar according to the user's emotions, allowing the generated AI avatar to learn and evolve behaviors that are more adapted to the user's emotions.

[0725] Hardware and software used

[0726] Hardware:

[0727] Server: Generates the metaverse space, manages avatars, recognizes emotions, and performs other tasks.

[0728] Smartphone: Allows users to customize avatars and serves as a voice input device.

[0729] Smart speaker: Used as a voice input device.

[0730] Personal ID Card Reader: Reads the user's personal ID card.

[0731] software:

[0732] Metaverse generation software: Software for generating virtual spaces.

[0733] Generative AI model: A model for constructing the intelligence of generative AI avatars.

[0734] Emotion engine: Software that recognizes the user's emotions and adjusts the avatar's facial expressions and tone of voice.

[0735] Avatar customization software: Software that allows users to customize their avatars.

[0736] Specific examples

[0737] Users visit a virtual store and use their smartphone camera to recognize their emotions. The avatar changes its facial expression and tone of voice depending on the user's emotions, and tries on and tests out products. For example, if the user is happy, the avatar will smile and explain the products in a cheerful voice.

[0738] Prompt Sentence Examples

[0739] When a user is shopping in a virtual store, design a system that uses an emotion engine to recognize the user's emotions and adjust the avatar's facial expressions and tone of voice accordingly. If the user is happy, the avatar will smile and explain the products in a cheerful voice, and if the user is sad, the avatar will talk in a calm voice.

[0740] The above is an embodiment of the present invention.

[0741] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0742] Step 1:

[0743] The server generates the metaverse space.

[0744] Input: Virtual reality technology configuration data.

[0745] Data processing: A virtual space is constructed based on the setting data.

[0746] Output: The metaverse space accessible to the user.

[0747] Specific operation: The server uses virtual reality technology to generate 3D models and environments, creating a virtual space that users can access.

[0748] Step 2:

[0749] The server generates a "generated AI avatar" whose intelligence is the generated AI.

[0750] Input: Parameters of the generative AI model.

[0751] Data processing: Building avatar intelligence using generative AI models.

[0752] Output: An intelligent generative AI avatar.

[0753] Specific operation: The server uses a generative AI model to build the avatar's intelligence and generate an avatar that can interact with and act on the user.

[0754] Step 3:

[0755] Users can create a human avatar by customizing their face, body shape, clothing, etc.

[0756] Input: User customization data.

[0757] Data processing: Generates the appearance of an avatar based on customization data.

[0758] Output: A customized human avatar.

[0759] Specific operation: Users use a smartphone or PC to customize their own face, body shape, clothing, etc. to create a human avatar.

[0760] Step 4:

[0761] The server provides a means for "generated AI avatars" and human avatars to coexist in the metaverse space.

[0762] Input: Generated AI avatar and human avatar data.

[0763] Data processing: Integrate the data of both avatars into the metaverse space.

[0764] Output: A metaverse space where coexisting avatars are displayed.

[0765] Specific operation: The server places the generated AI avatar and a human avatar in the metaverse space, allowing the two to interact with each other.

[0766] Step 5:

[0767] Users converse with the "generated AI avatar" through text or voice input devices.

[0768] Input: User's written or spoken input.

[0769] Data processing: Analyze input data and transmit it to the generated AI avatar.

[0770] Output: The response of the generated AI avatar.

[0771] Specific operation: The user uses a smartphone or smart speaker to input text or voice and converse with the generated AI avatar.

[0772] Step 6:

[0773] The server will provide a means for people to enjoy all the benefits that generative AI brings to humanity, including casual conversation, advice on everyday life issues, and various data generation.

[0774] Input: User's question or inquiry.

[0775] Data processing: Use generative AI models to generate appropriate responses.

[0776] Output: Response to the user.

[0777] Specific operation: The server analyzes the user's question or inquiry, uses a generative AI model to generate an appropriate response, and provides it to the user.

[0778] Step 7:

[0779] The server provides a means for identifying a person in association with a personal identification card.

[0780] Input: Personal identification card data.

[0781] Data processing: Analyzes card data to verify user identity.

[0782] Output: User identification result.

[0783] Specific operation: The server uses a personal identification card reader to read the card data and verify the user's identity.

[0784] Step 8:

[0785] The server uses an emotion engine to recognize the user's emotions and provides a means to change the facial expressions and tone of voice of the generated AI avatar according to those emotions.

[0786] Input: User's facial and voice data.

[0787] Data processing: Analyze emotions using an emotion engine.

[0788] Output: Changes in facial expressions and tone of voice of the generated AI avatar.

[0789] Specific operation: The server analyzes the user's facial expression and voice data using an emotion engine and adjusts the facial expression and tone of voice of the generated AI avatar.

[0790] Step 9:

[0791] The server provides a means to learn and evolve the behavioral patterns of the generated AI avatar according to the user's emotions.

[0792] Input: User emotion and reaction data.

[0793] Data processing: Using generative AI models to learn behavioral patterns.

[0794] Output: Behavioral patterns of an evolved generative AI avatar.

[0795] Specific operation: The server collects the user's emotional and reaction data, learns behavioral patterns using a generative AI model, and evolves the behavior of the generated AI avatar.

[0796] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0797] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0798] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0799] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0800] [Second embodiment]

[0801] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0802] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0803] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0804] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0805] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0806] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0807] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0808] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0809] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0810] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0811] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0812] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0813] "Example 1"

[0814] In one embodiment of the present invention, the means for generating the metaverse space uses 3D graphics technology or VR technology to generate a virtual space. This metaverse space is designed to allow users to move and act freely.

[0815] "Example 2"

[0816] As a means of generating "generative AI avatars" with generative AI intelligence, AI technologies such as deep learning and reinforcement learning are used to generate AI that can converse like a human. This generative AI avatar can learn from the user's actions and comments and generate dialogue and actions according to the user's needs.

[0817] "Example 3"

[0818] As a means of generating human avatars, users will be able to customize their own avatars, specifically by freely setting their own face, body shape, clothing, etc.

[0819] "Example 4"

[0820] As a means of identifying people through linking with My Number cards, the system reads the information on the My Number card and uses it to verify the user's identity, ensuring user safety and making it easy to distinguish between generated AI avatars and human avatars.

[0821] The processing flow of each embodiment will be described below.

[0822] "Example 1"

[0823] Step 1: Using 3D graphics and VR technology, a metaverse space is generated. This metaverse space is designed to allow users to move and act freely.

[0824] Step 2: Design a user interface that allows users to move and act freely within the metaverse space.

[0825] "Example 2"

[0826] Step 1: Use AI techniques such as deep learning and reinforcement learning to create an AI that can converse like a human.

[0827] Step 2: Place the generated AI in the metaverse space as a "generated AI avatar."

[0828] Step 3: Generate the AI ​​avatar so that it learns from the user's actions and utterances and generates dialogue and actions according to the user's needs.

[0829] "Example 3"

[0830] Step 1: Allow users to customize their own avatars. Specifically, users can freely set their own face, body shape, clothing, etc.

[0831] Step 2: Place the configured avatar in the metaverse space and allow the user to freely operate it.

[0832] "Example 4"

[0833] Step 1: Read the information on your My Number card and use it to verify the user's identity.

[0834] Step 2: The verified identity information is used to distinguish between the user's avatar and the generated AI avatar, ensuring user safety and making it easy to distinguish between the generated AI avatar and a human avatar.

[0835] Example 1

[0836] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] In conventional metaverse spaces, it was difficult for users to move freely and interact within the virtual space. Furthermore, in an environment where generated AI avatars and human avatars coexist, it was difficult for users to communicate smoothly with the generated AI avatars. Furthermore, realizing a metaverse space that dynamically changes based on the user's actions and comments was also a challenge.

[0838] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0839] In this invention, the server includes a means for transmitting a request for a user to access the metaverse space, a means for the server to receive the request and generate data for the metaverse space, a means for the server to transmit the generated data to the terminal, a means for the terminal to render the metaverse space based on the received data, and a means for the user to move and interact within the metaverse space through the terminal. This allows the user to move freely within the virtual space and smoothly communicate with the generated AI avatar. It also makes it possible to realize a metaverse space that dynamically changes based on the user's actions and comments.

[0840] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[0841] A "generative AI avatar" is a virtual person or character that has generative AI as its intelligence and is generated to converse and interact with users.

[0842] A "human avatar" is a digital alter ego created to represent a real person in a virtual space.

[0843] The "means for sending a request" is a device or software that has the function of sending the information necessary for a user to access the metaverse space to the server.

[0844] "Means for generating data" refers to devices or software that have the functionality to allow the server to create data for the metaverse space based on a user request.

[0845] "Means for transmitting data" refers to devices or software that have the function of transmitting data from the metaverse space generated by the server to the user's terminal.

[0846] A "means for rendering data" is a device or software that has the function of visually displaying the metaverse space based on the data that the terminal receives from the server.

[0847] An "interaction means" is a device or software that has the functionality to allow a user to interact with objects or other avatars within the Metaverse space.

[0848] "Identification information" refers to information used to identify a user, and includes, for example, a My Number card.

[0849] An "audio input device" is a device that allows a user to input voice, and includes, for example, a smart speaker or a microphone.

[0850] The present invention is a system that generates a metaverse space and allows users to freely move and interact within it. A specific embodiment of this system will be described below.

[0851] First, a user launches a dedicated application and sends a request to access the metaverse space. This request includes the user's authentication information and the type of virtual space they want. The device used by the user can be a PC or a VR headset.

[0852] The server receives requests from users and generates data for the Metaverse space using 3D graphics engines such as Unity or Unreal Engine. Specifically, it generates virtual terrain, buildings, and objects, allowing users to move and act freely. It also utilizes VR technology to provide a more immersive experience by allowing users to wear a VR headset.

[0853] The generated data for the Metaverse space is sent from the server to the user's device. This data includes 3D models, textures, lighting information, and more. The device uses Unity or Unreal Engine to render the Metaverse space based on the received data. Users can visually experience this virtual space through a VR headset or display.

[0854] Users can use a keyboard, mouse, or VR controller to move freely within the Metaverse and interact with objects. For example, they can walk around a virtual museum and view the artworks on display. They can also interact with specific objects to display additional information.

[0855] Furthermore, this system includes a generative AI avatar, with which users can converse via text or voice input. The generative AI avatar learns from the user's actions and comments, generating dialogue and actions according to the user's needs. This allows users to enjoy the benefits of generative AI, such as casual conversation, advice on daily life issues, and various data generation, all in one place.

[0856] As a concrete example, consider a scenario in which a user visits a virtual museum. The user wears a VR headset and accesses the metaverse space through their device. The server generates museum data using Unity and sends it to the device. The user can then freely walk around the museum using the VR controller and view the artworks on display.

[0857] An example of a prompt sentence might be:

[0858] "Generate a scenario in which a user visits a virtual museum. The user puts on a VR headset and uses the VR controllers to move freely around the museum and view the artworks on display."

[0859] In this way, users can move freely within the metaverse space and communicate smoothly with the generated AI avatar. It is also possible to realize a metaverse space that changes dynamically based on the user's actions and comments.

[0860] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0861] Step 1:

[0862] A user submits a request to access the metaverse space.

[0863] The user launches a dedicated application and enters their username and password on the login screen to authenticate. If authentication is successful, a request to access the metaverse space is sent to the server. This request includes the user's authentication information and the desired type of virtual space. The input is the user's authentication information and the desired type of virtual space, and the output is the request sent to the server.

[0864] Step 2:

[0865] The server receives the request and generates data for the metaverse space.

[0866] When the server receives a request from a user, it generates data for the metaverse space using a 3D graphics engine such as Unity or Unreal Engine. For example, if the user requests a "virtual city," the server generates the city's terrain data, 3D models of buildings, and objects such as roads and parks. The input is the user's request information, and the output is the data for the generated metaverse space.

[0867] Step 3:

[0868] The server transmits the generated data to the terminal.

[0869] The server compresses the data of the generated metaverse space and sends it to the user's device. The data includes the city's terrain, 3D building models, textures, and lighting information. The input is the data of the generated metaverse space, and the output is the data sent to the device.

[0870] Step 4:

[0871] The device renders the metaverse space based on the data received.

[0872] The device decompresses the data received from the server and uses Unity's rendering engine to render the virtual space. The user wears a VR headset and visually experiences the virtual space displayed on the display. The input is the data received from the server, and the output is the rendered Metaverse space.

[0873] Step 5:

[0874] Users move around the metaverse space and interact with it through their devices.

[0875] Users can freely move around the virtual space using VR controllers. For example, they can walk around a virtual museum and get up close and personal with the artworks on display. They can also interact with specific objects to reveal additional information. The input is the user's manipulation information, and the output is the user's actions and interactions within the virtual space.

[0876] Step 6:

[0877] The user converses with the generated AI avatar.

[0878] The user converses with the generated AI avatar using text or a voice input device. The generated AI avatar learns from the user's actions and comments, and generates dialogue and actions according to the user's needs. For example, if the user consults the generated AI avatar about a problem in their daily life, the generated AI avatar will provide appropriate advice. The input is the user's comments and behavioral information, and the output is the generated AI avatar's response.

[0879] (Application example 1)

[0880] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0881] In conventional metaverse spaces, users can move and act freely within the virtual space, but it is difficult to recreate a real-life shopping experience. Furthermore, it is not possible to obtain product information or check details in real time, which reduces user convenience. Furthermore, it is difficult to distinguish between generated AI avatars and human avatars, which can sometimes impair the user experience. There is a need to solve these issues and provide a more realistic and convenient virtual shopping experience.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0883] In this invention, the server includes: means for generating a metaverse space; means for generating a "generated AI avatar" whose intelligence is a generative AI; means for generating a human avatar; means for the "generated AI avatar" and the human avatar to coexist in the metaverse space; means for communicating with the "generated AI avatar" via text or a smart speaker; means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, all at once; means for identifying humans in conjunction with identification information; means for easily distinguishing humans from the generated AI avatar; means for generating a virtual store, allowing users to select products in the virtual space and obtain detailed information; and means for obtaining product information using a generative AI model and displaying it in the virtual space. This allows users to obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0884] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[0885] A "generative AI avatar" is a virtual entity that has generative AI as its intelligence and can interact and act with the user.

[0886] A "human avatar" is a digital alter ego that allows a user to represent themselves in a virtual space.

[0887] A "virtual store" is a virtual store set up within the metaverse space where users can select products and check detailed information.

[0888] A "generative AI model" is a model of artificial intelligence that has been trained to perform a specific task using generative AI techniques.

[0889] "Identification information" refers to information for identifying a user, and includes personal identification information such as a My Number card.

[0890] A "prompt" is an instruction entered into a generative AI model to make it perform a specific task.

[0891] The following system configuration will be described as an embodiment of the present invention.

[0892] The server first generates a metaverse space. The metaverse space is created using 3D graphics and VR technology, allowing users to move and act freely. Specifically, the virtual space is generated using a game engine such as Unreal Engine.

[0893] The server then generates a "generative AI avatar" that uses the generative AI as its intelligence. This generative AI avatar is a virtual entity that can interact with and act on the user, and is constructed using a generative AI model such as the OpenAI API.

[0894] The server also generates a human avatar, a digital alter ego that represents the user in the virtual space and changes dynamically according to the user's actions and words.

[0895] The server configures the "generated AI avatar" and the human avatar to coexist in the metaverse space, allowing the user to interact with and coexist with the generated AI avatar in the virtual space.

[0896] The server also provides the "generated AI avatar" with a means of conversation via text or a smart speaker. Users can communicate with the generated AI avatar through text input or voice input.

[0897] Furthermore, the server will provide a means for users to receive all the benefits that generative AI brings to humanity, such as chatting, advice on daily life issues, and various data generation, allowing users to utilize the diverse functions of generative AI.

[0898] The server provides a means to identify a person in association with their identification information. For example, the server identifies a user using personal identification information such as a My Number card.

[0899] The server also provides a means to easily distinguish between generated AI avatars and human avatars, allowing users to easily distinguish between generated AI avatars and human avatars.

[0900] Furthermore, the server generates a virtual store and provides a means for the user to select a product in the virtual space and obtain detailed information. The user can select a product in the virtual store in the virtual space and check detailed information.

[0901] The server uses the generative AI model to obtain product information and provides a means to display it in the virtual space. Specifically, it obtains product information using the OpenAI API and displays it in the virtual space using Unreal Engine.

[0902] As a concrete example, consider a scenario in which a user puts on a head-mounted display and enters a virtual store. The user can move freely within the virtual space and select a product they are interested in. Detailed information about the selected product is displayed, and the user can try it on or purchase it.

[0903] Examples of prompts for generative AI models include:

[0904] Please tell me more information about product ID 12345.

[0905] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0906] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0907] Step 1:

[0908] The server generates the metaverse space.

[0909] Input: 3D graphics technology and VR technology setting data

[0910] Data Processing: Building a Virtual World with Unreal Engine

[0911] Output: A virtual space where users can move and act freely

[0912] Step 2:

[0913] The server generates a "generated AI avatar" whose intelligence is the generated AI.

[0914] Input: Configuration data for the generative AI model

[0915] Data Processing: Building generative AI avatars using the OpenAI API

[0916] Output: A virtual entity that can interact with and act on the user.

[0917] Step 3:

[0918] The server generates a human avatar.

[0919] Input: User profile information

[0920] Data processing: Creating a digital avatar that changes dynamically based on the user's actions and words

[0921] Output: An avatar that represents the user in a virtual space

[0922] Step 4:

[0923] The server sets up the "generated AI avatar" and the human avatar to coexist in the metaverse space.

[0924] Input: Position information of the generated AI avatar and the human avatar

[0925] Data processing: Arrange them so that they can coexist in the virtual space

[0926] Output: An environment where users can interact and coexist with generated AI avatars in a virtual space.

[0927] Step 5:

[0928] The server provides the "generated AI avatar" with a means to converse with it via text or a smart speaker.

[0929] Input: Text and voice input from the user

[0930] Data processing: Using generative AI models to analyze input and generate appropriate responses

[0931] Output: Response message from the generated AI avatar

[0932] Step 6:

[0933] The server will provide a means for people to receive all the benefits that generative AI brings to humanity, such as chatting, advice on everyday life issues, and generating various data.

[0934] Input: User consultation details and data generation requests

[0935] Data processing: Using generative AI models to generate appropriate responses and data

[0936] Output: Response messages to the user and generated data

[0937] Step 7:

[0938] The server provides a means for identifying a person in association with the identity information.

[0939] Input: User identification information (e.g., My Number card)

[0940] Data processing: Identifying users based on identifying information

[0941] Output: User identification result

[0942] Step 8:

[0943] The server provides a means to easily distinguish between generated AI avatars.

[0944] Input: Identification information of the generated AI avatar and the human avatar

[0945] Data processing: Displaying information to distinguish between the two based on identification information

[0946] Output: A display that allows users to easily distinguish between generated AI avatars and human avatars.

[0947] Step 9:

[0948] The server generates a virtual store and provides a means for users to select products in the virtual space and obtain detailed information.

[0949] Input: Virtual store setting data

[0950] Data processing: Building a virtual store in a virtual space using Unreal Engine

[0951] Output: A virtual store where the user can select products and view more information.

[0952] Step 10:

[0953] The server uses the generative AI model to obtain product information and provides a means to display it within the virtual space.

[0954] Input: The ID of the product selected by the user

[0955] Data processing: Obtain product information using OpenAI API and display it in a virtual space using Unreal Engine

[0956] Output: User-visible product details

[0957] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[0958] Example 2

[0959] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] In conventional metaverse spaces, it was difficult for users to have natural conversations with generated AI avatars, and the avatars were unable to respond appropriately based on the user's actions and statements. Furthermore, the metaverse space could not dynamically change according to the user's real-time actions and situations, limiting the user experience.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0962] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" through text or voice input, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life problems, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing between humans and the generated AI avatar, means for collecting user behavior and speech data, means for preprocessing the collected data, means for training a generative AI model using the preprocessed data, means for evaluating the performance of the trained model, means for deploying the evaluated model as a generative AI avatar, and means for the user to interact with the generative AI avatar. This allows the generative AI avatar to respond appropriately based on the user's behavior and speech, and enables the metaverse space to dynamically change according to the user's real-time behavior and situation.

[0963] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[0964] A "generative AI avatar" is a virtual character created using generative AI technology that can converse and act like a human.

[0965] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[0966] "Means for communicating through text or voice input" refers to an interface that allows users to converse with the generated AI avatar using text or voice.

[0967] "Chat, advice on everyday life issues, various data generation, etc." is a general term for the various services provided by the generated AI avatar, and is a function to support users' daily lives.

[0968] "Means of identification in conjunction with identification information" refers to technology for recognizing a user using specific identification information (e.g., ID card or biometrics).

[0969] "Means to make it easier to distinguish from generated AI avatars" refers to methods or technologies that allow users to distinguish between generated AI avatars and human avatars.

[0970] "Means for collecting user behavior and comment data" refers to a system for recording the actions and comments that users make within the virtual space.

[0971] "Means for preprocessing collected data" refers to the process used to convert collected data into a form suitable for analysis and model training.

[0972] A "means for training a generative AI model using preprocessed data" is a method for training a generative AI model using preprocessed data.

[0973] "Means for evaluating the performance of a trained model" refers to a method for measuring the accuracy and effectiveness of a generative AI model once training is complete.

[0974] "Means for deploying a fully evaluated model as a generative AI avatar" refers to the process by which a fully evaluated generative AI model can be used as an actual avatar.

[0975] "Means for users to interact with generated AI avatars" refers to interfaces or systems that allow users to have natural interactions with generated AI avatars.

[0976] This invention provides a system that generates a metaverse space in which generated AI avatars and human avatars coexist. Specific embodiments of this system are described below.

[0977] System configuration

[0978] Hardware

[0979] The server uses a high-performance GPU (e.g., NVIDIA Tesla V100) to train and infer generative AI models, and also has a large amount of storage to store user behavior and speech data.

[0980] software

[0981] The server uses the following software to build the system.

[0982] TensorFlow: Used for training and inference of deep learning models.

[0983] PyTorch: Used for training and inference of deep learning models.

[0984] OpenAI GPT-3: Used as a natural language processing model.

[0985] Google Speech-to-Text API: Used to convert voice data into text.

[0986] Data collection and preprocessing

[0987] The server collects user behavior and comment data, including chat logs, voice data, and user operation history. The collected data is preprocessed using the following steps:

[0988] 1. Converting voice data to text: The server converts the voice data to text using the Google Speech-to-Text API.

[0989] 2. Tokenization of text data: The server tokenizes the text data and converts it into a format suitable for the deep learning model.

[0990] 3. Noise removal: The server removes unwanted noise from the data to create a clean dataset.

[0991] Training and evaluating the model

[0992] The server trains a generative AI model using the preprocessed data. Specifically, it uses TensorFlow or PyTorch to input the collected data into the model and update the model's parameters every epoch. Once the model is fully trained, its performance is evaluated using a test dataset. Evaluation metrics include the naturalness of the dialogue and user satisfaction.

[0993] Model Deployment

[0994] Once the evaluation is complete, the model is deployed as a generative AI avatar. The server deploys the model to a cloud environment (e.g., AWS, Google Cloud) and exposes an API endpoint, allowing users to start interacting with the generative AI avatar.

[0995] User interaction

[0996] The user interacts with the generated AI avatar. Based on the user's input, the avatar generates an appropriate response. Here is an example:

[0997] Example: A user asks a generative AI avatar, "What's the weather like today?" and the server uses a generative AI model to respond, "The weather is sunny today."

[0998] Example prompt: "When a user asks a generated AI avatar about today's weather, the avatar will provide up-to-date weather information."

[0999] In this way, the generative AI avatar can generate dialogue and behaviors according to the user's needs.

[1000] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1001] Step 1: Data collection

[1002] The server collects user behavior and speech data. Input includes chat logs, voice data, and user operation history. The server records this data in real time and saves it in storage. Specifically, it captures messages and voice commands sent by users in chat applications.

[1003] Step 2: Data Preprocessing

[1004] The server preprocesses the collected data. The input includes the raw data collected in step 1. The server converts the audio data to text, tokenizes the text data, and removes noise. Specifically, it transcribes the audio data using the Google Speech-to-Text API and then tokenizes the text. The output is a preprocessed, clean dataset.

[1005] Step 3: Model training

[1006] The server trains a generative AI model using the preprocessed data. The input includes the preprocessed data from step 2. The server uses TensorFlow or PyTorch to input the data into the model and update the model parameters every epoch. Specifically, the server trains the model using a deep learning algorithm (e.g., Transformer). The output is a trained generative AI model.

[1007] Step 4: Model evaluation

[1008] The server evaluates the performance of the trained model. The inputs include the model trained in step 3 and a test dataset. The server measures the model's predictive accuracy using the test data and collects and evaluates feedback from users. Specific operations include using the naturalness of the dialogue and user satisfaction as evaluation metrics. The output is the evaluation result.

[1009] Step 5: Model deployment

[1010] The server deploys the evaluated model as a generative AI avatar. The input includes the model evaluated in step 4. The server deploys the model to a cloud environment (e.g., AWS, Google Cloud) and exposes an API endpoint. Specific operations include uploading the model to the cloud server and configuring the API. The output makes the generative AI avatar available.

[1011] Step 6: Interact with the user

[1012] The user interacts with a generative AI avatar. Input includes the user's text and voice input. The server uses the generative AI model to generate a response based on the user's input. Specifically, when the user types "What's the weather like today?", the server responds with "It's sunny today" based on the latest weather information. The output is an appropriate response for the user.

[1013] (Application example 2)

[1014] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1015] Conventional metaverse spaces and generative AI avatars are limited in their ability to interact with users and generate behavior, and lack concrete means to improve the shopping experience in physical stores. Another issue is the difficulty of learning from users' actions and comments and responding appropriately in real time. Furthermore, effective means are needed to recommend products, provide in-store guidance, and provide coupons to users in physical stores.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generated AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans in conjunction with identification information, means for easily distinguishing between the generated AI avatar and the human avatar, means for engaging in natural conversation with the user in a physical store, recommending products, providing store guidance, and offering coupons, and means for the user to interact with the generated AI avatar through an application installed on a smartphone. This allows users to improve their shopping experience in physical stores.

[1017] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1018] A "generative AI avatar" is a virtual character that is generated using artificial intelligence technologies such as deep learning and reinforcement learning and can converse and act like a human.

[1019] A "human avatar" is a digital character that represents the user in a virtual space and reflects the user's actions and statements.

[1020] "Means of coexistence" is a mechanism that allows generated AI avatars and human avatars to exist simultaneously within the metaverse space and influence each other.

[1021] "Means of communicating through text or smart speakers" refers to a method by which users can interact with the generated AI avatar using text input or voice input.

[1022] "Chat, advice on everyday life issues, various data generation, etc." refers to the various services and information that the generated AI avatar provides through dialogue with the user.

[1023] "Means of identification in association with identification information" refers to a method of recognizing a user by associating them with specific identification information (e.g., My Number card).

[1024] "Measures to make it easier to distinguish" are methods to make it easier to distinguish between generated AI avatars and human avatars.

[1025] "A means of having natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons" refers to a method in which a generated AI avatar interacts with users in physical stores and provides appropriate product information and services.

[1026] "Means for a user to interact with the generated AI avatar through an application installed on a smartphone" refers to a method for a user to communicate with a generated AI avatar through an application on a smartphone.

[1027] This invention is a system for engaging in natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons. This system includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for collectively receiving the benefits that generative AI brings to humanity, such as casual conversation, advice on daily life concerns, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing them from the generated AI avatar, means for engaging in natural conversations with users in physical stores, recommending products, providing in-store guidance, and offering coupons, and means for the user to interact with the generated AI avatar through an application installed on a smartphone.

[1028] The server generates a metaverse space where generated AI avatars and human avatars coexist. Users can interact with the generated AI avatars via text and voice through an application installed on their smartphones. This interaction is generated using the OpenAI API. The generated AI avatar learns from the user's actions and statements and generates interactions and actions according to the user's needs.

[1029] Specifically, if a user asks, "What are the recommended products in this store?", the generated AI avatar will respond, "This week's recommendation is a new smartphone. There are also special discounts, so be sure to check it out." In this way, users can improve their shopping experience in physical stores.

[1030] The hardware used is a smartphone, and the software used is Python and the OpenAI API. Data processing and calculations are performed by taking user input, sending it as a prompt to a generative AI model, and returning the generated response to the user.

[1031] An example of a prompt is as follows:

[1032] User: What products do you recommend at this store?

[1033] AI Avatar: This week's picks are new smartphones, and we've got special deals for you to check out.

[1034] In this way, a system can be implemented to enhance the shopping experience in physical stores.

[1035] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1036] Step 1:

[1037] The user launches a smartphone application and begins interacting with the generated AI avatar. The user then asks questions or makes requests via text input or voice input. The input data is sent to the application in text format.

[1038] Step 2:

[1039] The device (smartphone) receives user input data and sends it to the generative AI model as a prompt. Specifically, it generates a prompt using the OpenAI API and makes an API request. The input data is the user's question or request, and the output data is the prompt sent to the generative AI model.

[1040] Step 3:

[1041] The server uses a generative AI model (OpenAI API) to generate a response based on the prompt. The generative AI model uses deep learning and reinforcement learning to generate an appropriate response to the user's input. The input data is the prompt, and the output data is the generated response.

[1042] Step 4:

[1043] The server returns the generated response to the terminal (smartphone). Specifically, it sends the generated response to the smartphone as an API response. The input data is the generated response, and the output data is the response sent to the smartphone.

[1044] Step 5:

[1045] The device (smartphone) displays the received response to the user. Specifically, it displays the response in text format on the screen or plays the response aloud using speech synthesis technology. The input data is the response received from the server, and the output data is the response that is displayed or played to the user.

[1046] Step 6:

[1047] The user checks the response from the generated AI avatar and asks further questions or requests as necessary, thus continuing the dialogue. The input data is the user's new question or request, and the output data is sent again to the generative AI model as a prompt.

[1048] In this way, users can interact with the generated AI avatar to enhance their shopping experience in physical stores.

[1049] Example 3

[1050] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1051] In conventional metaverse spaces, it was difficult for users to freely customize their own avatars, and there was a lack of a way to easily distinguish between generated AI avatars and human avatars. This limited the user experience and could lead to security and privacy issues. Furthermore, the lack of identity verification through linkage with personal identification cards meant that user safety was not adequately ensured.

[1052] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a means for generating a metaverse space, a means for generating a "generated AI avatar" whose intelligence is a generated AI, a means for generating a human avatar, a means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, a means for communicating with the "generated AI avatar" via a text or voice input device, a means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, a means for identifying humans in conjunction with a personal identification card, a means for easily distinguishing between humans and generated AI avatars, a means for users to customize their own avatars, and a means for saving the customized avatars. This allows users to freely customize their own avatars and easily distinguish between generated AI avatars and human avatars. Furthermore, identity verification using a personal identification card ensures user safety.

[1053] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1054] A "generative AI avatar" is a virtual character that has been given intelligence using generative AI technology, and is an avatar that can interact and take action with the user.

[1055] A "human avatar" is a digital character that represents the user in a virtual space and is a user-customizable avatar.

[1056] "Personal identification card" means an official means of identification for verifying a user's identity, and includes, for example, a My Number card.

[1057] "Customization" refers to the ability of users to freely change the appearance and attributes of their avatar.

[1058] An "audio input device" is a device that converts a user's voice into a digital signal and inputs it into the system.

[1059] A "session" refers to a unit of a series of operations and communications from the time a user logs in to the time they log out of the system.

[1060] A "database" is a digital storage device for storing and managing user information, avatar customization information, etc.

[1061] "Real-time" refers to immediate response to user operations and input.

[1062] "Identity verification" is the process of officially verifying a user's identity using a personal identification card or similar.

[1063] This invention provides a system that allows users to freely customize generated AI avatars and human avatars that coexist in the metaverse space. Specific embodiments of this system are described below.

[1064] System configuration

[1065] The server generates the metaverse space and makes it accessible to users. The server uses high-performance computing hardware (e.g., cloud servers) to build the metaverse space using 3D modeling tools (e.g., Blender) and user interfaces (e.g., React.js).

[1066] The terminal provides an interface for users to access the metaverse space and customize their avatars. The terminal transmits user input to the server and displays responses from the server to the user.

[1067] Users access the system through a web browser and log in. They are authenticated by entering their email address and password. If authentication is successful, the user can proceed to the avatar customization screen.

[1068] Avatar Customization

[1069] The customization screen allows the user to select options for face shape, body type, clothing, etc. For example, to change hairstyle, the user clicks on the "Hairstyle" option and selects a new hairstyle from the list.

[1070] The terminal transmits the user's selections to the server in real time.

[1071] The server updates the avatar's 3D model based on the received selection information and sends the updated 3D model data to the device.

[1072] The device displays the updated avatar to the user.

[1073] Save your avatar

[1074] Once the user has completed the customization, he or she clicks the "Save" button.

[1075] The terminal sends a save request to the server.

[1076] The server saves the user's customization information in the database and sends a confirmation message to the device when the saving is complete.

[1077] The terminal displays a confirmation message to the user.

[1078] Identification through linkage with My Number Card

[1079] The user holds their My Number card over the card reader.

[1080] The terminal reads the information on the My Number card through a card reader and sends the read information to the server.

[1081] The server verifies the user's identity based on the information on the My Number card it receives, by checking the information on the My Number card against a database to see if there is a match.

[1082] If the identity verification is successful, the server creates or updates the user's account and stores the account information in a database.

[1083] The terminal will display a successful account creation or update message to the user.

[1084] Specific examples

[1085] For example, if a user wants to change the hairstyle of their avatar, they would follow these steps:

[1086] 1. The user clicks on the "Hairstyle" option on the customization screen.

[1087] 2. The device sends the user's selection to the server.

[1088] 3. The server retrieves the 3D model data of the selected hairstyle and applies it to the avatar.

[1089] 4. The device displays the updated avatar to the user.

[1090] Additionally, when a user creates a new account, they follow these steps:

[1091] 1. The user selects the "Authenticate with My Number Card" option on the account creation screen.

[1092] 2. The terminal instructs the user to hold their My Number card over the card reader.

[1093] 3. The user holds their My Number card over the card reader.

[1094] 4. The terminal sends the information read from the My Number card to the server.

[1095] 5. The server verifies the user's identity based on the information received and creates an account.

[1096] 6. The terminal displays a success message to the user.

[1097] Prompt Sentence Examples

[1098] Examples of prompts to input to a generative AI model might include:

[1099] "Please explain what steps a user would take if they wanted to change their hairstyle."

[1100] "Please explain how to verify a user's identity using the My Number card."

[1101] This makes it easier for the generative AI model to understand specific procedures and processing contents. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1102] Step 1: User Login

[1103] The user opens a web browser and accesses the system's login page. The user enters their email address and password and clicks the login button. The terminal receives the user's input and sends it to the server. The server compares the received email address and password with a database and performs authentication. If authentication is successful, the user's session is started, and data for the customization screen is generated and sent to the terminal. The terminal displays the customization screen to the user.

[1104] Input: User's email address and password

[1105] Output: Authentication results and customization screen data

[1106] Step 2: Display the avatar customization screen

[1107] The server generates an avatar customization screen based on the user's session information. The device displays the customization screen received from the server to the user. The user selects options such as face shape, body type, and clothing on the customization screen.

[1108] Input: User session information

[1109] Output: Customization screen data

[1110] Step 3: Customize your avatar

[1111] The user selects options such as face shape, body type, and clothing on the customization screen. For example, to change their hairstyle, the user clicks the "Hairstyle" option and selects a new hairstyle from the list. The device sends the user's selection to the server in real time. The server updates the avatar's 3D model based on the received selection information. The updated 3D model data is sent to the device. The device then displays the updated avatar to the user.

[1112] Input: User's customized selection

[1113] Output: Updated avatar 3D model data

[1114] Step 4: Save your avatar

[1115] When the user has completed customization, he / she clicks the "Save" button. The terminal sends a save request to the server. The server saves the user's customization information in a database. When saving is complete, a confirmation message is sent to the terminal. The terminal displays the confirmation message to the user.

[1116] Input: Save request

[1117] Output: Save completion confirmation message

[1118] Step 5: Scan your My Number card

[1119] The user holds their My Number card over the card reader. The terminal reads the My Number card information through the card reader and sends the read information to the server.

[1120] Input: My Number card information

[1121] Output: Information from the scanned My Number card

[1122] Step 6: Identity Verification

[1123] The server verifies the user's identity based on the received My Number card information. Specifically, it compares the My Number card information with the database to see if there is a match. If identity verification is successful, the server creates or updates the user's account. The account information is saved in the database.

[1124] Input: My Number card information

[1125] Output: Identity verification results and account information

[1126] Step 7: Account Creation or Update Notification

[1127] The terminal will display a successful account creation or update message to the user.

[1128] Input: Account creation or update success message

[1129] Output: User notification

[1130] (Application example 3)

[1131] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1132] In today's virtual world, when users customize their own avatars and try on clothes and select products in virtual stores, there is a lack of means to verify the user's identity and ensure their safety. There is also a need for a way to easily distinguish between AI-generated avatars and human avatars. This will improve the user experience while providing a safe and reliable virtual shopping environment.

[1133] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generating AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generating AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans through linkage with My Number cards, means for easily distinguishing between humans and generated AI avatars, means for users to customize their own avatars, means for using the avatar to try on clothes and select products in a virtual store, and means for verifying identity using My Number cards. This allows users to customize their own avatars and try on clothes and select products in a virtual store in a safe and reliable virtual shopping environment.

[1134] A "metaverse space" is a three-dimensional virtual space generated using virtual reality technology, in which users can interact through avatars.

[1135] A "generative AI avatar" is a virtual character that has been given intelligence using generative AI technology and is capable of interacting with and acting in ways that interact with users.

[1136] A "human avatar" is a customizable virtual character that allows users to represent themselves in a virtual space.

[1137] The "My Number Card" is a personal identification number card used in Japan to verify personal identity and for various administrative procedures.

[1138] A "virtual store" is an online shopping environment created using virtual reality technology, allowing users to browse and purchase products in a virtual space.

[1139] "Customization" refers to the ability of users to freely set and change the appearance and clothing of their avatar.

[1140] "Identity verification" is the process of verifying a user's personal information and proving that the person is a legitimate user.

[1141] "Trying on" refers to a user's avatar virtually trying on clothes and accessories in a virtual store.

[1142] "Generative AI" is a system that uses artificial intelligence technology to generate data and interact with and take action on behalf of users.

[1143] "Coexistence" refers to a state in which multiple avatars exist simultaneously within the metaverse space and influence each other.

[1144] A system for carrying out this invention has the following configuration: The server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for receiving all the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with an Individual Number card, means for making it easy to distinguish from a generated AI avatar, means for users to customize their own avatar, means for using an avatar to try on clothes and select products in a virtual store, and means for verifying identity using an Individual Number card.

[1145] Hardware and software used

[1146] Hardware: Smartphones, smart glasses, servers

[1147] Software: OpenCV (image processing library), AvatarCustomizer (avatar customization library), MyNumberVerifier (My Number Card verification library)

[1148] Data processing and calculation

[1149] 1. Avatar Customization:

[1150] Users use their smartphones or smart glasses to take a photo of themselves and customize their avatar.

[1151] Use the AvatarCustomizer library to configure the user's face, body type, and clothing.

[1152] Use OpenCV to load a user's face image and apply it to an avatar.

[1153] 2. Check your My Number Card:

[1154] Users scan their My Number card with their smartphone camera to verify their identity.

[1155] Use the MyNumberVerifier library to verify the My Number card image and number.

[1156] Use OpenCV to read an image of the My Number card, extract the number, and confirm it.

[1157] 3. Try-on and product selection in the virtual store:

[1158] Users use their customized avatars to try on clothes and select products in a virtual store.

[1159] Software is used to simulate avatar movement and fitting within a virtual store.

[1160] Specific examples

[1161] Users open the app on their smartphone, take a photo of themselves, and customize their avatar. They then scan their My Number card with the smartphone camera to verify their identity. Once successfully verified, they can use the avatar to try on clothes and select products in a virtual store.

[1162] Prompt Sentence Examples

[1163] Users take a photo of themselves with their smartphone to customize their avatar, then scan their My Number card to verify their identity. Once successfully verified, they can use their avatar to try on clothes and select products in a virtual store.

[1164] In this way, users can customize their avatars, try on clothes and select products in virtual stores in a safe and reliable virtual shopping environment.

[1165] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1166] Step 1:

[1167] The user launches the app on their smartphone and takes a photo of their face. The input is the user's face photo, and the output is face photo data. The face photo is taken using the smartphone's camera function and saved as image data.

[1168] Step 2:

[1169] The server receives the facial photo data and customizes the avatar using the AvatarCustomizer library. The input is the facial photo data and the output is the customized avatar. The server analyzes the facial photo data, extracts the user's facial features, and applies them to the avatar.

[1170] Step 3:

[1171] The user scans their My Number card with their smartphone camera. The input is an image of the My Number card, and the output is the scanned card image data. The smartphone's camera function is used to take a photo of the My Number card and save it as image data.

[1172] Step 4:

[1173] The server receives the image data of the My Number card and performs identity verification using the MyNumberVerifier library. The input is the image data of the My Number card, and the output is the result of identity verification. The server analyzes the image data, extracts the card number, and compares it with the database.

[1174] Step 5:

[1175] The server notifies the user of the result of identity verification. The input is the result of identity verification, and the output is a notification message. If the verification is successful, the server sends the user a success message; if it fails, it sends a message encouraging the user to try again.

[1176] Step 6:

[1177] A user accesses a virtual store using a customized avatar. The input is the customized avatar, and the output is access rights to the virtual store. The user logs in to the virtual store through the app and moves around the store using the avatar.

[1178] Step 7:

[1179] Users use an avatar in a virtual store to try on clothes and select products. The input is the user's operation data, and the output is the results of the try-on and product selection. The user operates the avatar to select products and simulates trying on clothes.

[1180] Step 8:

[1181] The server saves the results of the user's try-on and product selection, and performs the purchase process as necessary. The input is the results of the try-on and product selection, and the output is a notification that the purchase process has been completed. The server records the user's selection, and if the purchase is confirmed, it processes the payment.

[1182] Through these steps, users can customize their own avatars in a safe and reliable virtual shopping environment, and try on clothes and select products in a virtual store.

[1183] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1184] "Example 1"

[1185] One form of invention combining an emotion engine is a system including an emotion engine that recognizes emotions from the user's tone of voice, facial expressions, behavior, etc. In this system, the emotion engine recognizes the user's emotions in real time and adjusts the dialogue and behavior of the generated AI avatar according to those emotions. For example, if the user expresses joy, the generated AI avatar will speak in a bright tone and generate cheerful behavior. Conversely, if the user expresses sadness, the generated AI avatar will speak in a calm tone and generate comforting behavior.

[1186] "Example 2"

[1187] Another possible system is one in which the emotion engine detects changes in a user's emotions in real time and dynamically changes the environment of the metaverse space in response to those changes. For example, if the user is expressing joy, the metaverse space will change to a bright and vibrant environment. Conversely, if the user is expressing sadness, the metaverse space will change to a quiet and calm environment. This will make it possible to provide the optimal metaverse space according to the user's emotions.

[1188] "Example 3"

[1189] Furthermore, it is conceivable that a system could use an emotion engine to recognize the user's emotions and adjust the dialogue and behavior of the generated AI avatar according to those emotions, as well as change the facial expression and tone of voice of the generated AI avatar itself according to the user's emotions. For example, if the user expresses happiness, the generated AI avatar could create a smiling facial expression and speak in a bright tone of voice. Conversely, if the user expresses sadness, the generated AI avatar could create a sad facial expression and speak in a calm tone of voice. This would make it possible to provide dialogue that is more adapted to the user's emotions.

[1190] "Example 4"

[1191] In addition, a system could be envisioned that not only uses an emotion engine to recognize the user's emotions and adjust the generated AI avatar's dialogue and behavior according to those emotions, but also allows the generated AI avatar to learn and evolve its own behavioral patterns according to the user's emotions. For example, if the generated AI avatar performs a specific action when the user expresses joy and the user's reaction is positive, the generated AI avatar will learn that behavior and reproduce it when a similar situation arises. This allows the generated AI avatar to learn and evolve behaviors that are more adapted to the user's emotions.

[1192] The processing flow of each embodiment will be described below.

[1193] "Example 1"

[1194] Step 1: Activate the emotion engine that recognizes emotions from the user's tone of voice, facial expressions, behavior, etc.

[1195] Step 2: The emotion engine recognizes the user's emotions in real time.

[1196] Step 3: Adjust the dialogue and behavior of the generated AI avatar based on the recognized emotions.

[1197] "Example 2"

[1198] Step 1: Launch an emotion engine that detects changes in user emotions in real time.

[1199] Step 2: The emotion engine detects changes in the user's emotions in real time.

[1200] Step 3: Dynamically change the environment of the metaverse space according to changes in detected emotions.

[1201] "Example 3"

[1202] Step 1: Activate the emotion engine that recognizes the user's emotions.

[1203] Step 2: The emotion engine recognizes the user's emotions in real time.

[1204] Step 3: Adjust the dialogue, behavior, facial expressions, and tone of voice of the generated AI avatar based on the recognized emotions.

[1205] "Example 4"

[1206] Step 1: Activate the emotion engine that recognizes the user's emotions.

[1207] Step 2: The emotion engine recognizes the user's emotions in real time.

[1208] Step 3: Adjust the dialogue and behavior of the generated AI avatar based on the recognized emotions.

[1209] Step 4: Learn and evolve the behavioral patterns of the generated AI avatar based on user responses.

[1210] Example 1

[1211] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1212] In conventional metaverse spaces, it was difficult to generate dialogue and behavior based on the user's emotions and behavior, limiting the user experience. Furthermore, there was a lack of technology to recognize the user's emotions in real time and adjust the avatar's dialogue and behavior accordingly. Furthermore, it was difficult to distinguish between the user and the generated AI avatar, making it difficult to identify the user.

[1213] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a means for generating a metaverse space, a means for generating a "generated AI avatar" whose intelligence is a generated AI, a means for generating a human avatar, a means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, a means for communicating with the "generated AI avatar" through text or voice, a means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, a means for identifying humans, a means for easily distinguishing between the generated AI avatar and the human avatar, a means for collecting the user's tone of voice, facial expressions, and behavior in real time, a means for analyzing the collected data and recognizing the user's emotions, and a means for adjusting the dialogue and behavior of the generated AI avatar based on the recognized emotions. This enables the generation of dialogue and behavior in accordance with the user's emotions and behavior, providing a richer user experience. It also makes it easier to distinguish between the user and the generated AI avatar, ensuring reliable user identification.

[1214] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, and is an environment in which users can move and act freely.

[1215] A "generative AI avatar" is a virtual character that has generative AI as its intelligence and can interact and act with the user.

[1216] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[1217] "Means of coexistence" is a mechanism that allows generated AI avatars and human avatars to exist simultaneously within the metaverse space and influence each other.

[1218] "Means of communicating through text or voice" refers to a method by which a user can communicate with a generated AI avatar using text input or voice input.

[1219] "Chat, advice on everyday life issues, various data generation, etc." are various services provided by the generation AI, and are functions to support users' daily lives.

[1220] "Means of human identification" refers to a method for identifying a user and distinguishing them from other users and generated AI avatars.

[1221] "Means to make it easy to distinguish from generated AI avatars" refers to a mechanism that makes it easy to distinguish between users and generated AI avatars.

[1222] "Means for collecting tone of voice, facial expressions, and behavior in real time" refers to a method for acquiring and analyzing a user's audio and video data in real time.

[1223] "Means for analyzing collected data and recognizing the user's emotions" refers to technology for analyzing acquired audio and video data to determine the user's emotional state.

[1224] "Means for adjusting the dialogue and behavior of the generated AI avatar based on the recognized emotions" refers to a method for changing the dialogue content and behavior of the generated AI avatar according to the user's emotional state.

[1225] MODE FOR CARRYING OUT THE INVENTION

[1226] This invention is a system that generates a metaverse space and adjusts the dialogue and behavior of the generated AI avatar according to the user's emotions. Specific embodiments of this system are described below.

[1227] Creating a Metaverse Space

[1228] The server uses 3D graphics technology (e.g., Unity or Unreal Engine) to generate the Metaverse space, a virtual environment in which users can move and act freely. The server loads 3D models, textures, and a physics engine to create virtual cities and natural environments that users can explore.

[1229] Generate AI avatars

[1230] The server generates a "generative AI avatar" whose intelligence is the generative AI. This avatar is a virtual character that interacts with and behaves with the user. The server uses a generative AI model (e.g., OpenAI GPT-4) to generate the avatar's dialogue and behavior.

[1231] Collecting user sentiment data

[1232] The device collects the user's tone of voice, facial expressions, and behavior in real time. Specifically, the device uses a microphone and a camera to capture the user's audio and video data. For example, the microphone collects the user's voice while the camera captures the user's facial expressions.

[1233] Emotional Data Analysis

[1234] The device sends the collected audio and video data to an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API) to analyze the user's emotions. The emotion engine analyzes changes in voice tone and facial expressions to determine whether the user is happy or sad.

[1235] Avatar dialogue and behavior generation

[1236] The server uses the generative AI model to generate the avatar's dialogue and behavior based on the emotion data received from the emotion engine. For example, if the user expresses joy, the server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate joyful behavior."

[1237] User feedback

[1238] The server reflects the generated dialogue and actions in the avatar and provides feedback to the user. Specifically, the avatar speaks to the user in a cheerful tone and makes fun movements. For example, the avatar might say in a cheerful tone, "Hello! What would you like to do for fun today?"

[1239] Specific examples

[1240] The user puts on a VR headset and enters the metaverse space. The device collects the user's voice and facial expressions and sends them to the emotion engine. The emotion engine analyzes the user's emotions and detects the emotion of "happiness." The server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate fun actions." The avatar then speaks in a bright tone, saying, "Hello! What would you like to do for fun today?"

[1241] Prompt Sentence Examples

[1242] 1. If the user is expressing happiness:

[1243] "Your users are happy, so interact with them in a light-hearted tone and generate pleasurable behavior."

[1244] 2. If the user expresses sadness:

[1245] "The user is sad, so speak in a calm tone and generate comforting actions."

[1246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1247] Step 1: Initialize the Metaverse Space

[1248] The server initializes the metaverse space using 3D graphics technology (e.g., Unity or Unreal Engine). As input, it loads 3D models, textures, and physics engine data. This generates a virtual environment in which the user can move and act freely. Specifically, the server builds a virtual city or natural environment, allowing the user to explore it.

[1249] Step 2: Collecting user sentiment data

[1250] The device collects the user's tone of voice, facial expressions, and behavior in real time. As input, it uses a microphone and a camera to capture the user's audio and video data, which provides data to understand the user's emotional state. Specifically, the device collects the user's speaking voice with a microphone and captures the user's facial expressions with a camera.

[1251] Step 3: Analyze the emotion data

[1252] The device sends the collected audio and video data to an emotion engine (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API) to analyze the user's emotions. The collected audio and video data is used as input. This provides emotional data to determine whether the user is happy or sad. Specifically, the emotion engine analyzes changes in voice tone and facial expressions to detect the user's emotional state.

[1253] Step 4: Generating Avatar Dialogue and Behavior

[1254] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate the avatar's dialogue and behavior based on the emotion data received from the emotion engine. The emotion data and prompt text are used as input. This generates dialogue content and behavior that corresponds to the user's emotions. Specifically, the server instructs the generative AI model, "The user is happy, so please speak in a bright tone and generate fun behavior."

[1255] Step 5: User feedback

[1256] The server reflects the generated dialogue and actions in the avatar and provides feedback to the user. The generated dialogue content and behavior data are used as input. This allows the avatar to engage in appropriate dialogue and actions with the user. A specific example of this behavior is the avatar speaking in a cheerful tone, saying, "Hello! What would you like to do for fun today?"

[1257] (Application example 1)

[1258] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1259] In conventional metaverse spaces, the user experience was limited due to a lack of functionality to recognize users' emotions in real time and adjust the avatar's dialogue and behavior accordingly. Furthermore, virtual stores lacked the functionality to generate dialogue and behavior according to emotions when users browse and purchase products, making it difficult to increase user satisfaction.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generated AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans in conjunction with identification information, means for easily distinguishing between humans and generated AI avatars, means for recognizing a user's emotions in real time and adjusting the dialogue and behavior of the generated AI avatar according to those emotions, and means for generating dialogue and behavior according to emotions when the user browses or purchases products in a virtual store. This enables dialogue and behavior according to the user's emotions to be generated in real time, improving the user experience in the virtual store.

[1261] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[1262] A "generative AI avatar" is a virtual character that has generative AI as its intelligence and can interact and act with the user.

[1263] A "human avatar" is a character that represents the user in a virtual space.

[1264] The "emotion engine" is a system that recognizes emotions from the user's tone of voice, facial expressions, and behavior, and adjusts the dialogue and behavior of the generated AI avatar according to those emotions.

[1265] A "virtual store" is a store set up in a virtual space where users can browse and purchase products.

[1266] "Identification information" refers to information for identifying a user, and includes personal identification information such as a My Number card.

[1267] "Real-time" refers to the system instantly recognizing the user's actions and emotions and responding accordingly.

[1268] "Dialogue" means that the user and the generated AI avatar exchange words and communicate.

[1269] "Behavior" refers to the actions and reactions that the generated AI avatar takes in response to the user's emotions and actions.

[1270] "Chat" is when the user casually discusses everyday topics with the generated AI avatar.

[1271] "Problem consultation" means that users consult with a generated AI avatar about worries and problems they face in their daily lives.

[1272] "Various data generation" means that the generation AI generates various data in response to user requests.

[1273] The system for implementing this invention generates a metaverse space, recognizes the user's emotions in real time, and adjusts the dialogue and behavior of the generated AI avatar. Specific embodiments are described below.

[1274] System configuration

[1275] The system uses the following hardware and software:

[1276] Hardware: Smartphone, head-mounted display, camera

[1277] Software: OpenCV, dlib, MetaverseSDK, EmotionRecognitionModel

[1278] Program processing explanation

[1279] Creating a Metaverse Space

[1280] The server uses 3D graphics and VR technology to generate the metaverse space, which is designed to allow users to move and act freely.

[1281] emotion recognition

[1282] The user's face is captured using a camera on the device (smartphone or head-mounted display). The face is detected using OpenCV and dlib, and facial feature points are extracted. The extracted feature points are input into the EmotionRecognitionModel, and the user's emotions are recognized in real time.

[1283] Coordinating the dialogue and behavior of generative AI avatars

[1284] The server adjusts the generated AI avatar's dialogue and behavior according to the recognized emotion. For example, if the user expresses joy, the generated AI avatar will speak in a bright tone and generate joyful behavior. Conversely, if the user expresses sadness, the generated AI avatar will speak in a calm tone and generate comforting behavior.

[1285] Application in virtual stores

[1286] When a user browses and purchases products in a virtual store, the system generates dialogue and behaviors based on their emotions. For example, if a user shows interest in a product, the generated AI avatar will respond in a cheerful tone, saying, "This product is very popular!". If the user appears depressed, the generated AI avatar will respond in a gentle tone, saying, "Is there something I can help you with? Please let me know if there's anything I can help you with."

[1287] Examples of concrete examples and prompts

[1288] Specific examples

[1289] As users browse products in a virtual store, an avatar cheerfully tells them, "This product is very popular!"

[1290] If the user appears depressed, the avatar will speak to them in a gentle tone, asking, "Is there something I can help you with? Please let me know if there's anything I can help you with."

[1291] Prompt Sentence Examples

[1292] "Generate a scenario where when the user smiles, the avatar speaks in a cheerful tone."

[1293] "Generate a scenario where, when the user makes a sad expression, the avatar will have a comforting conversation."

[1294] The above is a specific embodiment for carrying out the present invention.

[1295] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1296] Step 1:

[1297] The server generates the Metaverse space using 3D graphics and VR technology. It receives the design data of the virtual space as input and provides the virtual space in which users can move and act freely as output. Specifically, the server analyzes the design data and renders a 3D model to construct the virtual space.

[1298] Step 2:

[1299] The device uses a camera to capture the user's face. It receives real-time video data as input and generates facial image data as output. Specifically, the device acquires video data from the camera and cuts out the facial area.

[1300] Step 3:

[1301] The device uses OpenCV and dlib to detect faces and extract facial feature points. It receives facial image data as input and generates facial feature point data as output. Specifically, the device applies a face detection algorithm to identify facial feature points.

[1302] Step 4:

[1303] The device inputs the extracted feature points into the EmotionRecognitionModel to recognize the user's emotions in real time. It receives facial feature point data as input and generates emotion data as output. Specifically, the device inputs the feature point data into the model and applies an emotion recognition algorithm to identify the emotion.

[1304] Step 5:

[1305] The server adjusts the dialogue and behavior of the generated AI avatar according to the recognized emotions. It receives emotion data as input and generates dialogue and behavior data for the avatar as output. Specifically, the server analyzes the emotion data and inputs prompts to the generative AI model to generate appropriate dialogue and behavior.

[1306] Step 6:

[1307] The server reflects the generated dialogue and behavioral data to the generated AI avatar in the metaverse space. It receives the dialogue and behavioral data as input and controls the avatar's behavior as output. Specifically, the server sends the dialogue and behavioral data to the avatar's control system and updates the avatar's behavior.

[1308] Step 7:

[1309] When browsing and purchasing products in a virtual store, users experience emotionally appropriate interactions and behaviors. The input is the avatar's interactions and behaviors, and the output is increased user satisfaction. Specifically, users interact with the avatar to obtain product information and complete the purchase process.

[1310] Example 2

[1311] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1312] Conventional metaverse spaces and generative AI avatar systems have not adequately generated dynamic environmental changes and dialogue in response to the user's emotions and behavior. Furthermore, it is difficult to distinguish between generative AI avatars and human avatars, resulting in a poor user experience. Furthermore, there are issues with inefficient training and evaluation of generative AI models, resulting in suboptimal system performance.

[1313] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generating AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" through a text or voice input device, means for receiving all the benefits that the generating AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with identification information, means for making it easy to distinguish between the human and the generated AI avatar, means for detecting the user's emotions in real time, means for dynamically changing the environment of the metaverse space in response to changes in the emotions, means for training a generating AI model using a high-performance computing device, and means for evaluating the generating AI model and adjusting parameters as necessary. This enables the generation of dynamic environmental changes and dialogue in response to the user's emotions and behavior, makes it easier to distinguish between generated AI avatars and human avatars, and optimizes the performance of the entire system.

[1314] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1315] A "generative AI avatar" is a virtual character created using generative AI technology that has the intelligence to engage in natural conversation and dialogue with the user.

[1316] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[1317] "Text or voice input device" refers to a device that allows a user to input text or voice, including a keyboard, microphone, smart speaker, etc.

[1318] An "emotion engine" is software or hardware that detects a user's emotions in real time and changes the system's behavior in response to those emotions.

[1319] A "high-performance computing device" is a high-performance computer for efficiently training and evaluating generative AI models, including GPUs and dedicated computing units.

[1320] A "generative AI model" is an artificial intelligence model trained using techniques such as deep learning and reinforcement learning, which interacts with users and generates data.

[1321] "Identification information" refers to information used to identify a user, including ID cards and biometric data.

[1322] "Means of dynamic change" refers to technology that allows the state of a system or environment to be changed in real time in response to the user's behavior or emotions.

[1323] "Means for adjusting parameters" refers to methods for changing parameters such as model settings and learning rates in order to optimize the performance of a generative AI model.

[1324] MODE FOR CARRYING OUT THE INVENTION

[1325] This invention is a system that generates a metaverse space and uses the generated AI avatar to interact with users. Specific embodiments of this system are described below.

[1326] System configuration

[1327] The server system is configured using the following hardware and software.

[1328] Hardware: A computing device equipped with a high-performance GPU (e.g., NVIDIA Tesla V100)

[1329] Software: Deep learning frameworks (e.g. TensorFlow, PyTorch)

[1330] Data collection and preprocessing

[1331] The server collects user behavior and speech data. This includes text and voice data entered by the user through the device, as well as sensor data such as the user's facial expressions and movements. The collected data is cleaned and converted into a format suitable for the model. For example, text data is tokenized and voice data is converted into a spectrogram.

[1332] Model training and evaluation

[1333] The server uses the preprocessed data to train a generative AI model. It efficiently trains using a deep learning framework, leveraging a high-performance GPU. After training is complete, the server evaluates the model's performance and adjusts parameters as needed.

[1334] Emotion detection and changes in the metaverse space

[1335] The server uses an emotion engine to detect the user's emotions in real time, and dynamically changes the environment of the metaverse space depending on the user's emotions. For example, if the user expresses joy, the metaverse space will change to a bright and colorful environment.

[1336] User interaction

[1337] The generated AI avatar will converse with the user. It will generate an appropriate response based on what the user says and respond to the user via the device. For example, if the user asks, "What's the weather like today?" the generated AI avatar will respond, "It's sunny today. The temperature is about 25 degrees."

[1338] Examples and prompts

[1339] Below is a specific example of a user initiating a dialogue with a generated AI avatar.

[1340] User: "What's the weather like today?"

[1341] Generated AI avatar: "It's sunny today. The temperature is about 25 degrees."

[1342] Below is a concrete example of how the emotion engine detects the user's emotions in real time and dynamically changes the environment in the metaverse space.

[1343] User: "I'm so happy today!"

[1344] Metaverse space: Transforms into a bright and vibrant environment.

[1345] An example of a prompt sentence to be input to the generative AI model is shown below.

[1346] "How do you change the metaverse space when a user is expressing joy?"

[1347] "How does a generative AI avatar respond when a user displays the emotion of sadness?"

[1348] This will enable the provision of optimal dialogue and metaverse spaces that respond to the user's emotions and behavior.

[1349] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1350] Program processing flow

[1351] Step 1: Data collection

[1352] The server collects data on user behavior and speech, including text and voice data entered by the user through the device, as well as sensor data such as the user's facial expressions and movements.

[1353] Input: Text and voice data that you enter into your device

[1354] Output: raw data collected

[1355] Specific operation: When a user types "What's the weather like today?" into the device, the server collects this text data.

[1356] Step 2: Data Preprocessing

[1357] The server cleans the collected data and converts it into a format suitable for the model, for example, by tokenizing text data or converting audio data into a spectrogram.

[1358] Input: Raw data collected

[1359] Output: Preprocessed data

[1360] Specific operation: The server breaks down the text "What's the weather like today?" into words and converts them into numerical data.

[1361] Step 3: Model training

[1362] The server uses the preprocessed data to train the generative AI model, leveraging a high-performance GPU and a deep learning framework for efficient training.

[1363] Input: Preprocessed data

[1364] Output: A trained generative AI model

[1365] Specific operation: The server trains the neural network using tokenized text data as input.

[1366] Step 4: Model evaluation

[1367] The server evaluates the performance of the trained model using metrics such as precision and recall, and adjusts parameters as needed to optimize the model's performance.

[1368] Input: A trained generative AI model

[1369] Output: Evaluation results and adjusted model

[1370] Specific operation: The server evaluates the model's response accuracy using the test data and logs the results.

[1371] Step 5: Emotion detection

[1372] The server uses an emotion engine to detect the user's emotions in real time, and dynamically changes the environment of the metaverse space according to the user's emotions.

[1373] Input: Real-time user data (text, voice, facial expressions, etc.)

[1374] Output: Detected emotion data

[1375] Specific behavior: When a user types "I'm so happy today!", the server uses the emotion engine to detect the emotion "joy."

[1376] Step 6: Changing the Metaverse Space

[1377] The server dynamically changes the environment of the metaverse space based on the detected emotion data.

[1378] Input: Detected emotion data

[1379] Output: Transformed Metaverse space

[1380] Specific operation: After detecting the emotion of "joy," the server transforms the metaverse space into a bright and colorful environment.

[1381] Step 7: Interact with the user

[1382] The generated AI avatar will converse with the user, generate appropriate responses based on what the user says, and respond to the user via the device.

[1383] Input: User utterance data

[1384] Output: The generated response

[1385] Specific behavior: When the user asks, "What's the weather like today?", the generated AI avatar replies, "It's sunny today. The temperature is about 25 degrees."

[1386] (Application example 2)

[1387] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1388] In conventional metaverse spaces, natural interactions with users and dynamic changes in the environment based on their emotions have not been fully realized. Furthermore, there is no system that can detect users' emotions in real time and change the atmosphere of the metaverse space based on that. This limits the user experience and hinders the full appeal of the metaverse space.

[1389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1390] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" via text or a smart speaker, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing them from the generative AI avatar, means for detecting user emotions in real time, means for dynamically changing the environment of the metaverse space in response to the user's emotions, means for generating natural conversational responses based on user input, and means for changing the atmosphere of the store in response to the user's emotions. This enables natural dialogue with the user, realizes dynamic changes in the metaverse space in response to the user's emotions, and provides a more engaging user experience.

[1391] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1392] A "generative AI avatar" is a virtual character created using generative AI technology, and is an intelligent avatar that can converse and act like a human.

[1393] A "human avatar" is a character that represents the user in a virtual space and moves according to the user's actions.

[1394] "Emotion detection" is a technology that analyzes a user's emotions in real time from their facial expressions, voice, etc., and identifies their emotional state.

[1395] "Natural conversational response" is a function that allows the generative AI to generate replies to user input in a natural, human-like manner.

[1396] "Dynamic environmental changes" is a function that changes elements of the virtual space, such as lighting, music, and background, in real time according to the user's emotions and actions.

[1397] "Identification information" refers to information used to identify a user, including, for example, an ID card or biometric data.

[1398] "Store atmosphere" refers to environmental elements such as lighting, music, and decorations in a store within a virtual space, and the atmosphere is adjusted by changing these elements according to the user's emotions.

[1399] The following system configuration will be described as an embodiment of the present invention.

[1400] System Configuration

[1401] The system includes a server that generates the metaverse space, a server that generates generative AI avatars, a terminal that detects user emotions in real time, a server that generates natural conversational responses based on user input, and a server that changes the atmosphere of the store according to the user's emotions.

[1402] Hardware and software used

[1403] Hardware: webcam (to capture the user's facial expressions), microphone (for voice input), display (to display the metaverse space)

[1404] Software: OpenAI API (generative AI model), OpenCV (image processing), EmotionDetector (emotion detection), MetaverseEnvironment (metaverse environment management)

[1405] Data processing and calculation

[1406] 1. User facial expression capture: Capture user facial expressions in real time using the device's webcam.

[1407] 2. Emotion detection: The captured facial expression data is analyzed using EmotionDetector to detect the user's emotions.

[1408] 3. AI response generation: The server generates natural conversational responses to user input using OpenAI's generative AI model.

[1409] 4. Dynamic Environment Change: Update the MetaverseEnvironment based on the detected emotions to change the store's atmosphere.

[1410] Specific examples

[1411] For example, if a user is looking for a new smartphone, the following dialogue might take place:

[1412] Prompt Sentence Examples

[1413] User: I'm looking for a new smartphone.

[1414] Assistant: What features are important to you? For example, camera performance, battery life, etc.

[1415] During this interaction, if the user's facial expression indicates happiness, the system will brighten the store's lights and change the background music to an upbeat tone. Conversely, if the user indicates sadness, the system will change the lighting to a more subdued color and quieten the background music.

[1416] Processing flow

[1417] The server receives user input and generates natural conversational responses using a generative AI model. The device detects the user's emotions in real time using a webcam and Emotion Detector and sends the information to the server. The server updates the Metaverse Environment based on the received emotional information, dynamically changing the store's atmosphere.

[1418] In this way, natural dialogue with the user becomes possible, and the metaverse space changes dynamically in response to the user's emotions, providing a more engaging user experience.

[1419] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1420] Step 1:

[1421] The user captures facial expressions through the webcam on the device. The input is the user's real-time facial expression data, and the output is the captured image data. The device sends this image data to EmotionDetector.

[1422] Step 2:

[1423] The device uses EmotionDetector to analyze the captured facial expression data and detect the user's emotions. The input is the captured image data, and the output is the detected emotional information (e.g., joy, sadness, surprise, etc.). The device sends this emotional information to the server.

[1424] Step 3:

[1425] The user inputs text or voice through the terminal. The input is the user's text or voice data, and the output is the user input in text format. The terminal sends this user input to the server.

[1426] Step 4:

[1427] The server uses OpenAI's generative AI model to generate natural-sounding conversational responses based on the user's input. The input is the user's textual input data, and the output is the generated conversational response. The server then sends this conversational response to the device.

[1428] Step 5:

[1429] The terminal displays or plays audibly the conversational response received from the server to the user. The input is the generated conversational response, and the output is a display or audio output to the user. The user can then provide further input.

[1430] Step 6:

[1431] The server updates the MetaverseEnvironment based on the emotion information sent from the device, dynamically changing the store's atmosphere. The input is the detected emotion information, and the output is the updated Metaverse environment settings. The server sends this setting to the device.

[1432] Step 7:

[1433] The terminal changes the store atmosphere based on the updated metaverse environment settings received from the server. The input is the updated metaverse environment settings, and the output is the changed store atmosphere (e.g., lighting color, background music, etc.). The user can continue to interact with this changed environment.

[1434] Example 3

[1435] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1436] In conventional metaverse spaces, users have limited means to freely customize their own avatars, and there is a lack of means to easily distinguish between generated AI avatars and human avatars. Furthermore, it is difficult to generate dialogue and behavior that reflects the user's emotions, and there are insufficient means to improve the user experience. This leads to issues such as reduced user satisfaction and limited use of the metaverse space.

[1437] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes means for generating a metaverse space, means for generating an intelligent generated AI avatar, means for generating a human avatar, means for the generated AI avatar and the human avatar to coexist in the metaverse space, means for communicating with the generated AI avatar through text or voice, means for receiving all the benefits that the generated AI brings to humanity, such as casual conversation, advice on daily life worries, and various data generation, means for identifying humans in conjunction with a personal identification card, means for making it easy to distinguish between humans and generated AI avatars, means for a user to upload a photo of their face and generate an avatar using image processing technology, means for analyzing the user's emotions and adjusting the dialogue and behavior of the generated AI avatar according to the emotions, and means for recording the user's reactions and learning the behavior patterns of the generated AI avatar using a reinforcement learning algorithm. This allows users to freely customize their own avatars, making it easy to distinguish between generated AI avatars and human avatars, and also makes it possible to provide dialogue and behavior that responds to the user's emotions.

[1438] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1439] A "generated AI avatar" is a virtual character generated using artificial intelligence technology, which has intelligence and can interact and act with the user.

[1440] A "human avatar" is a character that represents the user in a virtual space, and is a virtual alter ego that can be customized by the user.

[1441] A "personal identification card" is a card used to verify a user's identity, such as an IC card that contains personal information, such as a My Number card.

[1442] "Image processing technology" is a technology for analyzing and processing digital images, and is a technology for performing processes such as facial recognition and feature extraction.

[1443] "Emotion analysis" is a technology that analyzes input data such as a user's facial expressions and voice to identify the user's emotional state.

[1444] A "reinforcement learning algorithm" is a machine learning algorithm that allows an agent to learn optimal behavior through interaction with the environment.

[1445] "Dialogue adjustment" is the process of appropriately changing the dialogue content and behavior of the generated AI avatar according to the user's emotions and situation.

[1446] "Behavioral pattern learning" is the process by which a generative AI avatar learns optimal behavior based on the user's reactions and reproduces appropriate behavior in similar situations in the future.

[1447] This invention is a system in which a user uses a generated AI avatar and a human avatar in a metaverse space to interact and act according to their emotions. Specific embodiments of this system are described below.

[1448] Avatar Customization

[1449] Users customize their avatars using a dedicated application. Users launch the application and upload a photo of their face. The device analyzes the photo using image processing libraries such as OpenCV and generates an avatar that resembles the user. Users can also customize the avatar's body shape and clothing.

[1450] Linking with My Number Card

[1451] The user holds their personal identification card (e.g., My Number card) over a dedicated reader. The device reads the card information through the NFC reader, and the server uses this information to verify the user's identity. This ensures the user's safety and guarantees that the generated avatar is the user.

[1452] Use of emotion engine

[1453] Users input their facial expressions and voice using a camera or microphone. The device analyzes the user's emotions using Microsoft Azure's Emotion API or Google Cloud's Natural Language API. For example, if the user smiles, the emotion engine will identify the emotion "joy."

[1454] Coordinating avatar dialogue and behavior

[1455] The server sends prompts to a generative AI model (e.g., GPT-4) based on the analysis results. The generative AI model generates dialogue content based on the user's emotions. For example, it generates a message like, "Hello, how are you feeling today?" The device displays the generated dialogue content to the user and adjusts the avatar's facial expression and tone of voice.

[1456] Learning avatar behavior patterns

[1457] The server records the user's reactions and uses a reinforcement learning algorithm to learn the avatar's behavioral patterns. For example, if the AI ​​avatar performs a specific action when the user expresses joy, and the user's reaction is positive, the AI ​​avatar will learn that behavior and reproduce it when a similar situation occurs.

[1458] Examples of concrete examples and prompts

[1459] A user launches the application and uploads a photo of their face. The application uses OpenCV to analyze the photo and generate an avatar that resembles the user. The user then holds their personal identification card over an NFC reader, and the server verifies the user's identity. When the user smiles into the camera, the emotion engine analyzes this, and the generated AI avatar creates a smiling expression and begins a conversation with a cheerful voice, asking, "Hello, how are you feeling today?"

[1460] Prompt Sentence Examples

[1461] "When a user smiles at the camera, the generative AI avatar should adopt a smiling facial expression and begin the dialogue with a cheerful tone of voice."

[1462] In this way, a system can be realized that provides dialogue and actions according to the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1463] Step 1:

[1464] The user launches a dedicated application. The user clicks the "Upload Face Photo" button, selects and uploads a photo of their face. The device receives the uploaded photo and analyzes it using the OpenCV library. Specifically, it extracts facial features (such as the position of the eyes, nose, and mouth) and uses this to generate an avatar that resembles the user. The input is the user's face photo, and the output is an avatar that resembles the user.

[1465] Step 2:

[1466] The user holds their personal identification card (e.g., My Number card) over an NFC reader. The device reads the card's information through the NFC reader. The read information is sent to a server, which then uses this information to verify the user's identity. Specifically, the server obtains personal information from the IC chip in the card and compares it with a government database. The input is the information on the personal identification card, and the output is the user's identity verification result.

[1467] Step 3:

[1468] Users input their facial expressions and voice through a camera or microphone. The device receives this input data and analyzes the user's emotions using Microsoft Azure's Emotion API or Google Cloud's Natural Language API. Specifically, it performs facial expression recognition and voice analysis to identify the user's emotional state. The input is the user's facial expressions and voice, and the output is the user's emotional state.

[1469] Step 4:

[1470] The server sends a prompt to a generative AI model (e.g., GPT-4) based on the analysis results. The prompt contains the user's emotional information. The generative AI model generates dialogue content based on the prompt according to the user's emotion. Specifically, if the user expresses joy, it generates the message "Hello, how are you feeling today?" The input is the user's emotional information, and the output is the generated dialogue content.

[1471] Step 5:

[1472] The device displays the generated dialogue to the user and adjusts the avatar's facial expression and tone of voice. Specifically, based on the dialogue received from the generative AI model, the device configures the avatar to speak with a smile and a bright tone of voice. The input is the generated dialogue, and the output is the adjusted avatar's facial expression and tone of voice.

[1473] Step 6:

[1474] The server records the user's reactions, specifically whether the user smiled and how they responded to the interaction. This data is used to learn the avatar's behavioral patterns using a reinforcement learning algorithm. The input is the user's reaction data, and the output is the learned behavioral patterns.

[1475] Step 7:

[1476] The server evolves the avatar's behavior based on the learning results. Specifically, the avatar is configured to perform a specific behavior when the user expresses joy. This allows the avatar to reproduce that behavior when a similar situation occurs. The input is the learning results, and the output is the evolved avatar's behavior.

[1477] (Application example 3)

[1478] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1479] In conventional metaverse spaces, users had limited means to customize their avatars, and the avatars' facial expressions and behaviors could not be changed in response to the user's emotions. This made it difficult to provide a more personalized experience for users. Furthermore, user identity verification was insufficient, creating security issues.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1481] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for communicating with the "generated AI avatar" via a text or voice input device, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life problems, and various data generation, means for identifying humans in conjunction with a personal identification card, means for easily distinguishing between the human and the generated AI avatar, means for recognizing a user's emotions and changing the facial expression and tone of voice of the generated AI avatar according to the emotions, and means for learning and evolving the behavioral patterns of the generated AI avatar according to the user's emotions. This enables a personalized experience according to the user's emotions, ensuring security and enabling more natural conversations and actions.

[1482] A "metaverse space" is a virtual space generated using virtual reality technology in which users can interact through digital avatars.

[1483] A "generated AI avatar" is an intelligent digital avatar generated using artificial intelligence technology that can interact with and take action with the user.

[1484] A "human avatar" is a digital character that represents the user in a virtual space, and allows the user to customize their face, body shape, clothing, etc.

[1485] A "personal identification card" is a card used to verify a user's identity, such as a My Number card.

[1486] An "emotion engine" is a technology that recognizes the user's emotions and adjusts the system's behavior according to those emotions.

[1487] A "voice input device" is a device that receives a user's voice as input and transmits it to the system.

[1488] A "behavioral pattern" is a series of actions and reactions that the generated AI avatar takes based on learning the user's actions and reactions.

[1489] A "personalized experience" is an experience that is customized based on the individual characteristics and emotions of each user, providing more individualized services and interactions.

[1490] "Security" refers to the technologies and methods used to ensure the safety of systems and data and to prevent unauthorized access and information leaks.

[1491] A system for implementing the present invention has the following configuration.

[1492] System Configuration

[1493] 1. Creating the Metaverse Space:

[1494] The server uses virtual reality technology to generate the Metaverse space, a virtual space where users can interact through digital avatars.

[1495] 2. Generate AI avatars:

[1496] The server generates a "generated AI avatar" that uses the generated AI as its intelligence. This avatar can interact with and act on behalf of the user.

[1497] 3. Human avatar generation:

[1498] Users can create a human avatar by customizing their face, body shape, clothing, etc. This avatar is a digital character that represents the user in a virtual space.

[1499] 4. Avatar Coexistence:

[1500] The server provides a means for "generated AI avatars" and human avatars to coexist in the metaverse space, allowing users and generated AI avatars to interact in the same space.

[1501] 5. Means of communication:

[1502] Users can converse with the "generated AI avatar" through text or voice input devices, such as smartphones or smart speakers.

[1503] 6. Receiving the benefits all at once:

[1504] The server will provide a means for people to enjoy all the benefits that generative AI brings to humanity, including casual conversation, advice on everyday life issues, and various data generation.

[1505] 7. Linking with personal identification cards:

[1506] The server provides a means to identify a person in association with a personal identification card (e.g., My Number Card), thereby verifying the user's identity.

[1507] 8. Emotion Recognition and Avatar Adjustment:

[1508] The server uses an emotion engine to recognize the user's emotions and provides a means to change the facial expressions and tone of voice of the generated AI avatar according to those emotions.

[1509] 9. Learning Behavioral Patterns:

[1510] The server provides a means for learning and evolving the behavioral patterns of the generated AI avatar according to the user's emotions, allowing the generated AI avatar to learn and evolve behaviors that are more adapted to the user's emotions.

[1511] Hardware and software used

[1512] Hardware:

[1513] Server: Generates the metaverse space, manages avatars, recognizes emotions, and performs other tasks.

[1514] Smartphone: Allows users to customize avatars and serves as a voice input device.

[1515] Smart speaker: Used as a voice input device.

[1516] Personal ID Card Reader: Reads the user's personal ID card.

[1517] software:

[1518] Metaverse generation software: Software for generating virtual spaces.

[1519] Generative AI model: A model for constructing the intelligence of generative AI avatars.

[1520] Emotion engine: Software that recognizes the user's emotions and adjusts the avatar's facial expressions and tone of voice.

[1521] Avatar customization software: Software that allows users to customize their avatars.

[1522] Specific examples

[1523] Users visit a virtual store and use their smartphone camera to recognize their emotions. The avatar changes its facial expression and tone of voice depending on the user's emotions, and tries on and tests out products. For example, if the user is happy, the avatar will smile and explain the products in a cheerful voice.

[1524] Prompt Sentence Examples

[1525] When a user is shopping in a virtual store, design a system that uses an emotion engine to recognize the user's emotions and adjust the avatar's facial expressions and tone of voice accordingly. If the user is happy, the avatar will smile and explain the products in a cheerful voice, and if the user is sad, the avatar will talk in a calm voice.

[1526] The above is an embodiment of the present invention.

[1527] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1528] Step 1:

[1529] The server generates the metaverse space.

[1530] Input: Virtual reality technology configuration data.

[1531] Data processing: A virtual space is constructed based on the setting data.

[1532] Output: The metaverse space accessible to the user.

[1533] Specific operation: The server uses virtual reality technology to generate 3D models and environments, creating a virtual space that users can access.

[1534] Step 2:

[1535] The server generates a "generated AI avatar" whose intelligence is the generated AI.

[1536] Input: Parameters of the generative AI model.

[1537] Data processing: Building avatar intelligence using generative AI models.

[1538] Output: An intelligent generative AI avatar.

[1539] Specific operation: The server uses a generative AI model to build the avatar's intelligence and generate an avatar that can interact with and act on the user.

[1540] Step 3:

[1541] Users can create a human avatar by customizing their face, body shape, clothing, etc.

[1542] Input: User customization data.

[1543] Data processing: Generates the appearance of an avatar based on customization data.

[1544] Output: A customized human avatar.

[1545] Specific operation: Users use a smartphone or PC to customize their own face, body shape, clothing, etc. to create a human avatar.

[1546] Step 4:

[1547] The server provides a means for "generated AI avatars" and human avatars to coexist in the metaverse space.

[1548] Input: Generated AI avatar and human avatar data.

[1549] Data processing: Integrate the data of both avatars into the metaverse space.

[1550] Output: A metaverse space where coexisting avatars are displayed.

[1551] Specific operation: The server places the generated AI avatar and a human avatar in the metaverse space, allowing the two to interact with each other.

[1552] Step 5:

[1553] Users converse with the "generated AI avatar" through text or voice input devices.

[1554] Input: User's written or spoken input.

[1555] Data processing: Analyze input data and transmit it to the generated AI avatar.

[1556] Output: The response of the generated AI avatar.

[1557] Specific operation: The user uses a smartphone or smart speaker to input text or voice and converse with the generated AI avatar.

[1558] Step 6:

[1559] The server will provide a means for people to enjoy all the benefits that generative AI brings to humanity, including casual conversation, advice on everyday life issues, and various data generation.

[1560] Input: User's question or inquiry.

[1561] Data processing: Use generative AI models to generate appropriate responses.

[1562] Output: Response to the user.

[1563] Specific operation: The server analyzes the user's question or inquiry, uses a generative AI model to generate an appropriate response, and provides it to the user.

[1564] Step 7:

[1565] The server provides a means for identifying a person in association with a personal identification card.

[1566] Input: Personal identification card data.

[1567] Data processing: Analyzes card data to verify user identity.

[1568] Output: User identification result.

[1569] Specific operation: The server uses a personal identification card reader to read the card data and verify the user's identity.

[1570] Step 8:

[1571] The server uses an emotion engine to recognize the user's emotions and provides a means to change the facial expressions and tone of voice of the generated AI avatar according to those emotions.

[1572] Input: User's facial and voice data.

[1573] Data processing: Analyze emotions using an emotion engine.

[1574] Output: Changes in facial expressions and tone of voice of the generated AI avatar.

[1575] Specific operation: The server analyzes the user's facial expression and voice data using an emotion engine and adjusts the facial expression and tone of voice of the generated AI avatar.

[1576] Step 9:

[1577] The server provides a means to learn and evolve the behavioral patterns of the generated AI avatar according to the user's emotions.

[1578] Input: User emotion and reaction data.

[1579] Data processing: Using generative AI models to learn behavioral patterns.

[1580] Output: Behavioral patterns of an evolved generative AI avatar.

[1581] Specific operation: The server collects the user's emotional and reaction data, learns behavioral patterns using a generative AI model, and evolves the behavior of the generated AI avatar.

[1582] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1583] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1584] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1585] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1586] [Third embodiment]

[1587] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1588] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1590] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1594] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1597] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1598] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1599] "Example 1"

[1600] In one embodiment of the present invention, the means for generating the metaverse space uses 3D graphics technology or VR technology to generate a virtual space. This metaverse space is designed to allow users to move and act freely.

[1601] "Example 2"

[1602] As a means of generating "generative AI avatars" with generative AI intelligence, AI technologies such as deep learning and reinforcement learning are used to generate AI that can converse like a human. This generative AI avatar can learn from the user's actions and comments and generate dialogue and actions according to the user's needs.

[1603] "Example 3"

[1604] As a means of generating human avatars, users will be able to customize their own avatars, specifically by freely setting their own face, body shape, clothing, etc.

[1605] "Example 4"

[1606] As a means of identifying people through linking with My Number cards, the system reads the information on the My Number card and uses it to verify the user's identity, ensuring user safety and making it easy to distinguish between generated AI avatars and human avatars.

[1607] The processing flow of each embodiment will be described below.

[1608] "Example 1"

[1609] Step 1: Using 3D graphics and VR technology, a metaverse space is generated. This metaverse space is designed to allow users to move and act freely.

[1610] Step 2: Design a user interface that allows users to move and act freely within the metaverse space.

[1611] "Example 2"

[1612] Step 1: Use AI techniques such as deep learning and reinforcement learning to create an AI that can converse like a human.

[1613] Step 2: Place the generated AI in the metaverse space as a "generated AI avatar."

[1614] Step 3: Generate the AI ​​avatar so that it learns from the user's actions and utterances and generates dialogue and actions according to the user's needs.

[1615] "Example 3"

[1616] Step 1: Allow users to customize their own avatars. Specifically, users can freely set their own face, body shape, clothing, etc.

[1617] Step 2: Place the configured avatar in the metaverse space and allow the user to freely operate it.

[1618] "Example 4"

[1619] Step 1: Read the information on your My Number card and use it to verify the user's identity.

[1620] Step 2: The verified identity information is used to distinguish between the user's avatar and the generated AI avatar, ensuring user safety and making it easy to distinguish between the generated AI avatar and a human avatar.

[1621] Example 1

[1622] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1623] In conventional metaverse spaces, it was difficult for users to move freely and interact within the virtual space. Furthermore, in an environment where generated AI avatars and human avatars coexist, it was difficult for users to communicate smoothly with the generated AI avatars. Furthermore, realizing a metaverse space that dynamically changes based on the user's actions and comments was also a challenge.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1625] In this invention, the server includes a means for transmitting a request for a user to access the metaverse space, a means for the server to receive the request and generate data for the metaverse space, a means for the server to transmit the generated data to the terminal, a means for the terminal to render the metaverse space based on the received data, and a means for the user to move and interact within the metaverse space through the terminal. This allows the user to move freely within the virtual space and smoothly communicate with the generated AI avatar. It also makes it possible to realize a metaverse space that dynamically changes based on the user's actions and comments.

[1626] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[1627] A "generative AI avatar" is a virtual person or character that has generative AI as its intelligence and is generated to converse and interact with users.

[1628] A "human avatar" is a digital alter ego created to represent a real person in a virtual space.

[1629] The "means for sending a request" is a device or software that has the function of sending the information necessary for a user to access the metaverse space to the server.

[1630] "Means for generating data" refers to devices or software that have the functionality to allow the server to create data for the metaverse space based on a user request.

[1631] "Means for transmitting data" refers to devices or software that have the function of transmitting data from the metaverse space generated by the server to the user's terminal.

[1632] A "means for rendering data" is a device or software that has the function of visually displaying the metaverse space based on the data that the terminal receives from the server.

[1633] An "interaction means" is a device or software that has the functionality to allow a user to interact with objects or other avatars within the Metaverse space.

[1634] "Identification information" refers to information used to identify a user, and includes, for example, a My Number card.

[1635] An "audio input device" is a device that allows a user to input voice, and includes, for example, a smart speaker or a microphone.

[1636] The present invention is a system that generates a metaverse space and allows users to freely move and interact within it. A specific embodiment of this system will be described below.

[1637] First, a user launches a dedicated application and sends a request to access the metaverse space. This request includes the user's authentication information and the type of virtual space they want. The device used by the user can be a PC or a VR headset.

[1638] The server receives requests from users and generates data for the Metaverse space using 3D graphics engines such as Unity or Unreal Engine. Specifically, it generates virtual terrain, buildings, and objects, allowing users to move and act freely. It also utilizes VR technology to provide a more immersive experience by allowing users to wear a VR headset.

[1639] The generated data for the Metaverse space is sent from the server to the user's device. This data includes 3D models, textures, lighting information, and more. The device uses Unity or Unreal Engine to render the Metaverse space based on the received data. Users can visually experience this virtual space through a VR headset or display.

[1640] Users can use a keyboard, mouse, or VR controller to move freely within the Metaverse and interact with objects. For example, they can walk around a virtual museum and view the artworks on display. They can also interact with specific objects to display additional information.

[1641] Furthermore, this system includes a generative AI avatar, with which users can converse via text or voice input. The generative AI avatar learns from the user's actions and comments, generating dialogue and actions according to the user's needs. This allows users to enjoy the benefits of generative AI, such as casual conversation, advice on daily life issues, and various data generation, all in one place.

[1642] As a concrete example, consider a scenario in which a user visits a virtual museum. The user wears a VR headset and accesses the metaverse space through their device. The server generates museum data using Unity and sends it to the device. The user can then freely walk around the museum using the VR controller and view the artworks on display.

[1643] An example of a prompt sentence might be:

[1644] "Generate a scenario in which a user visits a virtual museum. The user puts on a VR headset and uses the VR controllers to move freely around the museum and view the artworks on display."

[1645] In this way, users can move freely within the metaverse space and communicate smoothly with the generated AI avatar. It is also possible to realize a metaverse space that changes dynamically based on the user's actions and comments.

[1646] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1647] Step 1:

[1648] A user submits a request to access the metaverse space.

[1649] The user launches a dedicated application and enters their username and password on the login screen to authenticate. If authentication is successful, a request to access the metaverse space is sent to the server. This request includes the user's authentication information and the desired type of virtual space. The input is the user's authentication information and the desired type of virtual space, and the output is the request sent to the server.

[1650] Step 2:

[1651] The server receives the request and generates data for the metaverse space.

[1652] When the server receives a request from a user, it generates data for the metaverse space using a 3D graphics engine such as Unity or Unreal Engine. For example, if the user requests a "virtual city," the server generates the city's terrain data, 3D models of buildings, and objects such as roads and parks. The input is the user's request information, and the output is the data for the generated metaverse space.

[1653] Step 3:

[1654] The server transmits the generated data to the terminal.

[1655] The server compresses the data of the generated metaverse space and sends it to the user's device. The data includes the city's terrain, 3D building models, textures, and lighting information. The input is the data of the generated metaverse space, and the output is the data sent to the device.

[1656] Step 4:

[1657] The device renders the metaverse space based on the data received.

[1658] The device decompresses the data received from the server and uses Unity's rendering engine to render the virtual space. The user wears a VR headset and visually experiences the virtual space displayed on the display. The input is the data received from the server, and the output is the rendered Metaverse space.

[1659] Step 5:

[1660] Users move around the metaverse space and interact with it through their devices.

[1661] Users can freely move around the virtual space using VR controllers. For example, they can walk around a virtual museum and get up close and personal with the artworks on display. They can also interact with specific objects to reveal additional information. The input is the user's manipulation information, and the output is the user's actions and interactions within the virtual space.

[1662] Step 6:

[1663] The user converses with the generated AI avatar.

[1664] The user converses with the generated AI avatar using text or a voice input device. The generated AI avatar learns from the user's actions and comments, and generates dialogue and actions according to the user's needs. For example, if the user consults the generated AI avatar about a problem in their daily life, the generated AI avatar will provide appropriate advice. The input is the user's comments and behavioral information, and the output is the generated AI avatar's response.

[1665] (Application example 1)

[1666] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1667] In conventional metaverse spaces, users can move and act freely within the virtual space, but it is difficult to recreate a real-life shopping experience. Furthermore, it is not possible to obtain product information or check details in real time, which reduces user convenience. Furthermore, it is difficult to distinguish between generated AI avatars and human avatars, which can sometimes impair the user experience. There is a need to solve these issues and provide a more realistic and convenient virtual shopping experience.

[1668] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1669] In this invention, the server includes: means for generating a metaverse space; means for generating a "generated AI avatar" whose intelligence is a generative AI; means for generating a human avatar; means for the "generated AI avatar" and the human avatar to coexist in the metaverse space; means for communicating with the "generated AI avatar" via text or a smart speaker; means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life issues, and various data generation, all at once; means for identifying humans in conjunction with identification information; means for easily distinguishing humans from the generated AI avatar; means for generating a virtual store, allowing users to select products in the virtual space and obtain detailed information; and means for obtaining product information using a generative AI model and displaying it in the virtual space. This allows users to obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[1670] The "metaverse space" is a virtual space generated using 3D graphics and VR technology, in which users can move and act freely.

[1671] A "generative AI avatar" is a virtual entity that has generative AI as its intelligence and can interact and act with the user.

[1672] A "human avatar" is a digital alter ego that allows a user to represent themselves in a virtual space.

[1673] A "virtual store" is a virtual store set up within the metaverse space where users can select products and check detailed information.

[1674] A "generative AI model" is a model of artificial intelligence that has been trained to perform a specific task using generative AI techniques.

[1675] "Identification information" refers to information for identifying a user, and includes personal identification information such as a My Number card.

[1676] A "prompt" is an instruction entered into a generative AI model to make it perform a specific task.

[1677] The following system configuration will be described as an embodiment of the present invention.

[1678] The server first generates a metaverse space. The metaverse space is created using 3D graphics and VR technology, allowing users to move and act freely. Specifically, the virtual space is generated using a game engine such as Unreal Engine.

[1679] The server then generates a "generative AI avatar" that uses the generative AI as its intelligence. This generative AI avatar is a virtual entity that can interact with and act on the user, and is constructed using a generative AI model such as the OpenAI API.

[1680] The server also generates a human avatar, a digital alter ego that represents the user in the virtual space and changes dynamically according to the user's actions and words.

[1681] The server configures the "generated AI avatar" and the human avatar to coexist in the metaverse space, allowing the user to interact with and coexist with the generated AI avatar in the virtual space.

[1682] The server also provides the "generated AI avatar" with a means of conversation via text or a smart speaker. Users can communicate with the generated AI avatar through text input or voice input.

[1683] Furthermore, the server will provide a means for users to receive all the benefits that generative AI brings to humanity, such as chatting, advice on daily life issues, and various data generation, allowing users to utilize the diverse functions of generative AI.

[1684] The server provides a means to identify a person in association with their identification information. For example, the server identifies a user using personal identification information such as a My Number card.

[1685] The server also provides a means to easily distinguish between generated AI avatars and human avatars, allowing users to easily distinguish between generated AI avatars and human avatars.

[1686] Furthermore, the server generates a virtual store and provides a means for the user to select a product in the virtual space and obtain detailed information. The user can select a product in the virtual store in the virtual space and check detailed information.

[1687] The server uses the generative AI model to obtain product information and provides a means to display it in the virtual space. Specifically, it obtains product information using the OpenAI API and displays it in the virtual space using Unreal Engine.

[1688] As a concrete example, consider a scenario in which a user puts on a head-mounted display and enters a virtual store. The user can move freely within the virtual space and select a product they are interested in. Detailed information about the selected product is displayed, and the user can try it on or purchase it.

[1689] Examples of prompts for generative AI models include:

[1690] Please tell me more information about product ID 12345.

[1691] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[1692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1693] Step 1:

[1694] The server generates the metaverse space.

[1695] Input: 3D graphics technology and VR technology setting data

[1696] Data Processing: Building a Virtual World with Unreal Engine

[1697] Output: A virtual space where users can move and act freely

[1698] Step 2:

[1699] The server generates a "generated AI avatar" whose intelligence is the generated AI.

[1700] Input: Configuration data for the generative AI model

[1701] Data Processing: Building generative AI avatars using the OpenAI API

[1702] Output: A virtual entity that can interact with and act on the user.

[1703] Step 3:

[1704] The server generates a human avatar.

[1705] Input: User profile information

[1706] Data processing: Creating a digital avatar that changes dynamically based on the user's actions and words

[1707] Output: An avatar that represents the user in a virtual space

[1708] Step 4:

[1709] The server sets up the "generated AI avatar" and the human avatar to coexist in the metaverse space.

[1710] Input: Position information of the generated AI avatar and the human avatar

[1711] Data processing: Arrange them so that they can coexist in the virtual space

[1712] Output: An environment where users can interact and coexist with generated AI avatars in a virtual space.

[1713] Step 5:

[1714] The server provides the "generated AI avatar" with a means to converse with it via text or a smart speaker.

[1715] Input: Text and voice input from the user

[1716] Data processing: Using generative AI models to analyze input and generate appropriate responses

[1717] Output: Response message from the generated AI avatar

[1718] Step 6:

[1719] The server will provide a means for people to receive all the benefits that generative AI brings to humanity, such as chatting, advice on everyday life issues, and generating various data.

[1720] Input: User consultation details and data generation requests

[1721] Data processing: Using generative AI models to generate appropriate responses and data

[1722] Output: Response messages to the user and generated data

[1723] Step 7:

[1724] The server provides a means for identifying a person in association with the identity information.

[1725] Input: User identification information (e.g., My Number card)

[1726] Data processing: Identifying users based on identifying information

[1727] Output: User identification result

[1728] Step 8:

[1729] The server provides a means to easily distinguish between generated AI avatars.

[1730] Input: Identification information of the generated AI avatar and the human avatar

[1731] Data processing: Displaying information to distinguish between the two based on identification information

[1732] Output: A display that allows users to easily distinguish between generated AI avatars and human avatars.

[1733] Step 9:

[1734] The server generates a virtual store and provides a means for users to select products in the virtual space and obtain detailed information.

[1735] Input: Virtual store setting data

[1736] Data processing: Building a virtual store in a virtual space using Unreal Engine

[1737] Output: A virtual store where the user can select products and view more information.

[1738] Step 10:

[1739] The server uses the generative AI model to obtain product information and provides a means to display it within the virtual space.

[1740] Input: The ID of the product selected by the user

[1741] Data processing: Obtain product information using OpenAI API and display it in a virtual space using Unreal Engine

[1742] Output: User-visible product details

[1743] In this way, users can obtain product information and check details in real time, enabling a more realistic and convenient virtual shopping experience.

[1744] Example 2

[1745] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1746] In conventional metaverse spaces, it was difficult for users to have natural conversations with generated AI avatars, and the avatars were unable to respond appropriately based on the user's actions and statements. Furthermore, the metaverse space could not dynamically change according to the user's real-time actions and situations, limiting the user experience.

[1747] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1748] In this invention, the server includes means for generating a metaverse space, means for generating a "generated AI avatar" whose intelligence is a generative AI, means for generating a human avatar, means for the "generated AI avatar" and the human avatar to coexist in the metaverse space, means for conversing with the "generated AI avatar" through text or voice input, means for receiving the benefits that the generative AI brings to humanity, such as casual conversation, advice on daily life problems, and various data generation, means for identifying humans by linking them with identification information, means for easily distinguishing between humans and the generated AI avatar, means for collecting user behavior and speech data, means for preprocessing the collected data, means for training a generative AI model using the preprocessed data, means for evaluating the performance of the trained model, means for deploying the evaluated model as a generative AI avatar, and means for the user to interact with the generative AI avatar. This allows the generative AI avatar to respond appropriately based on the user's behavior and speech, and enables the metaverse space to dynamically change according to the user's real-time behavior and situation.

[1749] A "metaverse space" is a three-dimensional virtual environment generated using virtual reality technology, in which users can interact through avatars.

[1750] A "generative AI avatar" is a virtual character created using generative AI technology that can converse and act like a human.

[1751] A "human avatar" is a digital character that allows a user to represent themselves in a virtual space.

[1752] "Means for communicating through text or voice input" refers to an interface that allows users to converse with the generated AI avatar using text or voice.

[1753] "Chat, advice on everyday life issues, various data generation, etc." is a general term for the various services provided by the generated AI avatar, and is a function to support users' daily lives.

[1754] "Means of identification in conjunction with identification information" refers to technology for recognizing a user using specific identification information (e.g., ID card or biometrics).

[1755] "Means to make it easier to distinguish from generated AI avatars" refers to methods or technologies that allow users to distinguish between generated AI avatars and human avatars.

[1756] "Means for collecting user behavior and comment data" refers to a system for recording the actions and comments that users make within the virtual space.

[1757] "Means for preprocessing collected data" refers to the process used to convert collected data into a form suitable for analysis and model training.

[1758] A "means for training a generative AI model using preprocessed data" is a method for training a generative AI model using preprocessed data.

[1759] "Means for evaluating the performance of a trained model" refers to a method for measuring the accuracy and effectiveness of a generative AI model once training is complete.

[1760] "Means for deploying a fully evaluated model as a generative AI av...

Claims

1. A means for generating a metaverse space; A means for generating a generative AI avatar having the generative AI as intelligence; A means for generating a human avatar; A means for allowing the generated AI avatar and the human avatar to coexist in the metaverse space; A means for communicating with the generated AI avatar through text or voice; A means to receive the benefits that the AI ​​can bring to humanity through chatting, consulting about problems in daily life, or generating data, all at once; and means for distinguishing between the generated AI avatar and the human avatar in conjunction with identification information; A means for collecting user behavior and speech data; means for pre-processing the collected data; means for training the generative AI model using the pre-processed data; means for evaluating the performance of the trained model; A means for deploying the evaluated model as the generated AI avatar; A system including:

2. A means for generating a metaverse space; A means for generating a generative AI avatar having the generative AI as intelligence; A means for generating a human avatar; A means for allowing the generated AI avatar and the human avatar to coexist in the metaverse space; A means for communicating with the generated AI avatar through text or voice; A means to receive the benefits that the AI ​​can bring to humanity through chatting, consulting about problems in daily life, or generating data, all at once; and means for distinguishing between the generated AI avatar and the human avatar in conjunction with identification information; and means for recognizing a user's emotions and adjusting the generated AI avatar's dialogue and behavior in response to the emotions when the user browses or purchases items within a virtual store.

3. A means for generating a metaverse space; A means for generating a generative AI avatar having the generative AI as intelligence; A means for generating a human avatar; A means for allowing the generated AI avatar and the human avatar to coexist in the metaverse space; A means for communicating with the generated AI avatar through text or voice; A means to receive the benefits that the AI ​​can bring to humanity through chatting, consulting about problems in daily life, or generating data, all at once; and means for distinguishing between the generated AI avatar and the human avatar in conjunction with identification information; A system including: a means for recognizing a user's emotions and changing the environment of the metaverse space in accordance with the emotions.

Images