system

Abstract

To provide a system.SOLUTION: A system includes: means for installing AI, which learns CS data and can explain its own company service, in virtual space; means for contacting a person in charge of a physical store or reserving a visit to the store after initially coping with a user; means for summarizing interaction in the virtual space so as to share it for the physical store; means for receiving a question from the user and analyzing the question using a natural language processing technique; means for inputting a prompt sentence about the question analyzed by the analyzing means into a generative AI model and generating an appropriate answer; means for providing the user with the answer generated by the generating means through the AI; and means for recording and summarizing interaction data between the user and the AI.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is insufficient coordination between the metaverse and physical stores in terms of customer service, which means that there is an issue of being unable to ensure consistency in the customer experience. [Means for solving the problem]

[0005] To solve this problem, an AI that can learn CS data and explain the company's services will be installed in metaverse stores to handle initial user support. After that, the person in charge at the real store will be contacted or a reservation will be made for the store, and the exchanges in the metaverse will be summarized and shared with the store. This will strengthen the connection between customer support between the metaverse and real stores, ensuring a consistent customer experience. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16]FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, an AI that can learn CS data and explain the company's services is installed in a Metaverse store. This AI can understand questions from users using natural language processing technology, for example, and generate appropriate answers. The AI ​​also records and summarizes user questions and other dialogue data.

[0029] "Example 2"

[0030] After initial user interaction, the person in charge at the real store is contacted or a reservation is made for the visit. This contact is made, for example, by email, telephone, or through a dedicated reservation system. When making a reservation, the user is given the opportunity to specify the date and time of visit and the content of the service, taking into consideration the user's convenience. "Example 3"

[0031] The interactions in the metaverse are summarized and shared for the store. This can be done, for example, by storing the summary information in a cloud-based database and making it accessible to store personnel. The summary information includes the user's questions, the AI's answers, and other important dialogue data. This allows store personnel to understand the content of the interactions with the user in the metaverse and provide better service.

[0032] The process flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The AI ​​learns from the CS data, etc. This learning is performed using machine learning techniques such as supervised learning, unsupervised learning, and reinforcement learning.

[0035] Step 2: The AI ​​is installed in the Metaverse store and accepts questions from users. Questions are entered in text or voice format, and the AI ​​analyzes them using natural language processing technology.

[0036] Step 3: The AI ​​generates an appropriate answer and serves it to the user. The answer is generated based on the training data and contains the content that best fits the user's question.

[0037] Step 4: The AI ​​records the user's questions and other dialogue data and summarizes them. The summary is generated, for example, using keyword extraction and text summarization techniques.

[0038] "Example 2"

[0039] Step 1: After the initial user response is completed, the AI ​​contacts the person in charge at the brick-and-mortar store. For example, the contact can be made by accessing the brick-and-mortar store's reservation system via an API and sending the user's information and summarized interaction data.

[0040] Step 2: A store representative will receive the call and confirm the user's reservation. A notification of the reservation confirmation will be sent to the user via email or SMS, for example.

[0041] "Example 3"

[0042] Step 1: The AI ​​summarizes interactions in the metaverse and stores them in a cloud-based database, for example by accessing the database through an API and sending the summary data.

[0043] Step 2: A store associate accesses the database and reviews the summary data, for example through a dedicated dashboard that gives the associate an overview of the user’s questions, the AI’s responses, and other important interaction data.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0046] Conventional user response systems in virtual space have difficulty in providing prompt and appropriate answers to user questions, and the recording and summarization of dialogue data is insufficient. As a result, there are problems with poor user experience and inability to smoothly link with physical stores. Furthermore, the quality of answers to user questions is not consistent, which can lead to low user satisfaction.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing interactions in the virtual space so that they can be shared with the store, a means for receiving questions from users and analyzing the questions using natural language processing technology, a means for inputting a prompt sentence into a generative AI model and generating an appropriate answer, a means for providing the generated answer to the user, and a means for recording and summarizing dialogue data. This makes it possible to provide prompt and appropriate answers to user questions and effectively record and summarize dialogue data.

[0049] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0050] "AI that can explain a company's services" is artificial intelligence that automatically provides explanations in response to questions from users about the services a company offers.

[0051] A "virtual space" is a three-dimensional virtual environment constructed on the Internet, in which users can interact with other users and systems through avatars.

[0052] A "physical store" is a store that actually exists and is a location that users can visit in person.

[0053] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0054] A "generative AI model" is an artificial intelligence model that generates new information or answers based on input data.

[0055] A "prompt" is a sentence that is input into a generative AI model and serves as instructions for the model to generate an appropriate answer.

[0056] "Dialogue data" is a record of questions and answers exchanged between the user and the AI.

[0057] "Summarization" refers to extracting important parts of conversation data and summarizing them concisely.

[0058] This invention is a system that provides appropriate answers to questions from users in a virtual space and records and summarizes dialogue data. A specific embodiment of this system will be described below.

[0059] First, the server generates an AI program that uses natural language processing technology. This program has the function of understanding questions from users and generating appropriate answers. The hardware used is a high-performance server (e.g., a server equipped with an NVIDIA GPU). The software used is a natural language processing library (e.g., TENSORFLOW (registered trademark), PyTorch) and a database (e.g., MySQL (registered trademark)).

[0060] The server places the generated AI program in a virtual space. When this AI receives a question from a user, it analyzes the question using natural language processing technology. Specifically, it performs the following data processing and calculations:

[0061] 1. Data preprocessing: Tokenize the user question and remove unnecessary characters and words.

[0062] 2. Question analysis: Analyze the intent of the question using tokenized data.

[0063] 3. Answer generation: Generate appropriate answers based on the intent of the question, using a generative AI model (e.g. GPT-3 (registered trademark)).

[0064] 4. Recording dialogue data: Record user questions and AI answers in a database.

[0065] 5. Data Summarization: Summarize the recorded interaction data for future reference.

[0066] As a concrete example, when a user asks an AI in a virtual space, "Please tell me the features of this product," the following process takes place.

[0067] 1. The user enters a question through an interface in the virtual space.

[0068] 2. The device sends the user's question to the server.

[0069] 3. The server receives the question and uses a natural language processing library to tokenize the question and remove unnecessary characters and words.

[0070] 4. The server uses the tokenized data to analyze the intent of the question and inputs the prompt sentence, "User: What are the features of this product? AI:" into the generative AI model.

[0071] 5. The generative AI model generates an answer such as, "The features of this product are as follows. First, ..."

[0072] 6. The server sends the generated response to the device.

[0073] 7. The device displays the answer to the user, who then confirms the answer.

[0074] 8. The server records the user's questions and the AI's answers in a database and summarizes the dialogue data.

[0075] An example of a prompt sentence would be something like this:

[0076] User: What are the features of this product?

[0077] AI: The features of this product are as follows. First,...

[0078] In this way, the server, terminal, and user can work together to provide appropriate answers to user questions and effectively record and summarize dialogue data.

[0079] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0080] Step 1:

[0081] The user types a question in the virtual space.

[0082] Specific operation: The user inputs a question through the interface in the virtual space. For example, the user inputs, "Please tell me the features of this product."

[0083] Input: The text of the user's question

[0084] Output: The question text is sent to the terminal.

[0085] Step 2:

[0086] The terminal transmits the user's question to the server.

[0087] Specific operation: The terminal sends the question entered by the user to the server, which transmits data via the Internet.

[0088] Input: Question text

[0089] Output: The question text is sent to the server.

[0090] Step 3:

[0091] The server receives the question and begins natural language processing.

[0092] Specific operation: The server receives the question sent from the device and analyzes the received question using a natural language processing library (e.g. TensorFlow, PyTorch).

[0093] Input: Question text

[0094] Output: Tokenized question data

[0095] Step 4:

[0096] The server analyzes the question and inputs a prompt into the generative AI model.

[0097] How it works: The server tokenizes the question and removes unnecessary characters and words. It then analyzes the intent of the question and inputs a prompt to the generative AI model (e.g. GPT-3). For example, it inputs the prompt in the format: "User: What are the features of this product? AI:".

[0098] Input: Tokenized question data

[0099] Output: Prompt statement

[0100] Step 5:

[0101] A generative AI model generates the appropriate answer.

[0102] Specific behavior: The generative AI model generates an appropriate answer based on the input prompt sentence, for example, "The features of this product are as follows. First, ..."

[0103] Input: prompt statement

[0104] Output: The generated answer text

[0105] Step 6:

[0106] The server sends the generated answer to the terminal.

[0107] Specific operation: The server sends the answer obtained from the generative AI model to the terminal, converting the answer data into an appropriate format before sending it.

[0108] Input: Generated answer text

[0109] Output: The answer text is sent to the terminal.

[0110] Step 7:

[0111] The terminal displays the answer to the user.

[0112] Specific operation: The terminal displays the answer received from the server to the user, who then checks the answer through an interface in the virtual space.

[0113] Input: Answer text

[0114] Output: The answer that is displayed to the user

[0115] Step 8:

[0116] The server records and summarizes the interaction data.

[0117] How it works: The server records the user's questions and the AI's answers in a database, then summarizes the recorded dialogue data for future reference.

[0118] Input: User's question text and AI's answer text

[0119] Output: Recorded conversation data and a summary

[0120] (Application example 1)

[0121] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In conventional metaverse stores, when users had questions about products or services, it was difficult to get appropriate answers. In addition, there was a lack of a way to effectively record and summarize user questions and dialogue data and share them with staff at the physical store, so there was a need to improve the user experience. Furthermore, it was not possible to provide answers to user questions in real time, which could lead to a decrease in user satisfaction.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0124] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making them available for sharing with the store, a means for generating answers to user questions in real time and providing them to the user, and a means for recording dialogue data, summarizing it, and providing it to a store manager. This makes it possible for a user to receive an appropriate answer in real time when asking a question about a product or service in a metaverse store, and for dialogue data to be effectively recorded and summarized to be shared with a person in charge of a real store, improving the user experience.

[0125] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0126] "AI" is an abbreviation for artificial intelligence, a system that imitates human intellectual tasks using machine learning and natural language processing technologies.

[0127] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users and content through avatars.

[0128] "Real-time" refers to responsiveness in time to instantaneous response to user input.

[0129] "Dialogue data" is a record of the conversation between the user and the AI, and includes information such as questions and answers.

[0130] A "summary" is information that extracts the main points of dialogue data and summarizes them concisely.

[0131] A "store manager" is a person or organization in charge of operating a physical store or a metaverse store.

[0132] A "user" is a consumer or user who visits a metaverse store and asks questions about products and services.

[0133] A "real store" is a store that exists in a physical location where customers can visit in person to purchase goods or services.

[0134] A "store reservation" is the process by which a user reserves a date and time in advance to visit a physical store.

[0135] The following system configuration will be described as an embodiment of the present invention.

[0136] The server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user asks a question about a product or service in the Metaverse store, the AI ​​will use natural language processing technology to understand the user's question and generate an appropriate answer in real time. This answer will be provided to the user.

[0137] The server also records and summarizes the dialogue data, including the user's questions and the AI's responses, and provides it to the store manager, who can then more easily understand the user's needs and interests. In addition, after the initial user response, the system also includes a means to contact a brick-and-mortar store staff member or to schedule a visit.

[0138] This system is realized using the following hardware and software.

[0139] Hardware:

[0140] Smartphone or head mounted display

[0141] software:

[0142] Python

[0143] OpenAI® API

[0144] When a user enters a question into the application, the server uses OpenAI's API to generate an appropriate answer to the user's question. The generated answer is displayed to the user. The interaction data is recorded, summarized, and provided to the store manager.

[0145] For example, when a user asks "Tell me about this product" in a virtual store, the AI ​​responds, "This product uses the latest technology and is very high performance." The conversation data is also recorded and summarized as "User: Tell me about this product. AI: This product uses the latest technology and is very high performance."

[0146] Example prompt:

[0147] User: Tell me about this product.

[0148] AI: This product uses the latest technology and is extremely powerful.

[0149] In this way, when users ask questions about products or services in the metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with the brick-and-mortar store staff, improving the user experience.

[0150] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0151] Step 1:

[0152] Users access the metaverse store using a terminal (smartphone or head-mounted display) and input questions about products and services.

[0153] Input: The text of the user's question

[0154] Output: The user's question text is sent to the server.

[0155] Step 2:

[0156] The server receives the user's question text and analyzes the intent of the question using natural language processing techniques.

[0157] Input: The text of the user's question

[0158] Output: Data analyzed to determine the intent of the question

[0159] Step 3:

[0160] The server uses the OpenAI API to generate appropriate answers based on the intent of the analyzed question.

[0161] Input: Data analyzed to determine the intent of the question

[0162] Output: Answer text generated from the OpenAI API

[0163] Step 4:

[0164] The server transmits the generated answer text to the user's terminal and displays it to the user.

[0165] Input: Answer text generated from the OpenAI API

[0166] Output: The answer text displayed to the user's terminal.

[0167] Step 5:

[0168] The server records the user's question text and the generated answer text as dialogue data.

[0169] Input: User question text, generated answer text

[0170] Output: Recorded interaction data

[0171] Step 6:

[0172] The server summarizes the recorded interaction data and provides it to the store manager.

[0173] Input: Recorded interaction data

[0174] Output: Summarized conversation data

[0175] Step 7:

[0176] After initial user response, the server provides a means to contact a person in charge at a real store or to make a reservation to visit the store.

[0177] Input: User question text, generated answer text, recorded dialogue data

[0178] Output: Contact a store representative or schedule a visit

[0179] In this way, when users ask questions about products or services in a metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with store personnel, improving the user experience.

[0180] Example 2

[0181] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] In conventional systems, users had to take the time and effort to contact a real store representative or make a reservation to visit after receiving the initial response. In addition, there was a lack of efficient and accurate means for users to communicate information when specifying the date and time of their visit or the service content. Furthermore, when sharing interactions in the virtual space with the real store, the information was not summarized or organized sufficiently, which could lead to delayed responses from the store.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0184] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing the interactions in the virtual space and sharing them with the store, a means for providing an interface for the user to input the date and time of visit and the service content, a means for saving the input data in a database, a means for sending a notification according to a specified contact method based on the saved data, and a means for feeding back the notification sending result to the user. This allows the user to efficiently and accurately specify the date and time of visit and the service content, and quickly contact the staff member at the real store. In addition, by summarizing and sharing the interactions in the virtual space, the store side can respond quickly and accurately.

[0185] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0186] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[0187] A "virtual space" is a digital environment generated by computer simulation, which users can access via the Internet and in which they can carry out various activities.

[0188] A "physical store" is a real store that actually exists and is a place where users can visit in person to receive services.

[0189] An "interface" is the means by which a user interacts with a system, and includes user interface elements such as input forms and buttons.

[0190] A "database" is a system for efficiently storing, managing, and searching data, and is operated using a query language such as SQL.

[0191] A "notification" is information sent from the system to a user or person in charge, and may be provided in the form of email, telephone call, message, etc.

[0192] "Feedback" refers to the response or result that a system provides to a user, informing the user of the outcome of an action they took.

[0193] A "summary" is a concise summary of the content of an exchange in a virtual space, in which important information is extracted and organized.

[0194] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0195] This invention is a system that allows users to contact a real store staff member or make a reservation to visit the store after receiving the initial response. The system includes AI, a user interface, a database, and a notification function installed in a virtual space.

[0196] Hardware and Software Used

[0197] The server is developed using the Python programming language and implemented as a web application using the Flask framework, with MySQL as the database and an SMTP server and Twilio API for notifications.

[0198] Specific operation of the system

[0199] The user accesses the system's web page using a web browser. The web page displays a form for inputting the date and time of visit and the desired service. For example, the user inputs that they would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[0200] The server receives the data entered by the user in the form and processes the request using the Flask framework, which stores the data in a MySQL database by generating SQL queries to insert the data into the database.

[0201] The server then uses the information stored in the database to send notifications according to the contact method the user specified: for example, if it's email, it uses an SMTP server to send an email, if it's phone, it uses the Twilio API to make an automated voice call, if you use a dedicated reservation system, it sends the reservation information through the reservation system's API, etc.

[0202] Finally, the server checks whether the notification was sent successfully and gives feedback to the user about the result. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[0203] Examples of concrete examples and prompts

[0204] As a concrete example, consider a user making a reservation at a hair salon. The user enters into a web form that they would like a "cut and color" service on "October 15, 2023 at 14:00." The server receives this information and sends a notification to the salon representative via email.

[0205] Example prompt:

[0206] If a user fills in a web form to make an appointment at a hair salon and requests a "cut and color" service on "October 15, 2023 at 2:00 PM," the server should receive this information and send a notification to the salon representative via email.

[0207] In this way, a system is realized that allows the user to specify the date and time of visit and the content of the service, taking into consideration the user's convenience.

[0208] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0209] Step 1:

[0210] The user enters the date and time of visit and the details of the service into a web form.

[0211] A user accesses the system's web page using a web browser and enters information into a form to input the date and time of visit and the desired service. For example, the user enters that he / she would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[0212] Input: Visit date and time, service details

[0213] Output: Data entered by the user

[0214] Step 2:

[0215] The server receives the input data and stores it in a database.

[0216] The server receives the data entered by the user in the form, processes the request using the Flask framework, and saves the input data to a MySQL database by generating SQL queries to insert into the database.

[0217] Input: Data entered by the user

[0218] Output: Data stored in the database

[0219] Step 3:

[0220] The server will send notifications based on the stored data and in accordance with the specified contact method.

[0221] The server sends notifications according to the contact method specified by the user based on the information stored in the database. For example, if it is email, it will send an email using an SMTP server. If it is phone, it will make an automated voice call using the Twilio API. If a dedicated reservation system is used, the reservation information will be sent via the reservation system's API.

[0222] Input: Data stored in the database, contact method

[0223] Output: Notification sent result

[0224] Step 4:

[0225] The server provides feedback on the notification sending result to the user.

[0226] The server checks whether the notification was sent successfully and gives the result as feedback to the user. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[0227] Input: Notification sending result

[0228] Output: A feedback message to the user.

[0229] (Application example 2)

[0230] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] Conventional user support systems in the metaverse had problems with insufficient coordination with real stores after the initial response, resulting in reduced user convenience. In addition, there was a lack of a way to select the appropriate person in charge based on the content of the user's inquiry and to contact them quickly, which led to a risk of user satisfaction decreasing. Furthermore, there was no reservation system that allowed users to specify the date and time of visit or the content of the service, making it difficult to flexibly respond to user needs.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0233] In this invention, the server includes a means for installing an AI inside a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making it possible to share them with the store, a means for the AI ​​to automatically select an appropriate staff member based on the content of the user's inquiry, a means for contacting the staff member via email, telephone, or a dedicated reservation system, and a means including a reservation system that allows the user to specify the date and time of visit and the content of the service. This makes it possible to quickly select and contact an appropriate staff member based on the content of the user's inquiry, thereby improving user convenience and satisfaction.

[0234] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0235] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[0236] The "metaverse" is a digital environment built in a virtual space, a platform where users can interact with other users and systems through avatars.

[0237] A "real store" is a store that exists in a physical location and is a place where users can actually visit and receive products and services.

[0238] "Contact Person" refers to a store staff member or employee designated to handle user inquiries and reservations.

[0239] A "reservation system" is a system that enables a user to make a reservation by specifying the date and time of visit and the content of the service, and is used via email, telephone, or a dedicated online platform.

[0240] A "summary" is a concise summary of the content of interactions within the metaverse, including user questions and AI responses.

[0241] "Email" means a digital message sent over the Internet and is a means of exchanging information between users and personnel.

[0242] "Telephone" is a means of exchanging information in real time through voice communication, and is intended for direct communication between users and personnel.

[0243] "Automatic selection" is a process in which AI automatically selects the most suitable representative based on the content of the user's inquiry.

[0244] A system for implementing the present invention is configured as follows.

[0245] First, the server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user makes an initial inquiry within the Metaverse, the AI ​​will automatically generate a response based on the content of the inquiry and provide it to the user. This allows the user to quickly obtain the information they need.

[0246] Next, if the user's inquiry meets certain conditions, the server contacts the store's staff or schedules a visit. Specifically, the AI ​​automatically selects an appropriate staff member based on the user's inquiry and contacts the staff member via email, phone, or a dedicated reservation system. The contact includes the user's inquiry, the desired date and time of the visit, and the service details.

[0247] Additionally, interactions within the metaverse will be summarized and made available to stores, including the user's question and the AI's answer, allowing store staff to quickly understand the user's situation.

[0248] To realize this system, the server uses the following hardware and software: The hardware requires a server connected to the Internet and a user's smartphone. The software includes a Python program, an SMTP server (e.g. smtp.example.com), and a reservation system.

[0249] As a specific example, if a user wants to "consult about purchasing a new smartphone" and wants to make a reservation to visit the store at 14:00 on October 15, 2023, the following prompt sentence will be generated.

[0250] Username: Yamada Taro

[0251] Inquiry: Consultation about purchasing a new smartphone

[0252] Desired date and time: 2023-10-15 14:00

[0253] contact address:

[0254] Email: taro.yamada@example.com

[0255] Phone: 090-1234-5678

[0256] Based on this prompt, the server contacts the person in charge and confirms the user's reservation, allowing the user to smoothly receive services at a physical store.

[0257] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0258] Step 1:

[0259] A user makes an initial inquiry within the metaverse.

[0260] Input: User's inquiry (e.g. "Consultation on purchasing a new smartphone")

[0261] Processing: The user's inquiry is received and the AI ​​analyzes it.

[0262] Output: Parsed query content

[0263] Step 2:

[0264] AI automatically generates answers based on the user's inquiries and provides them to the user.

[0265] Input: Parsed query content

[0266] Processing: The AI ​​uses the generative AI model to generate the appropriate answer.

[0267] Output: The generated answer (e.g. "When it comes to buying a new smartphone, we recommend the following model...")

[0268] Step 3:

[0269] If the user's inquiry meets certain conditions, the server contacts a person in charge at the physical store or makes a reservation to visit the store.

[0270] Input: Analyzed inquiry details, user's desired visit date and time, service details

[0271] Processing: AI automatically selects the appropriate person to contact based on the inquiry and generates a prompt.

[0272] Output: Generated prompt text (Example: "User name: Yamada Taro, Inquiry: Consultation on purchasing a new smartphone, Desired date and time: 2023-10-15 14:00, Contact: Email: taro.yamada@example.com, Phone: 090-1234-5678")

[0273] Step 4:

[0274] The server will contact the person via email, phone, or a dedicated reservation system.

[0275] Input: Generated prompt text

[0276] Processing: The server uses an SMTP server to send an email to a person in charge, or uses a phone API to automatically make a phone call, or send a prompt to a reservation system.

[0277] Output: Contact the person in charge (e.g., the reservation information will be sent to the person in charge's mailbox)

[0278] Step 5:

[0279] Interactions within the metaverse are summarized and shared for stores.

[0280] Input: User question, AI answer

[0281] Processing: The server summarizes interactions within the metaverse and generates summary information.

[0282] Output: Generated summary information (e.g. "User question: Looking for advice on buying a new smartphone. AI answer: We recommend the following model...")

[0283] Step 6:

[0284] Share summary information for stores.

[0285] Input: Generated summary information

[0286] Processing: The server sends summary information to the person in charge.

[0287] Output: The agent receives summary information to quickly understand the user's situation.

[0288] Example 3

[0289] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0290] There is a need to efficiently collect, summarize, and share conversation data between users and AI in the metaverse with store staff in real stores in order to provide better services based on users' needs and questions. However, in current systems, the collection, summarization, and sharing of conversation data is often done manually, which makes it inefficient and lacks real-time capabilities.

[0291] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0292] In this invention, the server includes a means for collecting dialogue data in the metaverse, a means for storing the collected dialogue data in a cloud-based database, a means for summarizing the dialogue data using a generative AI model, a means for storing the summary data in the cloud-based database, and a means for allowing a person in a physical store to view the summary data. This makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time.

[0293] The "metaverse" is a three-dimensional virtual space constructed using virtual reality technology, in which users can interact with other users and AI through avatars.

[0294] "Interaction Data" refers to data that includes questions, answers, and other significant content of conversations between users and AI within the metaverse.

[0295] A "cloud-based database" is a database service accessible via the Internet in which data is stored, managed, and accessed in a cloud environment.

[0296] A "generative AI model" is a model that uses artificial intelligence technology to generate, summarize, and convert text data, such as a model that uses natural language processing technology.

[0297] "Summary data" is data that concisely summarizes important information extracted from dialogue data by a generative AI model.

[0298] "Brick and mortar store staff" refers to staff who deal with customers and provide services in a physical store.

[0299] "Viewing" is the act of displaying and checking stored data or information.

[0300] This invention is a system that efficiently collects dialogue data between users and AI in the metaverse, summarizes the data, and shares it with staff in real stores. A specific embodiment of this system will be described below.

[0301] The server collects user-AI interaction data in the metaverse in real time. This interaction data includes user questions, AI answers, and other important dialogue content. The server stores the collected interaction data in a cloud-based database (e.g., Amazon RDS or Google Cloud SQL).

[0302] The server then uses a generative AI model (e.g., a model using natural language processing techniques) to summarize the collected dialogue data. This summary includes the key points of the user's question and the AI's answer. The summarized information is again stored in a cloud-based database.

[0303] Store staff can use devices to access the cloud-based database and view summary information, allowing them to understand the content of interactions between users and the store staff in the metaverse and provide better service.

[0304] As a concrete example, consider a case where a user asks in the metaverse, "Is a new product in stock?" and an AI responds, "Yes, it's in stock." This dialogue data is collected by the server and a generative AI model is used to generate a summary: "The user checked the availability of a new product, and the AI ​​responded that it is in stock." This summary information is stored in a cloud-based database and can be accessed by staff at the brick-and-mortar store.

[0305] Example prompt:

[0306] Summarize the user's question and the AI's answer.

[0307] User Question: Are new items in stock?

[0308] AI answers: Yes, we have it in stock.

[0309] By inputting this prompt sentence into the generative AI model, summarized information is obtained. This system makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0310] Step 1:

[0311] The server collects the dialogue data between the user and the AI ​​in the metaverse in real time. Specifically, when the user asks "Is the new product in stock?" in the metaverse and the AI ​​responds "Yes, it is in stock," the server captures this dialogue data. The input is the user's question and the AI's answer, and the output is the collected dialogue data.

[0312] Step 2:

[0313] The server stores the collected dialogue data in a cloud-based database. Specifically, the server inserts the dialogue data into an appropriate table in the database. The input is the collected dialogue data, and the output is the dialogue data stored in the database.

[0314] Step 3:

[0315] The server inputs the saved dialogue data into the generative AI model and generates a summary. Specifically, it inputs the following prompt sentence into the generative AI model:

[0316] Summarize the user's question and the AI's answer.

[0317] User Question: Are new items in stock?

[0318] AI answers: Yes, we have it in stock.

[0319] The input is the prompt and the dialogue data, and the output is the generated summary data.

[0320] Step 4:

[0321] The server again stores the generated summary data in the cloud-based database. Specifically, the server inserts the summary data into an appropriate table in the database. The input is the generated summary data, and the output is the summary data stored in the database.

[0322] Step 5:

[0323] Using a terminal, a store associate accesses the cloud-based database and views the summary data. Specifically, the associate queries the database, retrieves the summary data, and displays it. The input is a request to access the database, and the output is the displayed summary data.

[0324] This series of processes makes it possible to efficiently collect, summarize, and share conversation data within the metaverse in real time.

[0325] (Application example 3)

[0326] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] In order for store personnel to understand the content of conversations between users and retailers in the metaverse and provide better services, a means of efficiently summarizing and sharing the content of the conversations is necessary. However, with current systems, it is difficult to summarize the conversation data in the metaverse in real time, store it in a cloud-based database, and have store personnel in retailers access it using their smartphones. This poses the challenge of quickly understanding the needs and interests of users and providing personalized services.

[0328] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0329] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing interactions in the metaverse and making it possible to share with the store, a means for storing the summary information in a cloud-based database and making it possible for a person in charge of a real store to access using a smartphone, and a means for including user questions, AI responses, and other important dialogue data in the summary information. This makes it possible to efficiently summarize the content of the dialogue in the metaverse and for a person in charge of a real store to quickly access it, thereby making it possible to grasp the needs and interests of the user and provide personalized services.

[0330] "CS Data" is data related to customer service, including customer inquiries, feedback, service history, etc.

[0331] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[0332] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users through avatars and carry out various activities.

[0333] A "physical store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[0334] A "cloud-based database" is a database that is accessible via the internet and in which data is stored and managed by a cloud service provider.

[0335] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[0336] "Summary information" is information that briefly summarizes detailed data or dialogue content and includes important points and main points.

[0337] A "user question" is an inquiry or question that a user makes to an AI within the metaverse.

[0338] An "AI answer" is a response or explanation that an AI generates in response to a user's question.

[0339] "Important dialogue data" refers to information or data that is considered to be particularly important in the dialogue between the user and AI, and contains content that is useful in providing services.

[0340] In order to implement the present invention, the following system configuration and processing procedure are used.

[0341] First, the server installs an AI inside the metaverse store that can learn CS data and explain the company's services. When a user asks the AI ​​a question in the metaverse, the AI ​​automatically generates an answer to the question and provides it to the user. This allows the user to obtain information about the company's services in the metaverse.

[0342] Next, after the initial user response, the server contacts the person in charge at the brick-and-mortar store and, if necessary, makes a reservation to visit the store. This completes the preparations for the user to receive services at the brick-and-mortar store.

[0343] In addition, the server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The summary information includes the user's questions, the AI's answers, and other important dialogue data. Store staff can access the cloud-based database using their smartphones to check the summary information. This allows store staff to quickly understand the user's needs and interests and provide personalized services.

[0344] As a concrete example, a user may ask in the metaverse, "What are the features of my new smartphone?" and the AI ​​may respond, "The new smartphone has a high-resolution camera and a long-lasting battery." This conversation is summarized and stored in a cloud-based database. A store associate can access this summary information using their smartphone and understand that the user is interested in the new smartphone.

[0345] Examples of prompts to input to a generative AI model include:

[0346] A user asks in the metaverse, "What are the features of my new smartphone?" and the AI ​​responds, "My new smartphone has a high-resolution camera and a long-lasting battery." Summarize this conversation:

[0347] In this way, the server, terminal, and user work together to efficiently summarize the content of conversations in the metaverse, which can then be quickly accessed by store personnel in the physical store, enabling them to understand the needs and interests of the user and provide personalized services.

[0348] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0349] Step 1:

[0350] The server will install AI inside Metaverse stores that can learn CS data and other data and explain the company's services. When a user asks the AI ​​a question within the Metaverse, the AI ​​will automatically generate an answer to that question and provide it to the user. The input is the user's question, and the output is the AI's answer. In concrete terms, it uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0351] Step 2:

[0352] After initially responding to the user, the server contacts the person in charge at the physical store and, if necessary, makes a reservation to visit the store. The input is the user's question and the AI's response, and the output is contact information for the person in charge. In concrete terms, the server sends a notification to the person in charge based on the content of the user's question and the AI's response, and processes the reservation to visit the store.

[0353] Step 3:

[0354] The server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The input is the user's question and the AI's answer, and the output is the summary information. Specifically, the server uses a generative AI model to summarize the dialogue and stores it in a cloud-based database.

[0355] Step 4:

[0356] A staff member at a brick-and-mortar store uses a smartphone to access the cloud-based database and check the summary information. The input is the summary information obtained from the cloud-based database, and the output is the staff member's understanding. The specific operation is to access the database through a smartphone application and display the summary information.

[0357] Step 5:

[0358] Based on the summary information, the staff member understands the user's needs and interests and provides personalized services. The input is the summary information, and the output is the service to be provided. Specifically, the system refers to the summary information and suggests products and services that are suitable for the user.

[0359] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0360] "Example 1"

[0361] In one embodiment of the present invention, the AI ​​includes an emotion engine that infers the user's emotional state from the user's questions and other dialogue data. Specifically, the emotion engine uses natural language processing and voice analysis techniques to infer emotions from the user's text and voice inputs. It also uses image analysis and motion analysis techniques to analyze the user's facial expressions and body movements.

[0362] "Example 2"

[0363] The user's emotional state estimated by the emotion engine is reflected in the AI's response generation. Specifically, if the user shows joy or excitement, the AI ​​generates a response that emphasizes that emotion. Conversely, if the user shows discomfort or confusion, the AI ​​generates a response that soothes that emotion. This allows the AI ​​to respond appropriately to the user's emotions.

[0364] "Example 3"

[0365] The emotion engine also influences summaries of interactions in the metaverse. Specifically, the user's emotional state is recorded as part of the summary, which is then shared with the store. For example, if a user expresses strong excitement about a product, that information is shared with the store to help predict the product's popularity and sales. If a user expresses dissatisfaction with a service, that information is shared with the store to help improve the service.

[0366] The process flow of each embodiment will be described below.

[0367] "Example 1"

[0368] Step 1: The AI ​​receives a question and other interaction data from the user.

[0369] Step 2: The emotion engine estimates the user's emotional state from the received data. Specifically, it estimates emotions from text and voice input using natural language processing and voice analysis technologies. It also estimates emotions from the user's facial expressions and body movements using image analysis and motion analysis technologies.

[0370] "Example 2"

[0371] Step 1: The emotion engine estimates the user's emotional state.

[0372] Step 2: The AI ​​generates answers based on the estimated emotional state. Specifically, if the user is happy or excited, the AI ​​generates answers that emphasize those emotions. Conversely, if the user is uncomfortable or confused, the AI ​​generates answers that soften those emotions.

[0373] "Example 3"

[0374] Step 1: The emotion engine estimates the user's emotional state and reflects the result in a summary of the interaction in the metaverse.

[0375] Step 2: The summary is shared with the store. Specifically, information about the user's excitement about the product and dissatisfaction with the service is shared with the store.

[0376] Step 3: Stores use the shared information to predict product popularity and sales, and use it as a reference for improving services.

[0377] Example 1

[0378] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0379] Conventional user response systems in virtual space have difficulty generating appropriate answers to user questions and are unable to accurately grasp the user's emotional state. As a result, there were problems with poor user experience, poor guidance to physical stores, and poor improvement in the quality of service. Furthermore, there was a lack of a way to effectively share interactions in virtual space with physical stores, which led to a lack of consistency in user response.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0381] In this invention, the server includes a means for installing an artificial intelligence in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing the exchange in the virtual space so that it can be shared with the store, a means for analyzing the user's questions using natural language processing technology and generating appropriate answers, a means for using voice analysis technology, image analysis technology, and motion analysis technology to estimate the user's emotional state, and a means for recording and summarizing the dialogue data. This makes it possible to generate appropriate answers to the user's questions and accurately grasp the user's emotional state. In addition, by effectively sharing the exchange in the virtual space with the real store, it is possible to maintain consistency in user responses and improve the quality of service.

[0382] "Artificial intelligence" refers to a system that includes techniques for generating appropriate answers to user questions and estimating the user's emotional state.

[0383] A "virtual space" is a three-dimensional virtual environment constructed on the Internet, in which users can interact with other users and systems through avatars.

[0384] "Natural language processing technology" is a technology that analyzes text input from a user, understands their intent, and generates an appropriate response.

[0385] "Voice analysis technology" is a technology for converting voice input from a user into text and analyzing that text.

[0386] "Image analysis technology" is a technology that captures a user's facial expressions and body movements with a camera and estimates their emotional state by analyzing the data.

[0387] "Motion analysis technology" is a technology that analyzes the user's physical movements and infers their emotional state based on that data.

[0388] "Dialogue data" is a record of the interactions between a user and artificial intelligence, and includes information such as questions asked, answers given, and emotional states.

[0389] A "summary" is a concise summary of important information extracted from dialogue data.

[0390] A "brick and mortar store" is a store that exists in a physical location and that a user can actually visit.

[0391] A "store reservation" is a procedure in which a user reserves a date and time in advance to visit a physical store.

[0392] The present invention relates to an artificial intelligence system that explains a company's services to users in a virtual space. A specific embodiment of this system will be described below.

[0393] First, the server generates a program that combines natural language processing technology, voice analysis technology, image analysis technology, and motion analysis technology. This program includes artificial intelligence to understand questions from users and generate appropriate answers. Specifically, the following technologies are used:

[0394] Natural language processing technology: The server analyzes the text input from the user and understands the intent of the question. For this technology, for example, BERT or GPT-3 is used.

[0395] Speech analysis technology: The server converts the user's voice input into text and analyzes the text. This technology uses Google Cloud Speech-to-Text and Amazon Transcribe.

[0396] Image analysis technology: The server captures the user's facial expressions and body movements with a camera and estimates their emotional state. This technology uses OpenCV and Google Cloud Vision.

[0397] Motion analysis technology: The server analyzes the user's body movements and estimates their emotional state. This technology uses Microsoft® Azure® Kinect DK.

[0398] Next, the user accesses the AI ​​in the virtual space. The user logs into a virtual store in the virtual space and accesses the AI ​​assistant. The user starts a conversation by clicking on the AI ​​assistant's icon or avatar in the virtual space.

[0399] When a user inputs a question, either by text or voice, the server receives the input and selects the appropriate analysis technology: natural language processing technology for text input, and voice analysis technology for voice input.

[0400] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer. For example, in response to a question such as "What are the features of this product?", the server generates an answer such as "This product is made of high-quality materials and is durable."

[0401] In addition, the server uses an emotion engine to estimate the user's emotional state. In the case of text input, natural language processing technology is used to analyze emotions, and in the case of voice input, voice analysis technology is used. In addition, the server analyzes the user's facial expressions and body movements through the camera to estimate the user's emotional state more accurately.

[0402] The server returns the generated answer to the user, either by displaying it in a text chat window or by playing the answer aloud, for example, "This product is made of high quality materials and is durable."

[0403] Finally, the server records the interaction data with the user and summarizes it for later analysis and improvement, for example recording the user's questions and emotional state to be used in the next interaction.

[0404] Examples:

[0405] User accesses AI in a virtual space: The user clicks on the AI ​​assistant's avatar in the virtual store and begins interacting with it.

[0406] User types question in via text or voice: The user types in text, such as "What are the features of this product?", or speaks the question into the microphone.

[0407] The server receives the input and selects the appropriate parsing technique: The server recognizes that the user's input is text and selects a natural language processing technique.

[0408] The server analyzes the question using natural language processing technology: The server analyzes the question using GPT-3 and extracts information about the product's features.

[0409] The server generates an appropriate answer: The server generates an answer such as "This product is made of high quality materials and is durable."

[0410] Server infers user's emotional state: The server analyzes emotions from the user's text input and infers whether the user is happy or not.

[0411] Server returns answer to user: The server displays the generated answer in a text chat window.

[0412] The server records and summarizes the dialogue data: The server records the user's questions and emotional state and summarizes them for use in the next dialogue.

[0413] Example prompt:

[0414] "Please tell me the features of this product."

[0415] "How long can this product be used for?"

[0416] Could you please tell me the pricing plan for this service?

[0417] In this way, the server can provide better service by generating appropriate answers to the user's questions and estimating the user's emotional state.

[0418] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0419] Step 1:

[0420] The user accesses the AI ​​in a virtual space.

[0421] Specific operations: The user logs into a virtual store in a virtual space and begins a dialogue by clicking on the AI ​​assistant's icon or avatar.

[0422] Input: User login information, location in the virtual world.

[0423] Output: Start of a conversation session with the AI ​​assistant.

[0424] Step 2:

[0425] The user enters a question by text or voice.

[0426] Specific Actions: The user types a question into a text chat window or speaks a question using a microphone, for example, "What are the features of this product?"

[0427] Input: User text or voice input.

[0428] Output: Text or audio data.

[0429] Step 3:

[0430] The server receives the input and selects the appropriate analysis technique.

[0431] Specific operation: The server determines whether the input from the user is text or voice, and selects natural language processing technology in the case of text input and voice analysis technology in the case of voice input.

[0432] Input: Text or audio data.

[0433] Output: The result of the analysis technique selection.

[0434] Step 4:

[0435] The server analyzes the question using natural language processing techniques.

[0436] Specific operation: The server uses natural language processing technology (e.g., GPT-3) to analyze the user's question and understand the intent of the question.

[0437] Input: Text data.

[0438] Output: Analysis of the intent of the question.

[0439] Step 5:

[0440] The server generates an appropriate response.

[0441] Specific behavior: The server generates an appropriate answer based on the analysis results, for example, "This product is made of high-quality materials and is durable."

[0442] Input: The analysis results for the intent of the question.

[0443] Output: The answer text.

[0444] Step 6:

[0445] The server estimates the user's emotional state.

[0446] Specific operation: The server uses an emotion engine to estimate the user's emotional state. In the case of text input, natural language processing technology is used to analyze emotions, and in the case of voice input, voice analysis technology is used. In addition, the server analyzes the user's facial expressions and body movements through the camera to more accurately estimate the user's emotional state.

[0447] Input: text data, audio data, image data.

[0448] Output: Emotion state estimation result.

[0449] Step 7:

[0450] The server returns the answer to the user.

[0451] Specific Behavior: The server returns the generated answer to the user, either by displaying it in a text chat window or by playing the answer aloud, for example, "This product is made of high-quality materials and is durable."

[0452] Input: The answer text.

[0453] Output: Display or audio the answer to the user.

[0454] Step 8:

[0455] The server records and summarizes the interaction data.

[0456] Specific operation: The server records the interaction data with the user and summarizes it for later analysis and improvement. For example, it records the user's questions and emotional state and uses them in the next interaction.

[0457] Input: Dialogue data, emotion state estimation results.

[0458] Output: Summarized interaction data.

[0459] (Application example 1)

[0460] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0461] Conventional user response systems in virtual space have had issues with providing appropriate answers to user questions and responding in a way that takes into account the user's emotional state. In addition, there was a lack of a way to effectively share interactions in virtual space with physical stores, and there was a need to improve the user experience.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a physical store or making a reservation for a visit after an initial user response, a means for summarizing the exchange in the virtual space so that it can be shared with the store, a means including an emotion engine for estimating the user's emotional state, a means for using natural language processing technology for analyzing the user's voice and text input, and a means for using image analysis technology for analyzing the user's facial expressions and actions. This makes it possible to provide appropriate answers to the user's questions and respond in consideration of the emotional state, and to effectively share the exchange in the virtual space with the physical store.

[0463] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0464] A "virtual space" is a three-dimensional virtual environment constructed using computer technology, which is a space that users can experience interactively.

[0465] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[0466] A "physical store" is a real, physical store that exists and is a location that customers can visit in person.

[0467] An "emotion engine" is a technology for estimating a user's emotional state, and is a system that recognizes emotions using voice analysis and image analysis.

[0468] "Natural language processing technology" is a technology that enables computers to understand and generate human language, and includes text analysis and speech recognition.

[0469] "Image analysis technology" refers to technology that analyzes image data and extracts information, including object recognition and facial recognition.

[0470] A "summary" is a short summary of a long piece of text or dialogue data, in which important information is extracted and expressed concisely.

[0471] "Contact" is the act of transmitting information, and is done using means such as telephone or email.

[0472] A "store reservation" is the act of a customer reserving a date and time to visit a physical store in advance, allowing the store to prepare for the visit.

[0473] A system for implementing the present invention has the following configuration.

[0474] First, the server installs an AI in a virtual space that can learn CS data and explain the company's services. Users can access the virtual space using a smartphone or head-mounted display and interact with the AI. The AI ​​automatically generates answers to the user's questions and provides them to the user.

[0475] Next, after the initial user response, the server contacts the person in charge at the physical store or makes a reservation to visit the store. This allows the user to receive services at the physical store based on the information obtained in the virtual space.

[0476] In addition, the server will summarize the virtual interactions and make them available to the store, allowing staff at the physical store to see what questions users asked in the virtual world and what answers they received.

[0477] The system includes an emotion engine that estimates the user's emotional state. The emotion engine uses natural language processing technology to analyze the user's voice and text input. It also uses image analysis technology to analyze the user's facial expressions and movements. This allows the system to estimate the user's emotional state and respond appropriately.

[0478] As a specific example, if a user asks, "Tell me about the latest smartphone," the AI ​​will respond, "It's the latest smartphone. It has a high-performance camera and a long-lasting battery." In addition, if a user asks, "Where can I buy this product?", the AI ​​will respond, "You can buy this product at the online store."

[0479] To realize this system, the following hardware and software are used.

[0480] Hardware: Smartphone, head-mounted display

[0481] Software: OpenAI API (generative AI model), Transformers library (natural language processing technology), OpenCV (image analysis technology)

[0482] An example of a prompt is as follows:

[0483] "Tell me about the latest smartphones."

[0484] "Where can I buy this product?"

[0485] "What are the features of the head mounted display?"

[0486] This allows users to receive the services of a physical store through an interactive experience in a virtual space.

[0487] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0488] Step 1:

[0489] The user accesses the virtual space using a smartphone or head-mounted display. The user inputs questions to the AI ​​assistant in the virtual space. Input is done by text or voice. The input data is sent to the server.

[0490] Step 2:

[0491] The server analyzes the received user question using natural language processing technology. Specifically, it uses the OpenAI API to understand the intent of the question and generate an appropriate answer. The input is the user's question, and the output is the generated answer.

[0492] Step 3:

[0493] The server provides the generated answer to the user. The user checks the answer in the virtual space. The output is the answer that is provided to the user.

[0494] Step 4:

[0495] The server analyzes the user's voice and text input with an emotion engine to estimate the user's emotional state. The emotion engine uses the Transformers library to recognize emotions. The input is the user's voice and text data, and the output is the estimated emotional state.

[0496] Step 5:

[0497] The server analyzes the user's facial expressions and movements using image analysis technology. Specifically, it uses OpenCV to recognize the user's facial expressions and movements and complement the emotional state. The input is the user's image data, and the output is the analyzed facial expression and movement information.

[0498] Step 6:

[0499] The server summarizes the virtual interaction and shares it with the physical store staff. The summary includes the user's questions and the AI's answers. The input is the virtual interaction data, and the output is the summarized information.

[0500] Step 7:

[0501] After the initial user interaction, the server contacts the person in charge at the physical store or schedules a visit. The means of communication are email and phone. The input is the user's contact information and the desired reservation date and time, and the output is a contact or reservation confirmation.

[0502] Step 8:

[0503] A user visits a physical store and receives services based on the information obtained in the virtual space. The staff at the physical store respond based on the summary information shared by the server. The input is the summary information, and the output is the provision of services to the user.

[0504] Example 2

[0505] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0506] Conventional systems had the problem of low user satisfaction because they responded uniformly without considering the user's emotional state. There were also issues with the time required to contact a brick-and-mortar store staff after the initial response and the complicated procedure for making a reservation to visit the store. Furthermore, there was a lack of functionality to specify the date and time of the user's visit and the content of the service, making it difficult to accommodate the user's needs.

[0507] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received input data to estimate the emotional state of the user, a means for generating an answer using a generation AI model based on the estimated emotional state, a means for providing the generated answer to the user, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, and a means for allowing the user to specify the date and time of the user's visit and the service content. This enables an appropriate response according to the user's emotional state, simplifies the contact and reservation procedures after the initial response, and provides convenience to the user.

[0508] "Means for receiving user input data" refers to a function for sending text and voice data input by a user through a terminal to a server and receiving the same.

[0509] "Means for analyzing received input data and inferring the emotional state of a user" refers to a function for analyzing received input data of a user and inferring the emotional state of the user (happiness, excitement, discomfort, confusion, etc.) from that data.

[0510] "Means for generating an answer using a generative AI model" is a function for generating an appropriate answer for a user using a generative AI model based on an estimated emotional state.

[0511] "Means for providing the generated answer to the user" refers to a function for sending and providing the answer generated by the generative AI model to the user.

[0512] "A means of contacting a physical store staff member or making a reservation to visit the store after the initial response to the user" refers to a function for contacting a physical store staff member or making a reservation to visit the store after the initial response has been completed.

[0513] "A means for allowing a user to specify the date and time of visit and the content of the service" is a function that enables a user to specify the date and time of visit and the content of the service that the user wishes to receive.

[0514] An "emotion engine" is software or algorithms for analyzing user input data and inferring an emotional state from that data.

[0515] A "generative AI model" is an artificial intelligence model that generates answers in natural language based on a user's input data and emotional state.

[0516] "Email, telephone, or dedicated reservation system" refers to the means of communication or reservation management system used to communicate between the user and the brick-and-mortar store personnel.

[0517] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0518] The present invention provides a system for responding appropriately to a user's emotional state and simplifying communication and reservation procedures after the initial response. A specific embodiment of this system will be described below.

[0519] A means of receiving user input data

[0520] A user accesses the system using a terminal (e.g., a PC or smartphone) and enters a question or information. The data entered by the user is sent from the terminal to the server. The server has a high-performance processor (e.g., an Intel Xeon processor) to process the received data.

[0521] A means of estimating the user's emotional state by analyzing received input data

[0522] The server passes the received user input data to an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer) for emotion analysis. The emotion engine estimates the user's emotional state (happiness, excitement, discomfort, confusion, etc.) from the text. The analysis results are fed back to the generative AI model.

[0523] A means of generating answers using a generative AI model based on an inferred emotional state

[0524] The server generates a response to the user using a generative AI model (e.g., OpenAI's GPT-4 (registered trademark)) based on the emotional state obtained from the emotion engine. For example, if the user expresses joy, the AI ​​generates a positive response such as "That's great! I'm glad you're interested."

[0525] A means of providing the generated answers to the user

[0526] The server sends and provides the answer generated by the generative AI model to the user, who can receive the answer through their device.

[0527] After initial user response, a means to contact a brick-and-mortar store representative or make a reservation to visit the store

[0528] If the user is satisfied with the initial response, the server contacts the person in charge at the physical store or makes a reservation to visit the store. This contact can be done via email (e.g., Gmail API), phone (e.g., Twilio API), or a dedicated reservation system (e.g., Calendly).

[0529] A means to specify the date and time of the user's visit and the content of the service

[0530] When making a reservation, the user can specify the date and time of visit and the service content. The server inputs this information into the reservation system and sends a reservation confirmation email to the user. It also notifies the person in charge at the physical store of the reservation information.

[0531] Examples

[0532] Example 1: When a user expresses joy

[0533] A user types, "I love your new product!" The emotion engine detects "delight," and the generative AI model responds, "That's great! We're glad you're interested. If you'd like more information, please reach out to us or schedule an appointment."

[0534] Example 2: When the user expresses confusion

[0535] A user types, "I don't really understand how to use this product." The emotion engine detects "confusion," and the generative AI model responds, "We apologize for the inconvenience. If you need further explanation, please contact our sales representative or schedule an appointment to visit."

[0536] Examples of prompt statements

[0537] "If a user is excited about a new product, generate a positive response."

[0538] "If a user is confused about how to use your product, generate a reassuring answer."

[0539] In this way, the system will respond appropriately depending on the user's emotional state, and if necessary, will contact a physical store staff member or make a reservation to visit the store.

[0540] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0541] Step 1: Receive user input data

[0542] A user accesses the system using a terminal (e.g., a PC or smartphone) and inputs a question or information. The data entered by the user is sent from the terminal to the server. In concrete terms, the user opens a browser, accesses the system's web page, enters a question in a text box, and clicks the "Submit" button. The input data is the user's question or information (e.g., "Please tell me the details of a new product"), and the output is the user's input data sent to the server.

[0543] Step 2: Analyze the received input data to infer the user's emotional state

[0544] The server passes the received user input data to an emotion engine (e.g. IBM Watson Tone Analyzer) for emotion analysis. The input data is the user's text data, and the emotion engine analyzes this data to estimate the emotional state (happiness, excitement, discomfort, confusion, etc.). In concrete terms, the server sends the user's input data to the emotion engine, which analyzes the text, estimates the emotional state, and returns the result to the server. The output is the estimated emotional state.

[0545] Step 3: Generate answers using a generative AI model based on the estimated emotional state

[0546] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an answer for the user based on the emotional state obtained from the emotion engine. The input data is the emotion analysis result, and the generative AI model uses this data to generate an appropriate answer. In concrete terms, the server passes the emotion analysis result to the generative AI model, which then generates an answer according to the emotional state. The output is the generated answer.

[0547] Step 4: Provide the generated answer to the user

[0548] The server sends and provides the answer generated by the generative AI model to the user. The input data is the generated answer, and the server sends this answer to the user. In specific operations, the server sends the generated answer to the user's terminal, and the user receives the answer through the terminal. The output is the answer provided to the user.

[0549] Step 5: After initial user interaction, contact a brick-and-mortar store representative or schedule a visit

[0550] If the user is satisfied with the initial response, the server will contact the person in charge at the physical store or make a reservation to visit. The input data is the user's satisfaction level and desired contact method or reservation information, and the server will contact or make a reservation based on this. In concrete terms, the user clicks the "Book a visit" button, and the server accesses the reservation system and inputs the user's desired date and time and service details. The output is a reservation confirmation email and / or a contact notification.

[0551] Step 6: Specify the date and time of the user's visit and the type of service

[0552] When making a reservation to visit a store, the user can specify the date and time of visit and the details of the service. The input data is the user's desired date and time of visit and the details of the service, which the server inputs into the reservation system. In concrete terms, the server inputs the user's desired date and time and the details of the service into the reservation system, sends a reservation confirmation email to the user, and also notifies the person in charge at the physical store of the reservation information. The output is a reservation confirmation email and a notification to the person in charge.

[0553] (Application example 2)

[0554] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0555] In the conventional metaverse store system, it was difficult to respond appropriately according to the user's emotional state, which led to a poor user experience. In addition, when a user expressed discomfort or confusion, there was a lack of a way to quickly contact a real store staff member, making it difficult to improve user satisfaction.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0557] In this invention, the server includes a means for installing an AI inside a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a store visit reservation after initial user response, a means for summarizing exchanges in the metaverse and making them available for sharing with the store, a means for the AI ​​to generate an appropriate response based on the user's emotional state estimated by the emotion engine, and a means for contacting a staff member when the user expresses discomfort or confusion. This enables appropriate responses according to the user's emotional state, improving the user experience and quickly resolving problems.

[0558] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0559] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and make inferences.

[0560] "Metaverse" refers to a virtual space, a 3D virtual world built on the Internet.

[0561] An "emotion engine" is an algorithm or system for estimating a user's emotional state, and is a technology that analyzes emotions using natural language processing and machine learning.

[0562] A "real store" refers to a store that exists in a physical location and that customers can actually visit.

[0563] A "store reservation" refers to a customer making a reservation to visit a store at a specific date and time.

[0564] A summary is a concise summary of detailed information, extracting the most important points and expressing them briefly.

[0565] "User experience" refers to the experience and satisfaction a user feels when using a product or service.

[0566] "Quick problem resolution" refers to quickly responding to and resolving problems that arise.

[0567] The system for implementing this invention is configured as follows. First, the server installs an AI inside a metaverse store that learns CS data and can explain the company's services. When a user makes an inquiry in the metaverse, the emotion engine estimates the user's emotional state, and the AI ​​generates an appropriate answer based on that emotion.

[0568] If the user expresses discomfort or confusion, the server will contact the person in charge at the real store. The means of communication will be email, telephone, or a dedicated reservation system. If the user wishes to make a reservation to visit the store, they can specify the date and time of the visit and the service content.

[0569] Interactions within the metaverse are summarized and shared with the store, including the user's question and the AI's answer, allowing store personnel to quickly understand the user's situation and take appropriate action.

[0570] Hardware and software used

[0571] Hardware: Smartphones, servers

[0572] Software: Python, smtplib (sending email), random (simulating emotion engine)

[0573] Data processing and calculation

[0574] 1. Emotion Engine: Takes user input and infers the emotional state by parsing emotions from the user's text using natural language processing models.

[0575] 2. AI answer generation: Generate appropriate answers based on the estimated emotions. For example, if the user is confused, generate an answer such as "I'm sorry for the inconvenience. How can I help you?"

[0576] 3. Contact with the person in charge: If the user expresses discomfort or confusion, the server will contact the person in charge. Email will be used as the means of contact. The email will include the user's desired date and time of visit and the details of the service.

[0577] Examples

[0578] If a user enters "I don't know how to use the product," the emotion engine infers "confusion," and the AI ​​responds, "We're sorry for the inconvenience. How can we help you?" The user then enters the desired date and time of visit and the details of the service, and a representative is contacted.

[0579] Examples of prompt statements

[0580] If a user enters "I don't know how to use the product," the emotion engine infers "confusion," and the AI ​​responds, "We're sorry for the inconvenience. How can we help you?" The user then enters the desired date and time of visit and the details of the service, and a representative is contacted.

[0581] In this way, appropriate responses can be made according to the user's emotional state, improving the user experience and quickly resolving issues.

[0582] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0583] Step 1:

[0584] A user makes a query in the metaverse. The user uses a smartphone or computer to access a virtual store in the metaverse and enters a question in text format. The entered text is sent to the server.

[0585] Step 2:

[0586] The server receives the user's input text and passes it to the emotion engine. The emotion engine uses a natural language processing model to infer the user's emotional state. Specifically, it analyzes the text data and classifies emotions such as happiness, excitement, discomfort, confusion, etc. The output of the emotion engine is the inferred emotional state.

[0587] Step 3:

[0588] The server receives the output of the emotion engine and passes it to the generative AI model. The generative AI model generates an appropriate answer based on the estimated emotional state. For example, if the user is confused, it generates an answer such as "I'm sorry for the inconvenience. How can I help you?" The generated answer is returned to the server.

[0589] Step 4:

[0590] The server provides the generated answer to the user. The user can check the answer from the AI ​​on the screen of their smartphone or computer. This allows the user to get the appropriate answer to their question.

[0591] Step 5:

[0592] If the user expresses discomfort or confusion, the server contacts the person in charge at the brick-and-mortar store. When the user inputs the desired date and time of visit and the details of the service, the information is sent to the server. The server contacts the person in charge via email. The email includes the user's desired date and time of visit and the details of the service.

[0593] Step 6:

[0594] The server summarizes the interactions in the metaverse and shares them with the store. The summary includes the user's questions and the AI's answers. This allows store staff to quickly understand the user's situation and take appropriate action.

[0595] In this way, appropriate responses can be made according to the user's emotional state, improving the user experience and quickly resolving problems.

[0596] Example 3

[0597] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0598] In the past, data on user-AI conversations in the metaverse was often not utilized as useful information for store staff. In addition, it was difficult to provide services that took into account the emotional state of the user. This made it difficult for store staff to accurately grasp the needs and emotions of users and provide appropriate services.

[0599] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0600] In this invention, the server includes a means for collecting dialogue data in the metaverse, a means for storing the collected dialogue data in a cloud-based database, a means for analyzing the stored dialogue data and generating summary information, a means for analyzing the emotional state of the user using an emotion engine, a means for storing the generated summary information and the emotional state in the cloud-based database, and a means for a staff member of a real store to access the cloud-based database and view the summary information. This enables the staff member of the real store to understand the content of the dialogue with the user and the emotional state in the metaverse and provide more appropriate and personalized services.

[0601] The "metaverse" is a three-dimensional virtual space constructed using virtual reality technology, in which users can interact with other users and AI through avatars.

[0602] "Interaction data" refers to data that includes information such as questions, answers, and timestamps exchanged between users and AI within the metaverse.

[0603] A "cloud-based database" is a database service accessible via the Internet and is a platform for storing, managing, and analyzing data.

[0604] "Summary information" is a concise summary of important information extracted from the dialogue data, including the user's question, the AI's response, and key dialogue points.

[0605] An "emotion engine" is software or algorithms that analyze user interaction data and identify the user's emotional state (e.g., excitement, frustration, joy, etc.).

[0606] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze text data and generate summaries and answers.

[0607] "Brick and mortar store staff" refers to staff who deal with customers and provide services in a physical store.

[0608] "Access" means the act of connecting to a cloud-based database and viewing, retrieving, or manipulating stored data.

[0609] The present invention is a system that collects dialogue data in the metaverse, analyzes it, generates summary information, and provides it to staff in a real store. A specific embodiment of this system will be described below.

[0610] The server obtains dialogue data from the metaverse platform (e.g., a three-dimensional virtual space using virtual reality technology) through an API. When a user asks an AI a question in the metaverse, the question and the AI's answer are sent to the server in real time. For example, when a user asks, "What are the features of the new smartphone?", the question and the AI's answer are collected as dialogue data.

[0611] The server stores the collected interaction data in a cloud-based database (e.g., a database service accessible via the Internet). The details of each interaction are recorded in the database. For example, the following records are added:

[0612] {

[0613] "timestamp": "2023-10-01T12:34:56Z",

[0614] "user_question": "What are the features of the new smartphone?",

[0615] "ai_response": "New smartphones have high-resolution cameras and long-lasting batteries."

[0616] }

[0617] The server then analyzes the stored dialogue data and generates summary information. A generative AI model (e.g., an artificial intelligence model using natural language processing technology) is used to generate a summary of the dialogue data. This summary includes the user's question, the AI's answer, and important dialogue points. The generated summary looks like this:

[0618] Summary: A user asked about features of a new smartphone, and the AI ​​responded with information about high-resolution cameras and long battery life.

[0619] Further, the server uses an emotion engine (e.g., software or algorithm for analyzing a user's emotional state) to analyze the user's emotional state. The analysis result includes the emotion (excitement, frustration, joy, etc.) that the user exhibited during the interaction. The analysis result is as follows:

[0620] Emotional state: Excited

[0621] The server stores the generated summary information and emotional state in a cloud-based database. The stored records are as follows:

[0622] {

[0623] "timestamp": "2023-10-01T12:34:56Z",

[0624] "summary": "A user asked about the features of a new smartphone, and the AI ​​answered about high-resolution cameras and long battery life.",

[0625] "emotion": "excitement"

[0626] }

[0627] Using the device, a brick-and-mortar salesperson can access a cloud-based database and view summary information, which allows the salesperson to understand the customer's interest in a new smartphone and provide a detailed explanation and suggest related accessories.

[0628] An example of a prompt sentence to be input to the generative AI model is as follows.

[0629] Please provide a summary of user-AI interaction data within the metaverse. Please include the following information:

[0630] User Questions

[0631] AI Answers

[0632] The user's emotional state

[0633] This system enables the staff of the real store to understand the content of the conversation with the user and the emotional state in the metaverse, and to provide more appropriate and personalized services. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0634] Step 1:

[0635] The server obtains dialogue data from the metaverse platform through API. When a user asks an AI a question in the metaverse, the question and the AI's answer are sent to the server in real time. The input is the user's question and the AI's answer, and the output is the dialogue data. In concrete terms, the server sends an API request and receives the dialogue data.

[0636] Step 2:

[0637] The server stores the collected dialogue data in a cloud-based database. The input is the dialogue data obtained in step 1, and the output is the records stored in the database. Specifically, the server executes an INSERT query on the database to store the dialogue data.

[0638] Step 3:

[0639] The server analyzes the stored dialogue data and generates summary information. The input is the dialogue data stored in the database, and the output is the generated summary information. In concrete terms, the server inputs the dialogue data into the generative AI model and generates a summary. The generative AI model uses natural language processing technology to generate a summary of the dialogue data.

[0640] Step 4:

[0641] The server uses the emotion engine to analyze the user's emotional state. The input is the dialogue data, and the output is the analyzed emotional state. In specific operations, the server inputs the dialogue data to the emotion engine and analyzes the emotional state. The emotion engine identifies the user's emotion (excitement, frustration, joy, etc.).

[0642] Step 5:

[0643] The server stores the generated summary information and emotional states in a cloud-based database. The input is the summary information and the emotional states, and the output is the records stored in the database. Specifically, the server executes an INSERT query on the database to store the summary information and the emotional states.

[0644] Step 6:

[0645] Using the terminal, the store associate accesses the cloud-based database and views the summary information. The input is the summary information and emotional state stored in the database, and the output is the information viewed by the associate. In concrete terms, the associate uses the terminal to send a query to the database and obtain the summary information and the emotional state.

[0646] (Application example 3)

[0647] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0648] There is a demand for a system that allows store staff to understand the conversations between users and AI in the metaverse, and use the information to improve services and predict product popularity. However, current systems have difficulty summarizing interactions in the metaverse in real time and sharing them, including emotional states. There is also a lack of ways to provide information in a form that store staff can easily access. This makes it difficult to quickly understand user needs and take appropriate action.

[0649] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a store visit reservation after an initial user response, a means for summarizing the exchange in the metaverse and making it possible to share it with the store, a means for storing the summary information in a cloud-based database and making it accessible to the real store staff, a means for recording the user's emotional state as part of the summary and sharing it with the store, and a means for providing an application that is installed on a smartphone or tablet and allows the real store staff member to check the summary information. This makes it possible to summarize the content of the dialogue with the user in the metaverse in real time and share it including the emotional state, making it possible for the real store staff member to quickly grasp the user's needs and take appropriate action.

[0650] "CS data" is data related to customer service, including information such as customer inquiry content, response history, and satisfaction ratings.

[0651] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[0652] The "metaverse" is a virtual reality space, a digital environment in which users can interact and perform activities with other users and AI through avatars.

[0653] A "real store" is a store that exists in a physical location and is a place where customers can actually visit to purchase products or receive services.

[0654] A "cloud-based database" is a database that is accessible via the Internet, where data is stored and managed on a cloud server.

[0655] "Summary information" is a concise summary of the content of the conversation between the user and the AI ​​within the metaverse, including the user's questions, the AI's answers, and the AI's emotional state.

[0656] An "emotional state" refers to the emotion a user exhibits during a conversation, such as excitement, frustration, joy, sadness, etc.

[0657] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[0658] A "tablet" is a portable computing device equipped with a touch screen, and is capable of using applications in the same way as a smartphone.

[0659] An "application" is a software program that provides specific functions or services and is installed and used on a smartphone or tablet.

[0660] A system for implementing the present invention is configured as follows.

[0661] First, the server installs an AI inside the Metaverse store that can learn CS data and other information and explain the company's services. When a user interacts with the AI ​​in the Metaverse, the content of the interaction is recorded in real time. The recorded content of the interaction is stored as summary information in a cloud-based database. This summary information includes the user's question, the AI's response, and the user's emotional state.

[0662] A store associate can then access this summary information using a dedicated application installed on a smartphone or tablet. The application retrieves the summary information from the cloud-based database and displays it to the associate, allowing the associate to understand the content of the interaction with the user in the metaverse and take appropriate action.

[0663] For example, if a user asks "Do you have this item in stock?" in the metaverse, the AI ​​will respond with "Yes, we do." The conversation is recorded along with the user's emotional state of excitement. This information is summarized and stored in a cloud-based database. Store personnel can access this information using a smartphone or tablet and provide more information about the product the user is interested in.

[0664] An example of a prompt sentence would be something like this:

[0665] Please summarize the interactions between the user and the AI ​​in the metaverse and output them in the following format:

[0666] User ID

[0667] question

[0668] answer

[0669] Emotional state

[0670] example:

[0671] User ID: user_001

[0672] Q: Is this item in stock?

[0673] Answer: Yes, we have it in stock.

[0674] Emotional state: Excited

[0675] In this way, conversations in the metaverse can be summarized in real time and shared, including emotional states, allowing store personnel to quickly understand the needs of users and provide appropriate responses.

[0676] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0677] Step 1:

[0678] The server will install an AI inside the Metaverse store that can learn CS data and explain the company's services. When a user starts a conversation with the AI ​​in the Metaverse, the server will record the conversation in real time. The input is the user's question and the AI's answer, and the output is the recorded conversation data.

[0679] Step 2:

[0680] The server stores the recorded dialogue data as summary information in a cloud-based database. The summary information includes the user's questions, the AI's answers, and the user's emotional state. The input is the recorded dialogue data, and the output is the summary information stored in the cloud-based database.

[0681] Step 3:

[0682] A staff member at a brick-and-mortar store retrieves summary information from the cloud-based database using a dedicated application installed on a smartphone or tablet. The input is an access request to the cloud-based database, and the output is the retrieved summary information.

[0683] Step 4:

[0684] The terminal displays the acquired summary information to the staff member. The displayed information includes the user's question, the AI's answer, the emotional state, etc. The input is the acquired summary information, and the output is the summary information displayed on the terminal screen.

[0685] Step 5:

[0686] Based on the displayed summary information, the salesperson can grasp the user's needs and respond appropriately. For example, if the user is excited, the salesperson can provide detailed information about the product. The input is the displayed summary information, and the output is the salesperson's response.

[0687] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0688] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0689] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0690] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0691] [Second embodiment]

[0692] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0693] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0694] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0695] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0696] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0697] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0698] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0699] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0700] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0701] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0702] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0703] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0704] "Example 1"

[0705] As one embodiment of the present invention, an AI that can learn CS data and explain the company's services is installed in a Metaverse store. This AI can understand questions from users using natural language processing technology, for example, and generate appropriate answers. The AI ​​also records and summarizes user questions and other dialogue data.

[0706] "Example 2"

[0707] After initial user interaction, the person in charge at the real store is contacted or a reservation is made for the visit. This contact is made, for example, by email, telephone, or through a dedicated reservation system. When making a reservation, the user is given the opportunity to specify the date and time of visit and the content of the service, taking into consideration the user's convenience. "Example 3"

[0708] The interactions in the metaverse are summarized and shared for the store. This can be done, for example, by storing the summary information in a cloud-based database and making it accessible to store personnel. The summary information includes the user's questions, the AI's answers, and other important dialogue data. This allows store personnel to understand the content of the interactions with the user in the metaverse and provide better service.

[0709] The process flow of each embodiment will be described below.

[0710] "Example 1"

[0711] Step 1: The AI ​​learns from the CS data, etc. This learning is performed using machine learning techniques such as supervised learning, unsupervised learning, and reinforcement learning.

[0712] Step 2: The AI ​​is installed in the Metaverse store and accepts questions from users. Questions are entered in text or voice format, and the AI ​​analyzes them using natural language processing technology.

[0713] Step 3: The AI ​​generates an appropriate answer and serves it to the user. The answer is generated based on the training data and contains the content that best fits the user's question.

[0714] Step 4: The AI ​​records the user's questions and other dialogue data and summarizes them. The summary is generated, for example, using keyword extraction and text summarization techniques.

[0715] "Example 2"

[0716] Step 1: After the initial user response is completed, the AI ​​contacts the person in charge at the brick-and-mortar store. For example, the contact can be made by accessing the brick-and-mortar store's reservation system via an API and sending the user's information and summarized interaction data.

[0717] Step 2: A store representative will receive the call and confirm the user's reservation. A notification of the reservation confirmation will be sent to the user via email or SMS, for example.

[0718] "Example 3"

[0719] Step 1: The AI ​​summarizes interactions in the metaverse and stores them in a cloud-based database, for example by accessing the database through an API and sending the summary data.

[0720] Step 2: A store associate accesses the database and reviews the summary data, for example through a dedicated dashboard that gives the associate an overview of the user’s questions, the AI’s responses, and other important interaction data.

[0721] Example 1

[0722] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0723] Conventional user response systems in virtual space have difficulty in providing prompt and appropriate answers to user questions, and the recording and summarization of dialogue data is insufficient. As a result, there are problems with poor user experience and inability to smoothly link with physical stores. Furthermore, the quality of answers to user questions is not consistent, which can lead to low user satisfaction.

[0724] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0725] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing interactions in the virtual space so that they can be shared with the store, a means for receiving questions from users and analyzing the questions using natural language processing technology, a means for inputting a prompt sentence into a generative AI model and generating an appropriate answer, a means for providing the generated answer to the user, and a means for recording and summarizing dialogue data. This makes it possible to provide prompt and appropriate answers to user questions and effectively record and summarize dialogue data.

[0726] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0727] "AI that can explain a company's services" is artificial intelligence that automatically provides explanations in response to questions from users about the services a company offers.

[0728] A "virtual space" is a three-dimensional virtual environment constructed on the Internet, in which users can interact with other users and systems through avatars.

[0729] A "physical store" is a store that actually exists and is a location that users can visit in person.

[0730] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0731] A "generative AI model" is an artificial intelligence model that generates new information or answers based on input data.

[0732] A "prompt" is a sentence that is input into a generative AI model and serves as instructions for the model to generate an appropriate answer.

[0733] "Dialogue data" is a record of questions and answers exchanged between the user and the AI.

[0734] "Summarization" refers to extracting important parts of conversation data and summarizing them concisely.

[0735] This invention is a system that provides appropriate answers to questions from users in a virtual space and records and summarizes dialogue data. A specific embodiment of this system will be described below.

[0736] First, the server generates an AI program that uses natural language processing technology. This program has the ability to understand questions from users and generate appropriate answers. The hardware used is a high-performance server (e.g., a server equipped with an NVIDIA GPU). The software used is a natural language processing library (e.g., TensorFlow, PyTorch) and a database (e.g., MySQL).

[0737] The server places the generated AI program in a virtual space. When this AI receives a question from a user, it analyzes the question using natural language processing technology. Specifically, it performs the following data processing and calculations:

[0738] 1. Data preprocessing: Tokenize the user question and remove unnecessary characters and words.

[0739] 2. Question analysis: Analyze the intent of the question using tokenized data.

[0740] 3. Answer generation: Generate appropriate answers based on the intent of the question, using a generative AI model (e.g. GPT-3).

[0741] 4. Recording dialogue data: Record user questions and AI answers in a database.

[0742] 5. Data Summarization: Summarize the recorded interaction data for future reference.

[0743] As a concrete example, when a user asks an AI in a virtual space, "Please tell me the features of this product," the following process takes place.

[0744] 1. The user enters a question through an interface in the virtual space.

[0745] 2. The device sends the user's question to the server.

[0746] 3. The server receives the question and uses a natural language processing library to tokenize the question and remove unnecessary characters and words.

[0747] 4. The server uses the tokenized data to analyze the intent of the question and inputs the prompt sentence, "User: What are the features of this product? AI:" into the generative AI model.

[0748] 5. The generative AI model generates an answer such as, "The features of this product are as follows. First, ..."

[0749] 6. The server sends the generated response to the device.

[0750] 7. The device displays the answer to the user, who then confirms the answer.

[0751] 8. The server records the user's questions and the AI's answers in a database and summarizes the dialogue data.

[0752] An example of a prompt sentence would be something like this:

[0753] User: What are the features of this product?

[0754] AI: The features of this product are as follows. First,...

[0755] In this way, the server, terminal, and user can work together to provide appropriate answers to user questions and effectively record and summarize dialogue data.

[0756] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0757] Step 1:

[0758] The user types a question in the virtual space.

[0759] Specific operation: The user inputs a question through the interface in the virtual space. For example, the user inputs, "Please tell me the features of this product."

[0760] Input: The text of the user's question

[0761] Output: The question text is sent to the terminal.

[0762] Step 2:

[0763] The terminal transmits the user's question to the server.

[0764] Specific operation: The terminal sends the question entered by the user to the server, which transmits data via the Internet.

[0765] Input: Question text

[0766] Output: The question text is sent to the server.

[0767] Step 3:

[0768] The server receives the question and begins natural language processing.

[0769] Specific operation: The server receives the question sent from the device and analyzes the received question using a natural language processing library (e.g. TensorFlow, PyTorch).

[0770] Input: Question text

[0771] Output: Tokenized question data

[0772] Step 4:

[0773] The server analyzes the question and inputs a prompt into the generative AI model.

[0774] How it works: The server tokenizes the question and removes unnecessary characters and words. It then analyzes the intent of the question and inputs a prompt to the generative AI model (e.g. GPT-3). For example, it inputs the prompt in the format: "User: What are the features of this product? AI:".

[0775] Input: Tokenized question data

[0776] Output: Prompt statement

[0777] Step 5:

[0778] A generative AI model generates the appropriate answer.

[0779] Specific behavior: The generative AI model generates an appropriate answer based on the input prompt sentence, for example, "The features of this product are as follows. First, ..."

[0780] Input: prompt statement

[0781] Output: The generated answer text

[0782] Step 6:

[0783] The server sends the generated answer to the terminal.

[0784] Specific operation: The server sends the answer obtained from the generative AI model to the terminal, converting the answer data into an appropriate format before sending it.

[0785] Input: Generated answer text

[0786] Output: The answer text is sent to the terminal.

[0787] Step 7:

[0788] The terminal displays the answer to the user.

[0789] Specific operation: The terminal displays the answer received from the server to the user, who then checks the answer through an interface in the virtual space.

[0790] Input: Answer text

[0791] Output: The answer that is displayed to the user

[0792] Step 8:

[0793] The server records and summarizes the interaction data.

[0794] How it works: The server records the user's questions and the AI's answers in a database, then summarizes the recorded dialogue data for future reference.

[0795] Input: User's question text and AI's answer text

[0796] Output: Recorded conversation data and a summary

[0797] (Application example 1)

[0798] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0799] In conventional metaverse stores, when users had questions about products or services, it was difficult to get appropriate answers. In addition, there was a lack of a way to effectively record and summarize user questions and dialogue data and share them with staff at the physical store, so there was a need to improve the user experience. Furthermore, it was not possible to provide answers to user questions in real time, which could lead to a decrease in user satisfaction.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0801] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making them available for sharing with the store, a means for generating answers to user questions in real time and providing them to the user, and a means for recording dialogue data, summarizing it, and providing it to a store manager. This makes it possible for a user to receive an appropriate answer in real time when asking a question about a product or service in a metaverse store, and for dialogue data to be effectively recorded and summarized to be shared with a person in charge of a real store, improving the user experience.

[0802] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0803] "AI" is an abbreviation for artificial intelligence, a system that imitates human intellectual tasks using machine learning and natural language processing technologies.

[0804] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users and content through avatars.

[0805] "Real-time" refers to responsiveness in time to instantaneous response to user input.

[0806] "Dialogue data" is a record of the conversation between the user and the AI, and includes information such as questions and answers.

[0807] A "summary" is information that extracts the main points of dialogue data and summarizes them concisely.

[0808] A "store manager" is a person or organization in charge of operating a physical store or a metaverse store.

[0809] A "user" is a consumer or user who visits a metaverse store and asks questions about products and services.

[0810] A "real store" is a store that exists in a physical location where customers can visit in person to purchase goods or services.

[0811] A "store reservation" is the process by which a user reserves a date and time in advance to visit a physical store.

[0812] The following system configuration will be described as an embodiment of the present invention.

[0813] The server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user asks a question about a product or service in the Metaverse store, the AI ​​will use natural language processing technology to understand the user's question and generate an appropriate answer in real time. This answer will be provided to the user.

[0814] The server also records and summarizes the dialogue data, including the user's questions and the AI's responses, and provides it to the store manager, who can then more easily understand the user's needs and interests. In addition, after the initial user response, the system also includes a means to contact a brick-and-mortar store staff member or to schedule a visit.

[0815] This system is realized using the following hardware and software.

[0816] Hardware:

[0817] Smartphone or head mounted display

[0818] software:

[0819] Python

[0820] OpenAI API

[0821] When a user enters a question into the application, the server uses OpenAI's API to generate an appropriate answer to the user's question. The generated answer is displayed to the user. The interaction data is recorded, summarized, and provided to the store manager.

[0822] For example, when a user asks "Tell me about this product" in a virtual store, the AI ​​responds, "This product uses the latest technology and is very high performance." The conversation data is also recorded and summarized as "User: Tell me about this product. AI: This product uses the latest technology and is very high performance."

[0823] Example prompt:

[0824] User: Tell me about this product.

[0825] AI: This product uses the latest technology and is extremely powerful.

[0826] In this way, when users ask questions about products or services in the metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with the brick-and-mortar store staff, improving the user experience.

[0827] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0828] Step 1:

[0829] Users access the metaverse store using a terminal (smartphone or head-mounted display) and input questions about products and services.

[0830] Input: The text of the user's question

[0831] Output: The user's question text is sent to the server.

[0832] Step 2:

[0833] The server receives the user's question text and analyzes the intent of the question using natural language processing techniques.

[0834] Input: The text of the user's question

[0835] Output: Data analyzed to determine the intent of the question

[0836] Step 3:

[0837] The server uses the OpenAI API to generate appropriate answers based on the intent of the analyzed question.

[0838] Input: Data analyzed to determine the intent of the question

[0839] Output: Answer text generated from the OpenAI API

[0840] Step 4:

[0841] The server transmits the generated answer text to the user's terminal and displays it to the user.

[0842] Input: Answer text generated from the OpenAI API

[0843] Output: The answer text displayed to the user's terminal.

[0844] Step 5:

[0845] The server records the user's question text and the generated answer text as dialogue data.

[0846] Input: User question text, generated answer text

[0847] Output: Recorded interaction data

[0848] Step 6:

[0849] The server summarizes the recorded interaction data and provides it to the store manager.

[0850] Input: Recorded interaction data

[0851] Output: Summarized conversation data

[0852] Step 7:

[0853] After initial user response, the server provides a means to contact a person in charge at a real store or to make a reservation to visit the store.

[0854] Input: User question text, generated answer text, recorded dialogue data

[0855] Output: Contact a store representative or schedule a visit

[0856] In this way, when users ask questions about products or services in a metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with store personnel, improving the user experience.

[0857] Example 2

[0858] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0859] In conventional systems, users had to take the time and effort to contact a real store representative or make a reservation to visit after receiving the initial response. In addition, there was a lack of efficient and accurate means for users to communicate information when specifying the date and time of their visit or the service content. Furthermore, when sharing interactions in the virtual space with the real store, the information was not summarized or organized sufficiently, which could lead to delayed responses from the store.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0861] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing the interactions in the virtual space and sharing them with the store, a means for providing an interface for the user to input the date and time of visit and the service content, a means for saving the input data in a database, a means for sending a notification according to a specified contact method based on the saved data, and a means for feeding back the notification sending result to the user. This allows the user to efficiently and accurately specify the date and time of visit and the service content, and quickly contact the staff member at the real store. In addition, by summarizing and sharing the interactions in the virtual space, the store side can respond quickly and accurately.

[0862] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0863] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[0864] A "virtual space" is a digital environment generated by computer simulation, which users can access via the Internet and in which they can carry out various activities.

[0865] A "physical store" is a real store that actually exists and is a place where users can visit in person to receive services.

[0866] An "interface" is the means by which a user interacts with a system, and includes user interface elements such as input forms and buttons.

[0867] A "database" is a system for efficiently storing, managing, and searching data, and is operated using a query language such as SQL.

[0868] A "notification" is information sent from the system to a user or person in charge, and may be provided in the form of email, telephone call, message, etc.

[0869] "Feedback" refers to the response or result that a system provides to a user, informing the user of the outcome of an action they took.

[0870] A "summary" is a concise summary of the content of an exchange in a virtual space, in which important information is extracted and organized.

[0871] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0872] This invention is a system that allows users to contact a real store staff member or make a reservation to visit the store after receiving the initial response. The system includes AI, a user interface, a database, and a notification function installed in a virtual space.

[0873] Hardware and Software Used

[0874] The server is developed using the Python programming language and implemented as a web application using the Flask framework, with MySQL as the database and an SMTP server and Twilio API for notifications.

[0875] Specific operation of the system

[0876] The user accesses the system's web page using a web browser. The web page displays a form for inputting the date and time of visit and the desired service. For example, the user inputs that they would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[0877] The server receives the data entered by the user in the form and processes the request using the Flask framework, which stores the data in a MySQL database by generating SQL queries to insert the data into the database.

[0878] The server then uses the information stored in the database to send notifications according to the contact method the user specified: for example, if it's email, it uses an SMTP server to send an email, if it's phone, it uses the Twilio API to make an automated voice call, if you use a dedicated reservation system, it sends the reservation information through the reservation system's API, etc.

[0879] Finally, the server checks whether the notification was sent successfully and gives feedback to the user about the result. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[0880] Examples of concrete examples and prompts

[0881] As a concrete example, consider a user making a reservation at a hair salon. The user enters into a web form that they would like a "cut and color" service on "October 15, 2023 at 14:00." The server receives this information and sends a notification to the salon representative via email.

[0882] Example prompt:

[0883] If a user fills in a web form to make an appointment at a hair salon and requests a "cut and color" service on "October 15, 2023 at 2:00 PM," the server should receive this information and send a notification to the salon representative via email.

[0884] In this way, a system is realized that allows the user to specify the date and time of visit and the content of the service, taking into consideration the user's convenience.

[0885] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0886] Step 1:

[0887] The user enters the date and time of visit and the details of the service into a web form.

[0888] A user accesses the system's web page using a web browser and enters information into a form to input the date and time of visit and the desired service. For example, the user enters that he / she would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[0889] Input: Visit date and time, service details

[0890] Output: Data entered by the user

[0891] Step 2:

[0892] The server receives the input data and stores it in a database.

[0893] The server receives the data entered by the user in the form, processes the request using the Flask framework, and saves the input data to a MySQL database by generating SQL queries to insert into the database.

[0894] Input: Data entered by the user

[0895] Output: Data stored in the database

[0896] Step 3:

[0897] The server will send notifications based on the stored data and in accordance with the specified contact method.

[0898] The server sends notifications according to the contact method specified by the user based on the information stored in the database. For example, if it is email, it will send an email using an SMTP server. If it is phone, it will make an automated voice call using the Twilio API. If a dedicated reservation system is used, the reservation information will be sent via the reservation system's API.

[0899] Input: Data stored in the database, contact method

[0900] Output: Notification sent result

[0901] Step 4:

[0902] The server provides feedback on the notification sending result to the user.

[0903] The server checks whether the notification was sent successfully and gives the result as feedback to the user. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[0904] Input: Notification sending result

[0905] Output: A feedback message to the user.

[0906] (Application example 2)

[0907] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0908] Conventional user support systems in the metaverse had problems with insufficient coordination with real stores after the initial response, resulting in reduced user convenience. In addition, there was a lack of a way to select the appropriate person in charge based on the content of the user's inquiry and to contact them quickly, which led to a risk of user satisfaction decreasing. Furthermore, there was no reservation system that allowed users to specify the date and time of visit or the content of the service, making it difficult to flexibly respond to user needs.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0910] In this invention, the server includes a means for installing an AI inside a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making it possible to share them with the store, a means for the AI ​​to automatically select an appropriate staff member based on the content of the user's inquiry, a means for contacting the staff member via email, telephone, or a dedicated reservation system, and a means including a reservation system that allows the user to specify the date and time of visit and the content of the service. This makes it possible to quickly select and contact an appropriate staff member based on the content of the user's inquiry, thereby improving user convenience and satisfaction.

[0911] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[0912] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[0913] The "metaverse" is a digital environment built in a virtual space, a platform where users can interact with other users and systems through avatars.

[0914] A "real store" is a store that exists in a physical location and is a place where users can actually visit and receive products and services.

[0915] "Contact Person" refers to a store staff member or employee designated to handle user inquiries and reservations.

[0916] A "reservation system" is a system that enables a user to make a reservation by specifying the date and time of visit and the content of the service, and is used via email, telephone, or a dedicated online platform.

[0917] A "summary" is a concise summary of the content of interactions within the metaverse, including user questions and AI responses.

[0918] "Email" means a digital message sent over the Internet and is a means of exchanging information between users and personnel.

[0919] "Telephone" is a means of exchanging information in real time through voice communication, and is intended for direct communication between users and personnel.

[0920] "Automatic selection" is a process in which AI automatically selects the most suitable representative based on the content of the user's inquiry.

[0921] A system for implementing the present invention is configured as follows.

[0922] First, the server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user makes an initial inquiry within the Metaverse, the AI ​​will automatically generate a response based on the content of the inquiry and provide it to the user. This allows the user to quickly obtain the information they need.

[0923] Next, if the user's inquiry meets certain conditions, the server contacts the store's staff or schedules a visit. Specifically, the AI ​​automatically selects an appropriate staff member based on the user's inquiry and contacts the staff member via email, phone, or a dedicated reservation system. The contact includes the user's inquiry, the desired date and time of the visit, and the service details.

[0924] Additionally, interactions within the metaverse will be summarized and made available to stores, including the user's question and the AI's answer, allowing store staff to quickly understand the user's situation.

[0925] To realize this system, the server uses the following hardware and software: The hardware requires a server connected to the Internet and a user's smartphone. The software includes a Python program, an SMTP server (e.g. smtp.example.com), and a reservation system.

[0926] As a specific example, if a user wants to "consult about purchasing a new smartphone" and wants to make a reservation to visit the store at 14:00 on October 15, 2023, the following prompt sentence will be generated.

[0927] Username: Yamada Taro

[0928] Inquiry: Consultation about purchasing a new smartphone

[0929] Desired date and time: 2023-10-15 14:00

[0930] contact address:

[0931] Email: taro.yamada@example.com

[0932] Phone: 090-1234-5678

[0933] Based on this prompt, the server contacts the person in charge and confirms the user's reservation, allowing the user to smoothly receive services at a physical store.

[0934] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0935] Step 1:

[0936] A user makes an initial inquiry within the metaverse.

[0937] Input: User's inquiry (e.g. "Consultation on purchasing a new smartphone")

[0938] Processing: The user's inquiry is received and the AI ​​analyzes it.

[0939] Output: Parsed query content

[0940] Step 2:

[0941] AI automatically generates answers based on the user's inquiries and provides them to the user.

[0942] Input: Parsed query content

[0943] Processing: The AI ​​uses the generative AI model to generate the appropriate answer.

[0944] Output: The generated answer (e.g. "When it comes to buying a new smartphone, we recommend the following model...")

[0945] Step 3:

[0946] If the user's inquiry meets certain conditions, the server contacts a person in charge at the physical store or makes a reservation to visit the store.

[0947] Input: Analyzed inquiry details, user's desired visit date and time, service details

[0948] Processing: AI automatically selects the appropriate person to contact based on the inquiry and generates a prompt.

[0949] Output: Generated prompt text (Example: "User name: Yamada Taro, Inquiry: Consultation on purchasing a new smartphone, Desired date and time: 2023-10-15 14:00, Contact: Email: taro.yamada@example.com, Phone: 090-1234-5678")

[0950] Step 4:

[0951] The server will contact the person via email, phone, or a dedicated reservation system.

[0952] Input: Generated prompt text

[0953] Processing: The server uses an SMTP server to send an email to a person in charge, or uses a phone API to automatically make a phone call, or send a prompt to a reservation system.

[0954] Output: Contact the person in charge (e.g., the reservation information will be sent to the person in charge's mailbox)

[0955] Step 5:

[0956] Interactions within the metaverse are summarized and shared for stores.

[0957] Input: User question, AI answer

[0958] Processing: The server summarizes interactions within the metaverse and generates summary information.

[0959] Output: Generated summary information (e.g. "User question: Looking for advice on buying a new smartphone. AI answer: We recommend the following model...")

[0960] Step 6:

[0961] Share summary information for stores.

[0962] Input: Generated summary information

[0963] Processing: The server sends summary information to the person in charge.

[0964] Output: The agent receives summary information to quickly understand the user's situation.

[0965] Example 3

[0966] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0967] There is a need to efficiently collect, summarize, and share conversation data between users and AI in the metaverse with store staff in real stores in order to provide better services based on users' needs and questions. However, in current systems, the collection, summarization, and sharing of conversation data is often done manually, which makes it inefficient and lacks real-time capabilities.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0969] In this invention, the server includes a means for collecting dialogue data in the metaverse, a means for storing the collected dialogue data in a cloud-based database, a means for summarizing the dialogue data using a generative AI model, a means for storing the summary data in the cloud-based database, and a means for allowing a person in a physical store to view the summary data. This makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time.

[0970] The "metaverse" is a three-dimensional virtual space constructed using virtual reality technology, in which users can interact with other users and AI through avatars.

[0971] "Interaction Data" refers to data that includes questions, answers, and other significant content of conversations between users and AI within the metaverse.

[0972] A "cloud-based database" is a database service accessible via the Internet in which data is stored, managed, and accessed in a cloud environment.

[0973] A "generative AI model" is a model that uses artificial intelligence technology to generate, summarize, and convert text data, such as a model that uses natural language processing technology.

[0974] "Summary data" is data that concisely summarizes important information extracted from dialogue data by a generative AI model.

[0975] "Brick and mortar store staff" refers to staff who deal with customers and provide services in a physical store.

[0976] "Viewing" is the act of displaying and checking stored data or information.

[0977] This invention is a system that efficiently collects dialogue data between users and AI in the metaverse, summarizes the data, and shares it with staff in real stores. A specific embodiment of this system will be described below.

[0978] The server collects real-time user-AI interaction data within the metaverse, including user questions, AI responses, and other important dialogue content. The server stores the collected interaction data in a cloud-based database (e.g., Amazon RDS or Google Cloud SQL).

[0979] The server then uses a generative AI model (e.g., a model using natural language processing techniques) to summarize the collected dialogue data. This summary includes the key points of the user's question and the AI's answer. The summarized information is again stored in a cloud-based database.

[0980] Store staff can use devices to access the cloud-based database and view summary information, allowing them to understand the content of interactions between users and the store staff in the metaverse and provide better service.

[0981] As a concrete example, consider a case where a user asks in the metaverse, "Is a new product in stock?" and an AI responds, "Yes, it's in stock." This dialogue data is collected by the server and a generative AI model is used to generate a summary: "The user checked the availability of a new product, and the AI ​​responded that it is in stock." This summary information is stored in a cloud-based database and can be accessed by staff at the brick-and-mortar store.

[0982] Example prompt:

[0983] Summarize the user's question and the AI's answer.

[0984] User Question: Are new items in stock?

[0985] AI answers: Yes, we have it in stock.

[0986] By inputting this prompt sentence into the generative AI model, summarized information is obtained. This system makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0987] Step 1:

[0988] The server collects the dialogue data between the user and the AI ​​in the metaverse in real time. Specifically, when the user asks "Is the new product in stock?" in the metaverse and the AI ​​responds "Yes, it is in stock," the server captures this dialogue data. The input is the user's question and the AI's answer, and the output is the collected dialogue data.

[0989] Step 2:

[0990] The server stores the collected dialogue data in a cloud-based database. Specifically, the server inserts the dialogue data into an appropriate table in the database. The input is the collected dialogue data, and the output is the dialogue data stored in the database.

[0991] Step 3:

[0992] The server inputs the saved dialogue data into the generative AI model and generates a summary. Specifically, it inputs the following prompt sentence into the generative AI model:

[0993] Summarize the user's question and the AI's answer.

[0994] User Question: Are new items in stock?

[0995] AI answers: Yes, we have it in stock.

[0996] The input is the prompt and the dialogue data, and the output is the generated summary data.

[0997] Step 4:

[0998] The server again stores the generated summary data in the cloud-based database. Specifically, the server inserts the summary data into an appropriate table in the database. The input is the generated summary data, and the output is the summary data stored in the database.

[0999] Step 5:

[1000] Using a terminal, a store associate accesses the cloud-based database and views the summary data. Specifically, the associate queries the database, retrieves the summary data, and displays it. The input is a request to access the database, and the output is the displayed summary data.

[1001] This series of processes makes it possible to efficiently collect, summarize, and share conversation data within the metaverse in real time.

[1002] (Application example 3)

[1003] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1004] In order for store personnel to understand the content of conversations between users and retailers in the metaverse and provide better services, a means of efficiently summarizing and sharing the content of the conversations is necessary. However, with current systems, it is difficult to summarize the conversation data in the metaverse in real time, store it in a cloud-based database, and have store personnel in retailers access it using their smartphones. This poses the challenge of quickly understanding the needs and interests of users and providing personalized services.

[1005] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1006] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing interactions in the metaverse and making it possible to share with the store, a means for storing the summary information in a cloud-based database and making it possible for a person in charge of a real store to access using a smartphone, and a means for including user questions, AI responses, and other important dialogue data in the summary information. This makes it possible to efficiently summarize the content of the dialogue in the metaverse and for a person in charge of a real store to quickly access it, thereby making it possible to grasp the needs and interests of the user and provide personalized services.

[1007] "CS Data" is data related to customer service, including customer inquiries, feedback, service history, etc.

[1008] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[1009] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users through avatars and carry out various activities.

[1010] A "physical store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1011] A "cloud-based database" is a database that is accessible via the internet and in which data is stored and managed by a cloud service provider.

[1012] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[1013] "Summary information" is information that briefly summarizes detailed data or dialogue content and includes important points and main points.

[1014] A "user question" is an inquiry or question that a user makes to an AI within the metaverse.

[1015] An "AI answer" is a response or explanation that an AI generates in response to a user's question.

[1016] "Important dialogue data" refers to information or data that is considered to be particularly important in the dialogue between the user and AI, and contains content that is useful in providing services.

[1017] In order to implement the present invention, the following system configuration and processing procedure are used.

[1018] First, the server installs an AI inside the metaverse store that can learn CS data and explain the company's services. When a user asks the AI ​​a question in the metaverse, the AI ​​automatically generates an answer to the question and provides it to the user. This allows the user to obtain information about the company's services in the metaverse.

[1019] Next, after the initial user response, the server contacts the person in charge at the brick-and-mortar store and, if necessary, makes a reservation to visit the store. This completes the preparations for the user to receive services at the brick-and-mortar store.

[1020] In addition, the server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The summary information includes the user's questions, the AI's answers, and other important dialogue data. Store staff can access the cloud-based database using their smartphones to check the summary information. This allows store staff to quickly understand the user's needs and interests and provide personalized services.

[1021] As a concrete example, a user may ask in the metaverse, "What are the features of my new smartphone?" and the AI ​​may respond, "The new smartphone has a high-resolution camera and a long-lasting battery." This conversation is summarized and stored in a cloud-based database. A store associate can access this summary information using their smartphone and understand that the user is interested in the new smartphone.

[1022] Examples of prompts to input to a generative AI model include:

[1023] A user asks in the metaverse, "What are the features of my new smartphone?" and the AI ​​responds, "My new smartphone has a high-resolution camera and a long-lasting battery." Summarize this conversation:

[1024] In this way, the server, terminal, and user work together to efficiently summarize the content of conversations in the metaverse, which can then be quickly accessed by store personnel in the physical store, enabling them to understand the needs and interests of the user and provide personalized services.

[1025] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1026] Step 1:

[1027] The server will install AI inside Metaverse stores that can learn CS data and other data and explain the company's services. When a user asks the AI ​​a question within the Metaverse, the AI ​​will automatically generate an answer to that question and provide it to the user. The input is the user's question, and the output is the AI's answer. In concrete terms, it uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1028] Step 2:

[1029] After initially responding to the user, the server contacts the person in charge at the physical store and, if necessary, makes a reservation to visit the store. The input is the user's question and the AI's response, and the output is contact information for the person in charge. In concrete terms, the server sends a notification to the person in charge based on the content of the user's question and the AI's response, and processes the reservation to visit the store.

[1030] Step 3:

[1031] The server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The input is the user's question and the AI's answer, and the output is the summary information. Specifically, the server uses a generative AI model to summarize the dialogue and stores it in a cloud-based database.

[1032] Step 4:

[1033] A staff member at a brick-and-mortar store uses a smartphone to access the cloud-based database and check the summary information. The input is the summary information obtained from the cloud-based database, and the output is the staff member's understanding. The specific operation is to access the database through a smartphone application and display the summary information.

[1034] Step 5:

[1035] Based on the summary information, the staff member understands the user's needs and interests and provides personalized services. The input is the summary information, and the output is the service to be provided. Specifically, the system refers to the summary information and suggests products and services that are suitable for the user.

[1036] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1037] "Example 1"

[1038] In one embodiment of the present invention, the AI ​​includes an emotion engine that infers the user's emotional state from the user's questions and other dialogue data. Specifically, the emotion engine uses natural language processing and voice analysis techniques to infer emotions from the user's text and voice inputs. It also uses image analysis and motion analysis techniques to analyze the user's facial expressions and body movements.

[1039] "Example 2"

[1040] The user's emotional state estimated by the emotion engine is reflected in the AI's response generation. Specifically, if the user shows joy or excitement, the AI ​​generates a response that emphasizes that emotion. Conversely, if the user shows discomfort or confusion, the AI ​​generates a response that soothes that emotion. This allows the AI ​​to respond appropriately to the user's emotions.

[1041] "Example 3"

[1042] The emotion engine also influences summaries of interactions in the metaverse. Specifically, the user's emotional state is recorded as part of the summary, which is then shared with the store. For example, if a user expresses strong excitement about a product, that information is shared with the store to help predict the product's popularity and sales. If a user expresses dissatisfaction with a service, that information is shared with the store to help improve the service.

[1043] The process flow of each embodiment will be described below.

[1044] "Example 1"

[1045] Step 1: The AI ​​receives a question and other interaction data from the user.

[1046] Step 2: The emotion engine estimates the user's emotional state from the received data. Specifically, it estimates emotions from text and voice input using natural language processing and voice analysis technologies. It also estimates emotions from the user's facial expressions and body movements using image analysis and motion analysis technologies.

[1047] "Example 2"

[1048] Step 1: The emotion engine estimates the user's emotional state.

[1049] Step 2: The AI ​​generates answers based on the estimated emotional state. Specifically, if the user is happy or excited, the AI ​​generates answers that emphasize those emotions. Conversely, if the user is uncomfortable or confused, the AI ​​generates answers that soften those emotions.

[1050] "Example 3"

[1051] Step 1: The emotion engine estimates the user's emotional state and reflects the result in a summary of the interaction in the metaverse.

[1052] Step 2: The summary is shared with the store. Specifically, information about the user's excitement about the product and dissatisfaction with the service is shared with the store.

[1053] Step 3: Stores use the shared information to predict product popularity and sales, and use it as a reference for improving services.

[1054] Example 1

[1055] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1056] Conventional user response systems in virtual space have difficulty generating appropriate answers to user questions and are unable to accurately grasp the user's emotional state. As a result, there were problems with poor user experience, poor guidance to physical stores, and poor improvement in the quality of service. Furthermore, there was a lack of a way to effectively share interactions in virtual space with physical stores, which led to a lack of consistency in user response.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1058] In this invention, the server includes a means for installing an artificial intelligence in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing the exchange in the virtual space so that it can be shared with the store, a means for analyzing the user's questions using natural language processing technology and generating appropriate answers, a means for using voice analysis technology, image analysis technology, and motion analysis technology to estimate the user's emotional state, and a means for recording and summarizing the dialogue data. This makes it possible to generate appropriate answers to the user's questions and accurately grasp the user's emotional state. In addition, by effectively sharing the exchange in the virtual space with the real store, it is possible to maintain consistency in user responses and improve the quality of service.

[1059] "Artificial intelligence" refers to a system that includes techniques for generating appropriate answers to user questions and estimating the user's emotional state.

[1060] A "virtual space" is a three-dimensional virtual environment constructed on the Internet, in which users can interact with other users and systems through avatars.

[1061] "Natural language processing technology" is a technology that analyzes text input from a user, understands their intent, and generates an appropriate response.

[1062] "Voice analysis technology" is a technology for converting voice input from a user into text and analyzing that text.

[1063] "Image analysis technology" is a technology that captures a user's facial expressions and body movements with a camera and estimates their emotional state by analyzing the data.

[1064] "Motion analysis technology" is a technology that analyzes the user's physical movements and infers their emotional state based on that data.

[1065] "Dialogue data" is a record of the interactions between a user and artificial intelligence, and includes information such as questions asked, answers given, and emotional states.

[1066] A "summary" is a concise summary of important information extracted from dialogue data.

[1067] A "brick and mortar store" is a store that exists in a physical location and that a user can actually visit.

[1068] A "store reservation" is a procedure in which a user reserves a date and time in advance to visit a physical store.

[1069] The present invention relates to an artificial intelligence system that explains a company's services to users in a virtual space. A specific embodiment of this system will be described below.

[1070] First, the server generates a program that combines natural language processing technology, voice analysis technology, image analysis technology, and motion analysis technology. This program includes artificial intelligence to understand questions from users and generate appropriate answers. Specifically, the following technologies are used:

[1071] Natural language processing technology: The server analyzes the text input from the user and understands the intent of the question. For this technology, for example, BERT or GPT-3 is used.

[1072] Speech analysis technology: The server converts the user's voice input into text and analyzes the text. This technology uses Google Cloud Speech-to-Text and Amazon Transcribe.

[1073] Image analysis technology: The server captures the user's facial expressions and body movements with a camera and estimates their emotional state. This technology uses OpenCV and Google Cloud Vision.

[1074] Motion analysis technology: The server analyzes the user's body movements and estimates their emotional state. This technology uses Microsoft Azure Kinect DK.

[1075] Next, the user accesses the AI ​​in the virtual space. The user logs into a virtual store in the virtual space and accesses the AI ​​assistant. The user starts a conversation by clicking on the AI ​​assistant's icon or avatar in the virtual space.

[1076] When a user inputs a question, either by text or voice, the server receives the input and selects the appropriate analysis technology: natural language processing technology for text input, and voice analysis technology for voice input.

[1077] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer. For example, in response to a question such as "What are the features of this product?", the server generates an answer such as "This product is made of high-quality materials and is durable."

[1078] In addition, the server uses an emotion engine to estimate the user's emotional state. In the case of text input, natural language processing technology is used to analyze emotions, and in the case of voice input, voice analysis technology is used. In addition, the server analyzes the user's facial expressions and body movements through the camera to estimate the user's emotional state more accurately.

[1079] The server returns the generated answer to the user, either by displaying it in a text chat window or by playing the answer aloud, for example, "This product is made of high quality materials and is durable."

[1080] Finally, the server records the interaction data with the user and summarizes it for later analysis and improvement, for example recording the user's questions and emotional state to be used in the next interaction.

[1081] Examples:

[1082] User accesses AI in a virtual space: The user clicks on the AI ​​assistant's avatar in the virtual store and begins interacting with it.

[1083] User types question in via text or voice: The user types in text, such as "What are the features of this product?", or speaks the question into the microphone.

[1084] The server receives the input and selects the appropriate parsing technique: The server recognizes that the user's input is text and selects a natural language processing technique.

[1085] The server analyzes the question using natural language processing technology: The server analyzes the question using GPT-3 and extracts information about the product's features.

[1086] The server generates an appropriate answer: The server generates an answer such as "This product is made of high quality materials and is durable."

[1087] Server infers user's emotional state: The server analyzes emotions from the user's text input and infers whether the user is happy or not.

[1088] Server returns answer to user: The server displays the generated answer in a text chat window.

[1089] The server records and summarizes the dialogue data: The server records the user's questions and emotional state and summarizes them for use in the next dialogue.

[1090] Example prompt:

[1091] "Please tell me the features of this product."

[1092] "How long can this product be used for?"

[1093] Could you please tell me the pricing plan for this service?

[1094] In this way, the server can provide better service by generating appropriate answers to the user's questions and estimating the user's emotional state.

[1095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1096] Step 1:

[1097] The user accesses the AI ​​in a virtual space.

[1098] Specific operations: The user logs into a virtual store in a virtual space and begins a dialogue by clicking on the AI ​​assistant's icon or avatar.

[1099] Input: User login information, location in the virtual world.

[1100] Output: Start of a conversation session with the AI ​​assistant.

[1101] Step 2:

[1102] The user enters a question by text or voice.

[1103] Specific Actions: The user types a question into a text chat window or speaks a question using a microphone, for example, "What are the features of this product?"

[1104] Input: User text or voice input.

[1105] Output: Text or audio data.

[1106] Step 3:

[1107] The server receives the input and selects the appropriate analysis technique.

[1108] Specific operation: The server determines whether the input from the user is text or voice, and selects natural language processing technology in the case of text input and voice analysis technology in the case of voice input.

[1109] Input: Text or audio data.

[1110] Output: The result of the analysis technique selection.

[1111] Step 4:

[1112] The server analyzes the question using natural language processing techniques.

[1113] Specific operation: The server uses natural language processing technology (e.g., GPT-3) to analyze the user's question and understand the intent of the question.

[1114] Input: Text data.

[1115] Output: Analysis of the intent of the question.

[1116] Step 5:

[1117] The server generates an appropriate response.

[1118] Specific behavior: The server generates an appropriate answer based on the analysis results, for example, "This product is made of high-quality materials and is durable."

[1119] Input: The analysis results for the intent of the question.

[1120] Output: The answer text.

[1121] Step 6:

[1122] The server estimates the user's emotional state.

[1123] Specific operation: The server uses an emotion engine to estimate the user's emotional state. In the case of text input, natural language processing technology is used to analyze emotions, and in the case of voice input, voice analysis technology is used. In addition, the server analyzes the user's facial expressions and body movements through the camera to more accurately estimate the user's emotional state.

[1124] Input: text data, audio data, image data.

[1125] Output: Emotion state estimation result.

[1126] Step 7:

[1127] The server returns the answer to the user.

[1128] Specific Behavior: The server returns the generated answer to the user, either by displaying it in a text chat window or by playing the answer aloud, for example, "This product is made of high-quality materials and is durable."

[1129] Input: The answer text.

[1130] Output: Display or audio the answer to the user.

[1131] Step 8:

[1132] The server records and summarizes the interaction data.

[1133] Specific operation: The server records the interaction data with the user and summarizes it for later analysis and improvement. For example, it records the user's questions and emotional state and uses them in the next interaction.

[1134] Input: Dialogue data, emotion state estimation results.

[1135] Output: Summarized interaction data.

[1136] (Application example 1)

[1137] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1138] Conventional user response systems in virtual space have had issues with providing appropriate answers to user questions and responding in a way that takes into account the user's emotional state. In addition, there was a lack of a way to effectively share interactions in virtual space with physical stores, and there was a need to improve the user experience.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a physical store or making a reservation for a visit after an initial user response, a means for summarizing the exchange in the virtual space so that it can be shared with the store, a means including an emotion engine for estimating the user's emotional state, a means for using natural language processing technology for analyzing the user's voice and text input, and a means for using image analysis technology for analyzing the user's facial expressions and actions. This makes it possible to provide appropriate answers to the user's questions and respond in consideration of the emotional state, and to effectively share the exchange in the virtual space with the physical store.

[1140] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1141] A "virtual space" is a three-dimensional virtual environment constructed using computer technology, which is a space that users can experience interactively.

[1142] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[1143] A "physical store" is a real, physical store that exists and is a location that customers can visit in person.

[1144] An "emotion engine" is a technology for estimating a user's emotional state, and is a system that recognizes emotions using voice analysis and image analysis.

[1145] "Natural language processing technology" is a technology that enables computers to understand and generate human language, and includes text analysis and speech recognition.

[1146] "Image analysis technology" refers to technology that analyzes image data and extracts information, including object recognition and facial recognition.

[1147] A "summary" is a short summary of a long piece of text or dialogue data, in which important information is extracted and expressed concisely.

[1148] "Contact" is the act of transmitting information, and is done using means such as telephone or email.

[1149] A "store reservation" is the act of a customer reserving a date and time to visit a physical store in advance, allowing the store to prepare for the visit.

[1150] A system for implementing the present invention has the following configuration.

[1151] First, the server installs an AI in a virtual space that can learn CS data and explain the company's services. Users can access the virtual space using a smartphone or head-mounted display and interact with the AI. The AI ​​automatically generates answers to the user's questions and provides them to the user.

[1152] Next, after the initial user response, the server contacts the person in charge at the physical store or makes a reservation to visit the store. This allows the user to receive services at the physical store based on the information obtained in the virtual space.

[1153] In addition, the server will summarize the virtual interactions and make them available to the store, allowing staff at the physical store to see what questions users asked in the virtual world and what answers they received.

[1154] The system includes an emotion engine that estimates the user's emotional state. The emotion engine uses natural language processing technology to analyze the user's voice and text input. It also uses image analysis technology to analyze the user's facial expressions and movements. This allows the system to estimate the user's emotional state and respond appropriately.

[1155] As a specific example, if a user asks, "Tell me about the latest smartphone," the AI ​​will respond, "It's the latest smartphone. It has a high-performance camera and a long-lasting battery." In addition, if a user asks, "Where can I buy this product?", the AI ​​will respond, "You can buy this product at the online store."

[1156] To realize this system, the following hardware and software are used.

[1157] Hardware: Smartphone, head-mounted display

[1158] Software: OpenAI API (generative AI model), Transformers library (natural language processing technology), OpenCV (image analysis technology)

[1159] An example of a prompt is as follows:

[1160] "Tell me about the latest smartphones."

[1161] "Where can I buy this product?"

[1162] "What are the features of the head mounted display?"

[1163] This allows users to receive the services of a physical store through an interactive experience in a virtual space.

[1164] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1165] Step 1:

[1166] The user accesses the virtual space using a smartphone or head-mounted display. The user inputs questions to the AI ​​assistant in the virtual space. Input is done by text or voice. The input data is sent to the server.

[1167] Step 2:

[1168] The server analyzes the received user question using natural language processing technology. Specifically, it uses the OpenAI API to understand the intent of the question and generate an appropriate answer. The input is the user's question, and the output is the generated answer.

[1169] Step 3:

[1170] The server provides the generated answer to the user. The user checks the answer in the virtual space. The output is the answer that is provided to the user.

[1171] Step 4:

[1172] The server analyzes the user's voice and text input with an emotion engine to estimate the user's emotional state. The emotion engine uses the Transformers library to recognize emotions. The input is the user's voice and text data, and the output is the estimated emotional state.

[1173] Step 5:

[1174] The server analyzes the user's facial expressions and movements using image analysis technology. Specifically, it uses OpenCV to recognize the user's facial expressions and movements and complement the emotional state. The input is the user's image data, and the output is the analyzed facial expression and movement information.

[1175] Step 6:

[1176] The server summarizes the virtual interaction and shares it with the physical store staff. The summary includes the user's questions and the AI's answers. The input is the virtual interaction data, and the output is the summarized information.

[1177] Step 7:

[1178] After the initial user interaction, the server contacts the person in charge at the physical store or schedules a visit. The means of communication are email and phone. The input is the user's contact information and the desired reservation date and time, and the output is a contact or reservation confirmation.

[1179] Step 8:

[1180] A user visits a physical store and receives services based on the information obtained in the virtual space. The staff at the physical store respond based on the summary information shared by the server. The input is the summary information, and the output is the provision of services to the user.

[1181] Example 2

[1182] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1183] Conventional systems had the problem of low user satisfaction because they responded uniformly without considering the user's emotional state. There were also issues with the time required to contact a brick-and-mortar store staff after the initial response and the complicated procedure for making a reservation to visit the store. Furthermore, there was a lack of functionality to specify the date and time of the user's visit and the content of the service, making it difficult to accommodate the user's needs.

[1184] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received input data to estimate the emotional state of the user, a means for generating an answer using a generation AI model based on the estimated emotional state, a means for providing the generated answer to the user, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, and a means for allowing the user to specify the date and time of the user's visit and the service content. This enables an appropriate response according to the user's emotional state, simplifies the contact and reservation procedures after the initial response, and provides convenience to the user.

[1185] "Means for receiving user input data" refers to a function for sending text and voice data input by a user through a terminal to a server and receiving the same.

[1186] "Means for analyzing received input data and inferring the emotional state of a user" refers to a function for analyzing received input data of a user and inferring the emotional state of the user (happiness, excitement, discomfort, confusion, etc.) from that data.

[1187] "Means for generating an answer using a generative AI model" is a function for generating an appropriate answer for a user using a generative AI model based on an estimated emotional state.

[1188] "Means for providing the generated answer to the user" refers to a function for sending and providing the answer generated by the generative AI model to the user.

[1189] "A means of contacting a physical store staff member or making a reservation to visit the store after the initial response to the user" refers to a function for contacting a physical store staff member or making a reservation to visit the store after the initial response has been completed.

[1190] "A means for allowing a user to specify the date and time of visit and the content of the service" is a function that enables a user to specify the date and time of visit and the content of the service that the user wishes to receive.

[1191] An "emotion engine" is software or algorithms for analyzing user input data and inferring an emotional state from that data.

[1192] A "generative AI model" is an artificial intelligence model that generates answers in natural language based on a user's input data and emotional state.

[1193] "Email, telephone, or dedicated reservation system" refers to the means of communication or reservation management system used to communicate between the user and the brick-and-mortar store personnel.

[1194] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1195] The present invention provides a system for responding appropriately to a user's emotional state and simplifying communication and reservation procedures after the initial response. A specific embodiment of this system will be described below.

[1196] A means of receiving user input data

[1197] A user accesses the system using a terminal (e.g., a PC or smartphone) and enters a question or information. The data entered by the user is sent from the terminal to the server. The server has a high-performance processor (e.g., an Intel Xeon processor) to process the received data.

[1198] A means of estimating the user's emotional state by analyzing received input data

[1199] The server passes the received user input data to an emotion engine (e.g., IBM Watson Tone Analyzer) for emotion analysis. The emotion engine estimates the user's emotional state (happiness, excitement, discomfort, confusion, etc.) from the text. The analysis results are fed back to the generative AI model.

[1200] A means of generating answers using a generative AI model based on an inferred emotional state

[1201] The server generates a response to the user using a generative AI model (e.g., OpenAI's GPT-4) based on the emotional state obtained from the emotion engine. For example, if the user expresses joy, the AI ​​generates a positive response such as, "That's great! I'm glad you're interested."

[1202] A means of providing the generated answers to the user

[1203] The server sends and provides the answer generated by the generative AI model to the user, who can receive the answer through their device.

[1204] After initial user response, a means to contact a brick-and-mortar store representative or make a reservation to visit the store

[1205] If the user is satisfied with the initial response, the server contacts the person in charge at the physical store or makes a reservation to visit the store. This contact can be done via email (e.g., Gmail API), phone (e.g., Twilio API), or a dedicated reservation system (e.g., Calendly).

[1206] A means to specify the date and time of the user's visit and the content of the service

[1207] When making a reservation, the user can specify the date and time of visit and the service content. The server inputs this information into the reservation system and sends a reservation confirmation email to the user. It also notifies the person in charge at the physical store of the reservation information.

[1208] Examples

[1209] Example 1: When a user expresses joy

[1210] A user types, "I love your new product!" The emotion engine detects "delight," and the generative AI model responds, "That's great! We're glad you're interested. If you'd like more information, please reach out to us or schedule an appointment."

[1211] Example 2: When the user expresses confusion

[1212] A user types, "I don't really understand how to use this product." The emotion engine detects "confusion," and the generative AI model responds, "We apologize for the inconvenience. If you need further explanation, please contact our sales representative or schedule an appointment to visit."

[1213] Examples of prompt statements

[1214] "If a user is excited about a new product, generate a positive response."

[1215] "If a user is confused about how to use your product, generate a reassuring answer."

[1216] In this way, the system will respond appropriately depending on the user's emotional state, and if necessary, will contact a physical store staff member or make a reservation to visit the store.

[1217] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1218] Step 1: Receive user input data

[1219] A user accesses the system using a terminal (e.g., a PC or smartphone) and inputs a question or information. The data entered by the user is sent from the terminal to the server. In concrete terms, the user opens a browser, accesses the system's web page, enters a question in a text box, and clicks the "Submit" button. The input data is the user's question or information (e.g., "Please tell me the details of a new product"), and the output is the user's input data sent to the server.

[1220] Step 2: Analyze the received input data to infer the user's emotional state

[1221] The server passes the received user input data to an emotion engine (e.g. IBM Watson Tone Analyzer) for emotion analysis. The input data is the user's text data, and the emotion engine analyzes this data to estimate the emotional state (happiness, excitement, discomfort, confusion, etc.). In concrete terms, the server sends the user's input data to the emotion engine, which analyzes the text, estimates the emotional state, and returns the result to the server. The output is the estimated emotional state.

[1222] Step 3: Generate answers using a generative AI model based on the estimated emotional state

[1223] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate an answer for the user based on the emotional state obtained from the emotion engine. The input data is the emotion analysis result, and the generative AI model uses this data to generate an appropriate answer. In concrete terms, the server passes the emotion analysis result to the generative AI model, which then generates an answer according to the emotional state. The output is the generated answer.

[1224] Step 4: Provide the generated answer to the user

[1225] The server sends and provides the answer generated by the generative AI model to the user. The input data is the generated answer, and the server sends this answer to the user. In specific operations, the server sends the generated answer to the user's terminal, and the user receives the answer through the terminal. The output is the answer provided to the user.

[1226] Step 5: After initial user interaction, contact a brick-and-mortar store representative or schedule a visit

[1227] If the user is satisfied with the initial response, the server will contact the person in charge at the physical store or make a reservation to visit. The input data is the user's satisfaction level and desired contact method or reservation information, and the server will contact or make a reservation based on this. In concrete terms, the user clicks the "Book a visit" button, and the server accesses the reservation system and inputs the user's desired date and time and service details. The output is a reservation confirmation email and / or a contact notification.

[1228] Step 6: Specify the date and time of the user's visit and the type of service

[1229] When making a reservation to visit a store, the user can specify the date and time of visit and the details of the service. The input data is the user's desired date and time of visit and the details of the service, which the server inputs into the reservation system. In concrete terms, the server inputs the user's desired date and time and the details of the service into the reservation system, sends a reservation confirmation email to the user, and also notifies the person in charge at the physical store of the reservation information. The output is a reservation confirmation email and a notification to the person in charge.

[1230] (Application example 2)

[1231] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1232] In the conventional metaverse store system, it was difficult to respond appropriately according to the user's emotional state, which led to a poor user experience. In addition, when a user expressed discomfort or confusion, there was a lack of a way to quickly contact a real store staff member, making it difficult to improve user satisfaction.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1234] In this invention, the server includes a means for installing an AI inside a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a store visit reservation after initial user response, a means for summarizing exchanges in the metaverse and making them available for sharing with the store, a means for the AI ​​to generate an appropriate response based on the user's emotional state estimated by the emotion engine, and a means for contacting a staff member when the user expresses discomfort or confusion. This enables appropriate responses according to the user's emotional state, improving the user experience and quickly resolving problems.

[1235] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1236] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate human intelligence to learn and make inferences.

[1237] "Metaverse" refers to a virtual space, a 3D virtual world built on the Internet.

[1238] An "emotion engine" is an algorithm or system for estimating a user's emotional state, and is a technology that analyzes emotions using natural language processing and machine learning.

[1239] A "real store" refers to a store that exists in a physical location and that customers can actually visit.

[1240] A "store reservation" refers to a customer making a reservation to visit a store at a specific date and time.

[1241] A summary is a concise summary of detailed information, extracting the most important points and expressing them briefly.

[1242] "User experience" refers to the experience and satisfaction a user feels when using a product or service.

[1243] "Quick problem resolution" refers to quickly responding to and resolving problems that arise.

[1244] The system for implementing this invention is configured as follows. First, the server installs an AI inside a metaverse store that learns CS data and can explain the company's services. When a user makes an inquiry in the metaverse, the emotion engine estimates the user's emotional state, and the AI ​​generates an appropriate answer based on that emotion.

[1245] If the user expresses discomfort or confusion, the server will contact the person in charge at the real store. The means of communication will be email, telephone, or a dedicated reservation system. If the user wishes to make a reservation to visit the store, they can specify the date and time of the visit and the service content.

[1246] Interactions within the metaverse are summarized and shared with the store, including the user's question and the AI's answer, allowing store personnel to quickly understand the user's situation and take appropriate action.

[1247] Hardware and software used

[1248] Hardware: Smartphones, servers

[1249] Software: Python, smtplib (sending email), random (simulating emotion engine)

[1250] Data processing and calculation

[1251] 1. Emotion Engine: Takes user input and infers the emotional state by parsing emotions from the user's text using natural language processing models.

[1252] 2. AI answer generation: Generate appropriate answers based on the estimated emotions. For example, if the user is confused, generate an answer such as "I'm sorry for the inconvenience. How can I help you?"

[1253] 3. Contact with the person in charge: If the user expresses discomfort or confusion, the server will contact the person in charge. Email will be used as the means of contact. The email will include the user's desired date and time of visit and the details of the service.

[1254] Examples

[1255] If a user enters "I don't know how to use the product," the emotion engine infers "confusion," and the AI ​​responds, "We're sorry for the inconvenience. How can we help you?" The user then enters the desired date and time of visit and the details of the service, and a representative is contacted.

[1256] Examples of prompt statements

[1257] If a user enters "I don't know how to use the product," the emotion engine infers "confusion," and the AI ​​responds, "We're sorry for the inconvenience. How can we help you?" The user then enters the desired date and time of visit and the details of the service, and a representative is contacted.

[1258] In this way, appropriate responses can be made according to the user's emotional state, improving the user experience and quickly resolving issues.

[1259] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1260] Step 1:

[1261] A user makes a query in the metaverse. The user uses a smartphone or computer to access a virtual store in the metaverse and enters a question in text format. The entered text is sent to the server.

[1262] Step 2:

[1263] The server receives the user's input text and passes it to the emotion engine. The emotion engine uses a natural language processing model to infer the user's emotional state. Specifically, it analyzes the text data and classifies emotions such as happiness, excitement, discomfort, confusion, etc. The output of the emotion engine is the inferred emotional state.

[1264] Step 3:

[1265] The server receives the output of the emotion engine and passes it to the generative AI model. The generative AI model generates an appropriate answer based on the estimated emotional state. For example, if the user is confused, it generates an answer such as "I'm sorry for the inconvenience. How can I help you?" The generated answer is returned to the server.

[1266] Step 4:

[1267] The server provides the generated answer to the user. The user can check the answer from the AI ​​on the screen of their smartphone or computer. This allows the user to get the appropriate answer to their question.

[1268] Step 5:

[1269] If the user expresses discomfort or confusion, the server contacts the person in charge at the brick-and-mortar store. When the user inputs the desired date and time of visit and the details of the service, the information is sent to the server. The server contacts the person in charge via email. The email includes the user's desired date and time of visit and the details of the service.

[1270] Step 6:

[1271] The server summarizes the interactions in the metaverse and shares them with the store. The summary includes the user's questions and the AI's answers. This allows store staff to quickly understand the user's situation and take appropriate action.

[1272] In this way, appropriate responses can be made according to the user's emotional state, improving the user experience and quickly resolving problems.

[1273] Example 3

[1274] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1275] In the past, data on user-AI conversations in the metaverse was often not utilized as useful information for store staff. In addition, it was difficult to provide services that took into account the emotional state of the user. This made it difficult for store staff to accurately grasp the needs and emotions of users and provide appropriate services.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1277] In this invention, the server includes a means for collecting dialogue data in the metaverse, a means for storing the collected dialogue data in a cloud-based database, a means for analyzing the stored dialogue data and generating summary information, a means for analyzing the emotional state of the user using an emotion engine, a means for storing the generated summary information and the emotional state in the cloud-based database, and a means for a staff member of a real store to access the cloud-based database and view the summary information. This enables the staff member of the real store to understand the content of the dialogue with the user and the emotional state in the metaverse and provide more appropriate and personalized services.

[1278] The "metaverse" is a three-dimensional virtual space constructed using virtual reality technology, in which users can interact with other users and AI through avatars.

[1279] "Interaction data" refers to data that includes information such as questions, answers, and timestamps exchanged between users and AI within the metaverse.

[1280] A "cloud-based database" is a database service accessible via the Internet and is a platform for storing, managing, and analyzing data.

[1281] "Summary information" is a concise summary of important information extracted from the dialogue data, including the user's question, the AI's response, and key dialogue points.

[1282] An "emotion engine" is software or algorithms that analyze user interaction data and identify the user's emotional state (e.g., excitement, frustration, joy, etc.).

[1283] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze text data and generate summaries and answers.

[1284] "Brick and mortar store staff" refers to staff who deal with customers and provide services in a physical store.

[1285] "Access" means the act of connecting to a cloud-based database and viewing, retrieving, or manipulating stored data.

[1286] The present invention is a system that collects dialogue data in the metaverse, analyzes it, generates summary information, and provides it to staff in a real store. A specific embodiment of this system will be described below.

[1287] The server obtains dialogue data from the metaverse platform (e.g., a three-dimensional virtual space using virtual reality technology) through an API. When a user asks an AI a question in the metaverse, the question and the AI's answer are sent to the server in real time. For example, when a user asks, "What are the features of the new smartphone?", the question and the AI's answer are collected as dialogue data.

[1288] The server stores the collected interaction data in a cloud-based database (e.g., a database service accessible via the Internet). The details of each interaction are recorded in the database. For example, the following records are added:

[1289] {

[1290] "timestamp": "2023-10-01T12:34:56Z",

[1291] "user_question": "What are the features of the new smartphone?",

[1292] "ai_response": "New smartphones have high-resolution cameras and long-lasting batteries."

[1293] }

[1294] The server then analyzes the stored dialogue data and generates summary information. A generative AI model (e.g., an artificial intelligence model using natural language processing technology) is used to generate a summary of the dialogue data. This summary includes the user's question, the AI's answer, and important dialogue points. The generated summary looks like this:

[1295] Summary: A user asked about features of a new smartphone, and the AI ​​responded with information about high-resolution cameras and long battery life.

[1296] Further, the server uses an emotion engine (e.g., software or algorithm for analyzing a user's emotional state) to analyze the user's emotional state. The analysis result includes the emotion (excitement, frustration, joy, etc.) that the user exhibited during the interaction. The analysis result is as follows:

[1297] Emotional state: Excited

[1298] The server stores the generated summary information and emotional state in a cloud-based database. The stored records are as follows:

[1299] {

[1300] "timestamp": "2023-10-01T12:34:56Z",

[1301] "summary": "A user asked about the features of a new smartphone, and the AI ​​answered about high-resolution cameras and long battery life.",

[1302] "emotion": "excitement"

[1303] }

[1304] Using the device, a brick-and-mortar salesperson can access a cloud-based database and view summary information, which allows the salesperson to understand the customer's interest in a new smartphone and provide a detailed explanation and suggest related accessories.

[1305] An example of a prompt sentence to be input to the generative AI model is as follows.

[1306] Please provide a summary of user-AI interaction data within the metaverse. Please include the following information:

[1307] User Questions

[1308] AI Answers

[1309] The user's emotional state

[1310] This system enables the staff of the real store to understand the content of the conversation with the user and the emotional state in the metaverse, and to provide more appropriate and personalized services. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1311] Step 1:

[1312] The server obtains dialogue data from the metaverse platform through API. When a user asks an AI a question in the metaverse, the question and the AI's answer are sent to the server in real time. The input is the user's question and the AI's answer, and the output is the dialogue data. In concrete terms, the server sends an API request and receives the dialogue data.

[1313] Step 2:

[1314] The server stores the collected dialogue data in a cloud-based database. The input is the dialogue data obtained in step 1, and the output is the records stored in the database. Specifically, the server executes an INSERT query on the database to store the dialogue data.

[1315] Step 3:

[1316] The server analyzes the stored dialogue data and generates summary information. The input is the dialogue data stored in the database, and the output is the generated summary information. In concrete terms, the server inputs the dialogue data into the generative AI model and generates a summary. The generative AI model uses natural language processing technology to generate a summary of the dialogue data.

[1317] Step 4:

[1318] The server uses the emotion engine to analyze the user's emotional state. The input is the dialogue data, and the output is the analyzed emotional state. In specific operations, the server inputs the dialogue data to the emotion engine and analyzes the emotional state. The emotion engine identifies the user's emotion (excitement, frustration, joy, etc.).

[1319] Step 5:

[1320] The server stores the generated summary information and emotional states in a cloud-based database. The input is the summary information and the emotional states, and the output is the records stored in the database. Specifically, the server executes an INSERT query on the database to store the summary information and the emotional states.

[1321] Step 6:

[1322] Using the terminal, the store associate accesses the cloud-based database and views the summary information. The input is the summary information and emotional state stored in the database, and the output is the information viewed by the associate. In concrete terms, the associate uses the terminal to send a query to the database and obtain the summary information and the emotional state.

[1323] (Application example 3)

[1324] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1325] There is a demand for a system that allows store staff to understand the conversations between users and AI in the metaverse, and use the information to improve services and predict product popularity. However, current systems have difficulty summarizing interactions in the metaverse in real time and sharing them, including emotional states. There is also a lack of ways to provide information in a form that store staff can easily access. This makes it difficult to quickly understand user needs and take appropriate action.

[1326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a store visit reservation after an initial user response, a means for summarizing the exchange in the metaverse and making it possible to share it with the store, a means for storing the summary information in a cloud-based database and making it accessible to the real store staff, a means for recording the user's emotional state as part of the summary and sharing it with the store, and a means for providing an application that is installed on a smartphone or tablet and allows the real store staff member to check the summary information. This makes it possible to summarize the content of the dialogue with the user in the metaverse in real time and share it including the emotional state, making it possible for the real store staff member to quickly grasp the user's needs and take appropriate action.

[1327] "CS data" is data related to customer service, including information such as customer inquiry content, response history, and satisfaction ratings.

[1328] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[1329] The "metaverse" is a virtual reality space, a digital environment in which users can interact and perform activities with other users and AI through avatars.

[1330] A "real store" is a store that exists in a physical location and is a place where customers can actually visit to purchase products or receive services.

[1331] A "cloud-based database" is a database that is accessible via the Internet, where data is stored and managed on a cloud server.

[1332] "Summary information" is a concise summary of the content of the conversation between the user and the AI ​​within the metaverse, including the user's questions, the AI's answers, and the AI's emotional state.

[1333] An "emotional state" refers to the emotion a user exhibits during a conversation, such as excitement, frustration, joy, sadness, etc.

[1334] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[1335] A "tablet" is a portable computing device equipped with a touch screen, and is capable of using applications in the same way as a smartphone.

[1336] An "application" is a software program that provides specific functions or services and is installed and used on a smartphone or tablet.

[1337] A system for implementing the present invention is configured as follows.

[1338] First, the server installs an AI inside the Metaverse store that can learn CS data and other information and explain the company's services. When a user interacts with the AI ​​in the Metaverse, the content of the interaction is recorded in real time. The recorded content of the interaction is stored as summary information in a cloud-based database. This summary information includes the user's question, the AI's response, and the user's emotional state.

[1339] A store associate can then access this summary information using a dedicated application installed on a smartphone or tablet. The application retrieves the summary information from the cloud-based database and displays it to the associate, allowing the associate to understand the content of the interaction with the user in the metaverse and take appropriate action.

[1340] For example, if a user asks "Do you have this item in stock?" in the metaverse, the AI ​​will respond with "Yes, we do." The conversation is recorded along with the user's emotional state of excitement. This information is summarized and stored in a cloud-based database. Store personnel can access this information using a smartphone or tablet and provide more information about the product the user is interested in.

[1341] An example of a prompt sentence would be something like this:

[1342] Please summarize the interactions between the user and the AI ​​in the metaverse and output them in the following format:

[1343] User ID

[1344] question

[1345] answer

[1346] Emotional state

[1347] example:

[1348] User ID: user_001

[1349] Q: Is this item in stock?

[1350] Answer: Yes, we have it in stock.

[1351] Emotional state: Excited

[1352] In this way, conversations in the metaverse can be summarized in real time and shared, including emotional states, allowing store personnel to quickly understand the needs of users and provide appropriate responses.

[1353] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1354] Step 1:

[1355] The server will install an AI inside the Metaverse store that can learn CS data and explain the company's services. When a user starts a conversation with the AI ​​in the Metaverse, the server will record the conversation in real time. The input is the user's question and the AI's answer, and the output is the recorded conversation data.

[1356] Step 2:

[1357] The server stores the recorded dialogue data as summary information in a cloud-based database. The summary information includes the user's questions, the AI's answers, and the user's emotional state. The input is the recorded dialogue data, and the output is the summary information stored in the cloud-based database.

[1358] Step 3:

[1359] A staff member at a brick-and-mortar store retrieves summary information from the cloud-based database using a dedicated application installed on a smartphone or tablet. The input is an access request to the cloud-based database, and the output is the retrieved summary information.

[1360] Step 4:

[1361] The terminal displays the acquired summary information to the staff member. The displayed information includes the user's question, the AI's answer, the emotional state, etc. The input is the acquired summary information, and the output is the summary information displayed on the terminal screen.

[1362] Step 5:

[1363] Based on the displayed summary information, the salesperson can grasp the user's needs and respond appropriately. For example, if the user is excited, the salesperson can provide detailed information about the product. The input is the displayed summary information, and the output is the salesperson's response.

[1364] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1365] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1366] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1367] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1368] [Third embodiment]

[1369] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1370] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1372] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1374] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1376] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1379] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1380] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1381] "Example 1"

[1382] As one embodiment of the present invention, an AI that can learn CS data and explain the company's services is installed in a Metaverse store. This AI can understand questions from users using natural language processing technology, for example, and generate appropriate answers. The AI ​​also records and summarizes user questions and other dialogue data.

[1383] "Example 2"

[1384] After initial user interaction, the person in charge at the real store is contacted or a reservation is made for the visit. This contact is made, for example, by email, telephone, or through a dedicated reservation system. When making a reservation, the user is given the opportunity to specify the date and time of visit and the content of the service, taking into consideration the user's convenience. "Example 3"

[1385] The interactions in the metaverse are summarized and shared for the store. This can be done, for example, by storing the summary information in a cloud-based database and making it accessible to store personnel. The summary information includes the user's questions, the AI's answers, and other important dialogue data. This allows store personnel to understand the content of the interactions with the user in the metaverse and provide better service.

[1386] The process flow of each embodiment will be described below.

[1387] "Example 1"

[1388] Step 1: The AI ​​learns from the CS data, etc. This learning is performed using machine learning techniques such as supervised learning, unsupervised learning, and reinforcement learning.

[1389] Step 2: The AI ​​is installed in the Metaverse store and accepts questions from users. Questions are entered in text or voice format, and the AI ​​analyzes them using natural language processing technology.

[1390] Step 3: The AI ​​generates an appropriate answer and serves it to the user. The answer is generated based on the training data and contains the content that best fits the user's question.

[1391] Step 4: The AI ​​records the user's questions and other dialogue data and summarizes them. The summary is generated, for example, using keyword extraction and text summarization techniques.

[1392] "Example 2"

[1393] Step 1: After the initial user response is completed, the AI ​​contacts the person in charge at the brick-and-mortar store. For example, the contact can be made by accessing the brick-and-mortar store's reservation system via an API and sending the user's information and summarized interaction data.

[1394] Step 2: A store representative will receive the call and confirm the user's reservation. A notification of the reservation confirmation will be sent to the user via email or SMS, for example.

[1395] "Example 3"

[1396] Step 1: The AI ​​summarizes interactions in the metaverse and stores them in a cloud-based database, for example by accessing the database through an API and sending the summary data.

[1397] Step 2: A store associate accesses the database and reviews the summary data, for example through a dedicated dashboard that gives the associate an overview of the user’s questions, the AI’s responses, and other important interaction data.

[1398] Example 1

[1399] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1400] Conventional user response systems in virtual space have difficulty in providing prompt and appropriate answers to user questions, and the recording and summarization of dialogue data is insufficient. As a result, there are problems with poor user experience and inability to smoothly link with physical stores. Furthermore, the quality of answers to user questions is not consistent, which can lead to low user satisfaction.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1402] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing interactions in the virtual space so that they can be shared with the store, a means for receiving questions from users and analyzing the questions using natural language processing technology, a means for inputting a prompt sentence into a generative AI model and generating an appropriate answer, a means for providing the generated answer to the user, and a means for recording and summarizing dialogue data. This makes it possible to provide prompt and appropriate answers to user questions and effectively record and summarize dialogue data.

[1403] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1404] "AI that can explain a company's services" is artificial intelligence that automatically provides explanations in response to questions from users about the services a company offers.

[1405] A "virtual space" is a three-dimensional virtual environment constructed on the Internet, in which users can interact with other users and systems through avatars.

[1406] A "physical store" is a store that actually exists and is a location that users can visit in person.

[1407] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[1408] A "generative AI model" is an artificial intelligence model that generates new information or answers based on input data.

[1409] A "prompt" is a sentence that is input into a generative AI model and serves as instructions for the model to generate an appropriate answer.

[1410] "Dialogue data" is a record of questions and answers exchanged between the user and the AI.

[1411] "Summarization" refers to extracting important parts of conversation data and summarizing them concisely.

[1412] This invention is a system that provides appropriate answers to questions from users in a virtual space and records and summarizes dialogue data. A specific embodiment of this system will be described below.

[1413] First, the server generates an AI program that uses natural language processing technology. This program has the ability to understand questions from users and generate appropriate answers. The hardware used is a high-performance server (e.g., a server equipped with an NVIDIA GPU). The software used is a natural language processing library (e.g., TensorFlow, PyTorch) and a database (e.g., MySQL).

[1414] The server places the generated AI program in a virtual space. When this AI receives a question from a user, it analyzes the question using natural language processing technology. Specifically, it performs the following data processing and calculations:

[1415] 1. Data preprocessing: Tokenize the user question and remove unnecessary characters and words.

[1416] 2. Question analysis: Analyze the intent of the question using tokenized data.

[1417] 3. Answer generation: Generate appropriate answers based on the intent of the question, using a generative AI model (e.g. GPT-3).

[1418] 4. Recording dialogue data: Record user questions and AI answers in a database.

[1419] 5. Data Summarization: Summarize the recorded interaction data for future reference.

[1420] As a concrete example, when a user asks an AI in a virtual space, "Please tell me the features of this product," the following process takes place.

[1421] 1. The user enters a question through an interface in the virtual space.

[1422] 2. The device sends the user's question to the server.

[1423] 3. The server receives the question and uses a natural language processing library to tokenize the question and remove unnecessary characters and words.

[1424] 4. The server uses the tokenized data to analyze the intent of the question and inputs the prompt sentence, "User: What are the features of this product? AI:" into the generative AI model.

[1425] 5. The generative AI model generates an answer such as, "The features of this product are as follows. First, ..."

[1426] 6. The server sends the generated response to the device.

[1427] 7. The device displays the answer to the user, who then confirms the answer.

[1428] 8. The server records the user's questions and the AI's answers in a database and summarizes the dialogue data.

[1429] An example of a prompt sentence would be something like this:

[1430] User: What are the features of this product?

[1431] AI: The features of this product are as follows. First,...

[1432] In this way, the server, terminal, and user can work together to provide appropriate answers to user questions and effectively record and summarize dialogue data.

[1433] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1434] Step 1:

[1435] The user types a question in the virtual space.

[1436] Specific operation: The user inputs a question through the interface in the virtual space. For example, the user inputs, "Please tell me the features of this product."

[1437] Input: The text of the user's question

[1438] Output: The question text is sent to the terminal.

[1439] Step 2:

[1440] The terminal transmits the user's question to the server.

[1441] Specific operation: The terminal sends the question entered by the user to the server, which transmits data via the Internet.

[1442] Input: Question text

[1443] Output: The question text is sent to the server.

[1444] Step 3:

[1445] The server receives the question and begins natural language processing.

[1446] Specific operation: The server receives the question sent from the device and analyzes the received question using a natural language processing library (e.g. TensorFlow, PyTorch).

[1447] Input: Question text

[1448] Output: Tokenized question data

[1449] Step 4:

[1450] The server analyzes the question and inputs a prompt into the generative AI model.

[1451] How it works: The server tokenizes the question and removes unnecessary characters and words. It then analyzes the intent of the question and inputs a prompt to the generative AI model (e.g. GPT-3). For example, it inputs the prompt in the format: "User: What are the features of this product? AI:".

[1452] Input: Tokenized question data

[1453] Output: Prompt statement

[1454] Step 5:

[1455] A generative AI model generates the appropriate answer.

[1456] Specific behavior: The generative AI model generates an appropriate answer based on the input prompt sentence, for example, "The features of this product are as follows. First, ..."

[1457] Input: prompt statement

[1458] Output: The generated answer text

[1459] Step 6:

[1460] The server sends the generated answer to the terminal.

[1461] Specific operation: The server sends the answer obtained from the generative AI model to the terminal, converting the answer data into an appropriate format before sending it.

[1462] Input: Generated answer text

[1463] Output: The answer text is sent to the terminal.

[1464] Step 7:

[1465] The terminal displays the answer to the user.

[1466] Specific operation: The terminal displays the answer received from the server to the user, who then checks the answer through an interface in the virtual space.

[1467] Input: Answer text

[1468] Output: The answer that is displayed to the user

[1469] Step 8:

[1470] The server records and summarizes the interaction data.

[1471] How it works: The server records the user's questions and the AI's answers in a database, then summarizes the recorded dialogue data for future reference.

[1472] Input: User's question text and AI's answer text

[1473] Output: Recorded conversation data and a summary

[1474] (Application example 1)

[1475] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1476] In conventional metaverse stores, when users had questions about products or services, it was difficult to get appropriate answers. In addition, there was a lack of a way to effectively record and summarize user questions and dialogue data and share them with staff at the physical store, so there was a need to improve the user experience. Furthermore, it was not possible to provide answers to user questions in real time, which could lead to a decrease in user satisfaction.

[1477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1478] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making them available for sharing with the store, a means for generating answers to user questions in real time and providing them to the user, and a means for recording dialogue data, summarizing it, and providing it to a store manager. This makes it possible for a user to receive an appropriate answer in real time when asking a question about a product or service in a metaverse store, and for dialogue data to be effectively recorded and summarized to be shared with a person in charge of a real store, improving the user experience.

[1479] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1480] "AI" is an abbreviation for artificial intelligence, a system that imitates human intellectual tasks using machine learning and natural language processing technologies.

[1481] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users and content through avatars.

[1482] "Real-time" refers to responsiveness in time to instantaneous response to user input.

[1483] "Dialogue data" is a record of the conversation between the user and the AI, and includes information such as questions and answers.

[1484] A "summary" is information that extracts the main points of dialogue data and summarizes them concisely.

[1485] A "store manager" is a person or organization in charge of operating a physical store or a metaverse store.

[1486] A "user" is a consumer or user who visits a metaverse store and asks questions about products and services.

[1487] A "real store" is a store that exists in a physical location where customers can visit in person to purchase goods or services.

[1488] A "store reservation" is the process by which a user reserves a date and time in advance to visit a physical store.

[1489] The following system configuration will be described as an embodiment of the present invention.

[1490] The server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user asks a question about a product or service in the Metaverse store, the AI ​​will use natural language processing technology to understand the user's question and generate an appropriate answer in real time. This answer will be provided to the user.

[1491] The server also records and summarizes the dialogue data, including the user's questions and the AI's responses, and provides it to the store manager, who can then more easily understand the user's needs and interests. In addition, after the initial user response, the system also includes a means to contact a brick-and-mortar store staff member or to schedule a visit.

[1492] This system is realized using the following hardware and software.

[1493] Hardware:

[1494] Smartphone or head mounted display

[1495] software:

[1496] Python

[1497] OpenAI API

[1498] When a user enters a question into the application, the server uses OpenAI's API to generate an appropriate answer to the user's question. The generated answer is displayed to the user. The interaction data is recorded, summarized, and provided to the store manager.

[1499] For example, when a user asks "Tell me about this product" in a virtual store, the AI ​​responds, "This product uses the latest technology and is very high performance." The conversation data is also recorded and summarized as "User: Tell me about this product. AI: This product uses the latest technology and is very high performance."

[1500] Example prompt:

[1501] User: Tell me about this product.

[1502] AI: This product uses the latest technology and is extremely powerful.

[1503] In this way, when users ask questions about products or services in the metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with the brick-and-mortar store staff, improving the user experience.

[1504] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1505] Step 1:

[1506] Users access the metaverse store using a terminal (smartphone or head-mounted display) and input questions about products and services.

[1507] Input: The text of the user's question

[1508] Output: The user's question text is sent to the server.

[1509] Step 2:

[1510] The server receives the user's question text and analyzes the intent of the question using natural language processing techniques.

[1511] Input: The text of the user's question

[1512] Output: Data analyzed to determine the intent of the question

[1513] Step 3:

[1514] The server uses the OpenAI API to generate appropriate answers based on the intent of the analyzed question.

[1515] Input: Data analyzed to determine the intent of the question

[1516] Output: Answer text generated from the OpenAI API

[1517] Step 4:

[1518] The server transmits the generated answer text to the user's terminal and displays it to the user.

[1519] Input: Answer text generated from the OpenAI API

[1520] Output: The answer text displayed to the user's terminal.

[1521] Step 5:

[1522] The server records the user's question text and the generated answer text as dialogue data.

[1523] Input: User question text, generated answer text

[1524] Output: Recorded interaction data

[1525] Step 6:

[1526] The server summarizes the recorded interaction data and provides it to the store manager.

[1527] Input: Recorded interaction data

[1528] Output: Summarized conversation data

[1529] Step 7:

[1530] After initial user response, the server provides a means to contact a person in charge at a real store or to make a reservation to visit the store.

[1531] Input: User question text, generated answer text, recorded dialogue data

[1532] Output: Contact a store representative or schedule a visit

[1533] In this way, when users ask questions about products or services in a metaverse store, they can get appropriate answers in real time, and the conversation data can be effectively recorded and summarized to share with store personnel, improving the user experience.

[1534] Example 2

[1535] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1536] In conventional systems, users had to take the time and effort to contact a real store representative or make a reservation to visit after receiving the initial response. In addition, there was a lack of efficient and accurate means for users to communicate information when specifying the date and time of their visit or the service content. Furthermore, when sharing interactions in the virtual space with the real store, the information was not summarized or organized sufficiently, which could lead to delayed responses from the store.

[1537] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1538] In this invention, the server includes a means for installing an AI in a virtual space that can learn CS data and the like and explain the company's services, a means for contacting a physical store staff member or making a store visit reservation after an initial user response, a means for summarizing the interactions in the virtual space and sharing them with the store, a means for providing an interface for the user to input the date and time of visit and the service content, a means for saving the input data in a database, a means for sending a notification according to a specified contact method based on the saved data, and a means for feeding back the notification sending result to the user. This allows the user to efficiently and accurately specify the date and time of visit and the service content, and quickly contact the staff member at the real store. In addition, by summarizing and sharing the interactions in the virtual space, the store side can respond quickly and accurately.

[1539] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1540] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[1541] A "virtual space" is a digital environment generated by computer simulation, which users can access via the Internet and in which they can carry out various activities.

[1542] A "physical store" is a real store that actually exists and is a place where users can visit in person to receive services.

[1543] An "interface" is the means by which a user interacts with a system, and includes user interface elements such as input forms and buttons.

[1544] A "database" is a system for efficiently storing, managing, and searching data, and is operated using a query language such as SQL.

[1545] A "notification" is information sent from the system to a user or person in charge, and may be provided in the form of email, telephone call, message, etc.

[1546] "Feedback" refers to the response or result that a system provides to a user, informing the user of the outcome of an action they took.

[1547] A "summary" is a concise summary of the content of an exchange in a virtual space, in which important information is extracted and organized.

[1548] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1549] This invention is a system that allows users to contact a real store staff member or make a reservation to visit the store after receiving the initial response. The system includes AI, a user interface, a database, and a notification function installed in a virtual space.

[1550] Hardware and Software Used

[1551] The server is developed using the Python programming language and implemented as a web application using the Flask framework, with MySQL as the database and an SMTP server and Twilio API for notifications.

[1552] Specific operation of the system

[1553] The user accesses the system's web page using a web browser. The web page displays a form for inputting the date and time of visit and the desired service. For example, the user inputs that they would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[1554] The server receives the data entered by the user in the form and processes the request using the Flask framework, which stores the data in a MySQL database by generating SQL queries to insert the data into the database.

[1555] The server then uses the information stored in the database to send notifications according to the contact method the user specified: for example, if it's email, it uses an SMTP server to send an email, if it's phone, it uses the Twilio API to make an automated voice call, if you use a dedicated reservation system, it sends the reservation information through the reservation system's API, etc.

[1556] Finally, the server checks whether the notification was sent successfully and gives feedback to the user about the result. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[1557] Examples of concrete examples and prompts

[1558] As a concrete example, consider a user making a reservation at a hair salon. The user enters into a web form that they would like a "cut and color" service on "October 15, 2023 at 14:00." The server receives this information and sends a notification to the salon representative via email.

[1559] Example prompt:

[1560] If a user fills in a web form to make an appointment at a hair salon and requests a "cut and color" service on "October 15, 2023 at 2:00 PM," the server should receive this information and send a notification to the salon representative via email.

[1561] In this way, a system is realized that allows the user to specify the date and time of visit and the content of the service, taking into consideration the user's convenience.

[1562] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1563] Step 1:

[1564] The user enters the date and time of visit and the details of the service into a web form.

[1565] A user accesses the system's web page using a web browser and enters information into a form to input the date and time of visit and the desired service. For example, the user enters that he / she would like a "cut and color" on "October 15, 2023 at 14:00" and clicks the submit button.

[1566] Input: Visit date and time, service details

[1567] Output: Data entered by the user

[1568] Step 2:

[1569] The server receives the input data and stores it in a database.

[1570] The server receives the data entered by the user in the form, processes the request using the Flask framework, and saves the input data to a MySQL database by generating SQL queries to insert into the database.

[1571] Input: Data entered by the user

[1572] Output: Data stored in the database

[1573] Step 3:

[1574] The server will send notifications based on the stored data and in accordance with the specified contact method.

[1575] The server sends notifications according to the contact method specified by the user based on the information stored in the database. For example, if it is email, it will send an email using an SMTP server. If it is phone, it will make an automated voice call using the Twilio API. If a dedicated reservation system is used, the reservation information will be sent via the reservation system's API.

[1576] Input: Data stored in the database, contact method

[1577] Output: Notification sent result

[1578] Step 4:

[1579] The server provides feedback on the notification sending result to the user.

[1580] The server checks whether the notification was sent successfully and gives the result as feedback to the user. For example, if the notification was sent successfully, it displays a message saying "Reservation completed." If the notification was not sent successfully, it displays a message saying "Reservation failed. Please try again."

[1581] Input: Notification sending result

[1582] Output: A feedback message to the user.

[1583] (Application example 2)

[1584] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1585] Conventional user support systems in the metaverse had problems with insufficient coordination with real stores after the initial response, resulting in reduced user convenience. In addition, there was a lack of a way to select the appropriate person in charge based on the content of the user's inquiry and to contact them quickly, which led to a risk of user satisfaction decreasing. Furthermore, there was no reservation system that allowed users to specify the date and time of visit or the content of the service, making it difficult to flexibly respond to user needs.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1587] In this invention, the server includes a means for installing an AI inside a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a real store staff member or making a reservation for a visit after an initial user response, a means for summarizing exchanges in the metaverse and making it possible to share them with the store, a means for the AI ​​to automatically select an appropriate staff member based on the content of the user's inquiry, a means for contacting the staff member via email, telephone, or a dedicated reservation system, and a means including a reservation system that allows the user to specify the date and time of visit and the content of the service. This makes it possible to quickly select and contact an appropriate staff member based on the content of the user's inquiry, thereby improving user convenience and satisfaction.

[1588] "CS data" is data related to customer service, including information such as the content of customer inquiries and response history.

[1589] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and natural language processing to automatically process information and provide users with appropriate answers and services.

[1590] The "metaverse" is a digital environment built in a virtual space, a platform where users can interact with other users and systems through avatars.

[1591] A "real store" is a store that exists in a physical location and is a place where users can actually visit and receive products and services.

[1592] "Contact Person" refers to a store staff member or employee designated to handle user inquiries and reservations.

[1593] A "reservation system" is a system that enables a user to make a reservation by specifying the date and time of visit and the content of the service, and is used via email, telephone, or a dedicated online platform.

[1594] A "summary" is a concise summary of the content of interactions within the metaverse, including user questions and AI responses.

[1595] "Email" means a digital message sent over the Internet and is a means of exchanging information between users and personnel.

[1596] "Telephone" is a means of exchanging information in real time through voice communication, and is intended for direct communication between users and personnel.

[1597] "Automatic selection" is a process in which AI automatically selects the most suitable representative based on the content of the user's inquiry.

[1598] A system for implementing the present invention is configured as follows.

[1599] First, the server will install AI inside the Metaverse store that can learn CS data and explain the company's services. When a user makes an initial inquiry within the Metaverse, the AI ​​will automatically generate a response based on the content of the inquiry and provide it to the user. This allows the user to quickly obtain the information they need.

[1600] Next, if the user's inquiry meets certain conditions, the server contacts the store's staff or schedules a visit. Specifically, the AI ​​automatically selects an appropriate staff member based on the user's inquiry and contacts the staff member via email, phone, or a dedicated reservation system. The contact includes the user's inquiry, the desired date and time of the visit, and the service details.

[1601] Additionally, interactions within the metaverse will be summarized and made available to stores, including the user's question and the AI's answer, allowing store staff to quickly understand the user's situation.

[1602] To realize this system, the server uses the following hardware and software: The hardware requires a server connected to the Internet and a user's smartphone. The software includes a Python program, an SMTP server (e.g. smtp.example.com), and a reservation system.

[1603] As a specific example, if a user wants to "consult about purchasing a new smartphone" and wants to make a reservation to visit the store at 14:00 on October 15, 2023, the following prompt sentence will be generated.

[1604] Username: Yamada Taro

[1605] Inquiry: Consultation about purchasing a new smartphone

[1606] Desired date and time: 2023-10-15 14:00

[1607] contact address:

[1608] Email: taro.yamada@example.com

[1609] Phone: 090-1234-5678

[1610] Based on this prompt, the server contacts the person in charge and confirms the user's reservation, allowing the user to smoothly receive services at a physical store.

[1611] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1612] Step 1:

[1613] A user makes an initial inquiry within the metaverse.

[1614] Input: User's inquiry (e.g. "Consultation on purchasing a new smartphone")

[1615] Processing: The user's inquiry is received and the AI ​​analyzes it.

[1616] Output: Parsed query content

[1617] Step 2:

[1618] AI automatically generates answers based on the user's inquiries and provides them to the user.

[1619] Input: Parsed query content

[1620] Processing: The AI ​​uses the generative AI model to generate the appropriate answer.

[1621] Output: The generated answer (e.g. "When it comes to buying a new smartphone, we recommend the following model...")

[1622] Step 3:

[1623] If the user's inquiry meets certain conditions, the server contacts a person in charge at the physical store or makes a reservation to visit the store.

[1624] Input: Analyzed inquiry details, user's desired visit date and time, service details

[1625] Processing: AI automatically selects the appropriate person to contact based on the inquiry and generates a prompt.

[1626] Output: Generated prompt text (Example: "User name: Yamada Taro, Inquiry: Consultation on purchasing a new smartphone, Desired date and time: 2023-10-15 14:00, Contact: Email: taro.yamada@example.com, Phone: 090-1234-5678")

[1627] Step 4:

[1628] The server will contact the person via email, phone, or a dedicated reservation system.

[1629] Input: Generated prompt text

[1630] Processing: The server uses an SMTP server to send an email to a person in charge, or uses a phone API to automatically make a phone call, or send a prompt to a reservation system.

[1631] Output: Contact the person in charge (e.g., the reservation information will be sent to the person in charge's mailbox)

[1632] Step 5:

[1633] Interactions within the metaverse are summarized and shared for stores.

[1634] Input: User question, AI answer

[1635] Processing: The server summarizes interactions within the metaverse and generates summary information.

[1636] Output: Generated summary information (e.g. "User question: Looking for advice on buying a new smartphone. AI answer: We recommend the following model...")

[1637] Step 6:

[1638] Share summary information for stores.

[1639] Input: Generated summary information

[1640] Processing: The server sends summary information to the person in charge.

[1641] Output: The agent receives summary information to quickly understand the user's situation.

[1642] Example 3

[1643] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1644] There is a need to efficiently collect, summarize, and share conversation data between users and AI in the metaverse with store staff in real stores in order to provide better services based on users' needs and questions. However, in current systems, the collection, summarization, and sharing of conversation data is often done manually, which makes it inefficient and lacks real-time capabilities.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1646] In this invention, the server includes a means for collecting dialogue data in the metaverse, a means for storing the collected dialogue data in a cloud-based database, a means for summarizing the dialogue data using a generative AI model, a means for storing the summary data in the cloud-based database, and a means for allowing a person in a physical store to view the summary data. This makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time.

[1647] The "metaverse" is a three-dimensional virtual space constructed using virtual reality technology, in which users can interact with other users and AI through avatars.

[1648] "Interaction Data" refers to data that includes questions, answers, and other significant content of conversations between users and AI within the metaverse.

[1649] A "cloud-based database" is a database service accessible via the Internet in which data is stored, managed, and accessed in a cloud environment.

[1650] A "generative AI model" is a model that uses artificial intelligence technology to generate, summarize, and convert text data, such as a model that uses natural language processing technology.

[1651] "Summary data" is data that concisely summarizes important information extracted from dialogue data by a generative AI model.

[1652] "Brick and mortar store staff" refers to staff who deal with customers and provide services in a physical store.

[1653] "Viewing" is the act of displaying and checking stored data or information.

[1654] This invention is a system that efficiently collects dialogue data between users and AI in the metaverse, summarizes the data, and shares it with staff in real stores. A specific embodiment of this system will be described below.

[1655] The server collects real-time user-AI interaction data within the metaverse, including user questions, AI responses, and other important dialogue content. The server stores the collected interaction data in a cloud-based database (e.g., Amazon RDS or Google Cloud SQL).

[1656] The server then uses a generative AI model (e.g., a model using natural language processing techniques) to summarize the collected dialogue data. This summary includes the key points of the user's question and the AI's answer. The summarized information is again stored in a cloud-based database.

[1657] Store staff can use devices to access the cloud-based database and view summary information, allowing them to understand the content of interactions between users and the store staff in the metaverse and provide better service.

[1658] As a concrete example, consider a case where a user asks in the metaverse, "Is a new product in stock?" and an AI responds, "Yes, it's in stock." This dialogue data is collected by the server and a generative AI model is used to generate a summary: "The user checked the availability of a new product, and the AI ​​responded that it is in stock." This summary information is stored in a cloud-based database and can be accessed by staff at the brick-and-mortar store.

[1659] Example prompt:

[1660] Summarize the user's question and the AI's answer.

[1661] User Question: Are new items in stock?

[1662] AI answers: Yes, we have it in stock.

[1663] By inputting this prompt sentence into the generative AI model, summarized information is obtained. This system makes it possible to efficiently collect dialogue data in the metaverse, summarize it, and share it in real time. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1664] Step 1:

[1665] The server collects the dialogue data between the user and the AI ​​in the metaverse in real time. Specifically, when the user asks "Is the new product in stock?" in the metaverse and the AI ​​responds "Yes, it is in stock," the server captures this dialogue data. The input is the user's question and the AI's answer, and the output is the collected dialogue data.

[1666] Step 2:

[1667] The server stores the collected dialogue data in a cloud-based database. Specifically, the server inserts the dialogue data into an appropriate table in the database. The input is the collected dialogue data, and the output is the dialogue data stored in the database.

[1668] Step 3:

[1669] The server inputs the saved dialogue data into the generative AI model and generates a summary. Specifically, it inputs the following prompt sentence into the generative AI model:

[1670] Summarize the user's question and the AI's answer.

[1671] User Question: Are new items in stock?

[1672] AI answers: Yes, we have it in stock.

[1673] The input is the prompt and the dialogue data, and the output is the generated summary data.

[1674] Step 4:

[1675] The server again stores the generated summary data in the cloud-based database. Specifically, the server inserts the summary data into an appropriate table in the database. The input is the generated summary data, and the output is the summary data stored in the database.

[1676] Step 5:

[1677] Using a terminal, a store associate accesses the cloud-based database and views the summary data. Specifically, the associate queries the database, retrieves the summary data, and displays it. The input is a request to access the database, and the output is the displayed summary data.

[1678] This series of processes makes it possible to efficiently collect, summarize, and share conversation data within the metaverse in real time.

[1679] (Application example 3)

[1680] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1681] In order for store personnel to understand the content of conversations between users and retailers in the metaverse and provide better services, a means of efficiently summarizing and sharing the content of the conversations is necessary. However, with current systems, it is difficult to summarize the conversation data in the metaverse in real time, store it in a cloud-based database, and have store personnel in retailers access it using their smartphones. This poses the challenge of quickly understanding the needs and interests of users and providing personalized services.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1683] In this invention, the server includes a means for installing an AI in a metaverse store that can learn CS data and the like and explain the company's services, a means for contacting a person in charge of a real store or making a reservation for a visit after an initial user response, a means for summarizing interactions in the metaverse and making it possible to share with the store, a means for storing the summary information in a cloud-based database and making it possible for a person in charge of a real store to access using a smartphone, and a means for including user questions, AI responses, and other important dialogue data in the summary information. This makes it possible to efficiently summarize the content of the dialogue in the metaverse and for a person in charge of a real store to quickly access it, thereby making it possible to grasp the needs and interests of the user and provide personalized services.

[1684] "CS Data" is data related to customer service, including customer inquiries, feedback, service history, etc.

[1685] "AI" is an abbreviation for artificial intelligence, a system that mimics human intelligence using technologies such as machine learning and natural language processing.

[1686] The "metaverse" is a virtual reality space, a digital environment where users can interact with other users through avatars and carry out various activities.

[1687] A "physical store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1688] A "cloud-based database" is a database that is accessible via the internet and in which data is stored and managed by a cloud service provider.

[1689] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[1690] "Summary information" is information that briefly summarizes detailed data or dialogue content and includes important points and main points.

[1691] A "user question" is an inquiry or question that a user makes to an AI within the metaverse.

[1692] An "AI answer" is a response or explanation that an AI generates in response to a user's question.

[1693] "Important dialogue data" refers to information or data that is considered to be particularly important in the dialogue between the user and AI, and contains content that is useful in providing services.

[1694] In order to implement the present invention, the following system configuration and processing procedure are used.

[1695] First, the server installs an AI inside the metaverse store that can learn CS data and explain the company's services. When a user asks the AI ​​a question in the metaverse, the AI ​​automatically generates an answer to the question and provides it to the user. This allows the user to obtain information about the company's services in the metaverse.

[1696] Next, after the initial user response, the server contacts the person in charge at the brick-and-mortar store and, if necessary, makes a reservation to visit the store. This completes the preparations for the user to receive services at the brick-and-mortar store.

[1697] In addition, the server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The summary information includes the user's questions, the AI's answers, and other important dialogue data. Store staff can access the cloud-based database using their smartphones to check the summary information. This allows store staff to quickly understand the user's needs and interests and provide personalized services.

[1698] As a concrete example, a user may ask in the metaverse, "What are the features of my new smartphone?" and the AI ​​may respond, "The new smartphone has a high-resolution camera and a long-lasting battery." This conversation is summarized and stored in a cloud-based database. A store associate can access this summary information using their smartphone and understand that the user is interested in the new smartphone.

[1699] Examples of prompts to input to a generative AI model include:

[1700] A user asks in the metaverse, "What are the features of my new smartphone?" and the AI ​​responds, "My new smartphone has a high-resolution camera and a long-lasting battery." Summarize this conversation:

[1701] In this way, the server, terminal, and user work together to efficiently summarize the content of conversations in the metaverse, which can then be quickly accessed by store personnel in the physical store, enabling them to understand the needs and interests of the user and provide personalized services.

[1702] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1703] Step 1:

[1704] The server will install AI inside Metaverse stores that can learn CS data and other data and explain the company's services. When a user asks the AI ​​a question within the Metaverse, the AI ​​will automatically generate an answer to that question and provide it to the user. The input is the user's question, and the output is the AI's answer. In concrete terms, it uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1705] Step 2:

[1706] After initially responding to the user, the server contacts the person in charge at the physical store and, if necessary, makes a reservation to visit the store. The input is the user's question and the AI's response, and the output is contact information for the person in charge. In concrete terms, the server sends a notification to the person in charge based on the content of the user's question and the AI's response, and processes the reservation to visit the store.

[1707] Step 3:

[1708] The server summarizes the interactions in the metaverse and stores the summary information in a cloud-based database. The input is the user's question and the AI's answer, and the output is the summary information. Specifically, the server uses a generative AI model to summarize the dialogue and stores it in a cloud-based database.

[1709] Step 4:

[1710] A staff member at a brick-and-mortar store uses a smartphone to access the cloud-based database and check the summary information. The input is the summary information obtained from the cloud-based database, and the output is the staff member's understanding. The specific operation is to access the database through a smartphone application and display the summary information.

[1711] Step 5:

[1712] Based on the summary information, the staff member understands the user's needs and interests and provides personalized services. The input is the summary information, and the output is the service to be provided. Specifically, the system refers to the summary information and suggests products and services that are suitable for the user.

[1713] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1714] "Example 1"

[1715] In one embodiment of the present invention, the AI ​​includes an emotion engine that infers the user's emotional state from the user's questions and other dialogue data. Specifically, the emotion engine uses natural language processing and voice analysis techniques to infer emotions from the user's text and voice inputs. It also uses image analysis and motion analysis techniques to analyze the user's facial expressions and body movements.

[1716] "Example 2"

[1717] The user's emotional state estimated by the emotion engine is reflected in the AI's response generation. Specifically, if the user shows joy or excitement, the AI ​​generates a response that emphasizes that emotion. Conversely, if the user shows discomfort or confusion, the AI ​​generates a response that soothes that emotion. This allows the AI ​​to respond appropriately to the user's emotions.

[1718] "Example 3"

[1719] The emotion engine also influences summaries of interactions in the metaverse. Specifically, the user's emotional state is recorded as part of the summary, which is then shared with the store. For example, if a user expresses strong excitement about a product, that information is shared with the store to help predict the product's popularity and sales. If a user expresses dissatisfaction with a service, that information is shared with the store to help improve the service.

[1720] The process flow of each embodiment will be described below.

[1721] "Example 1"

[1722] Step 1: The AI ​​receives a question and other interaction data from the user.

[1723] Step 2: The emotion engine estimates the user's emotional state from the received data. Specifically, it estimates emotions from text and voice input using natural language processing and voice analysis technologies. It also estimates emotions from the user's facial expressions and body movements using image analysis and motion analysis technologies.

[1724] "Example 2"

[1725] Step 1: The emotion engine estimates the user's emotional state.

[1726] Step 2: The AI ​​generates answers based on the estimated emotional state. Specifically, if the user is happy or excited, the AI ​​generates answers that emphasize those emotions. Conversely, if the user is uncomfortable or confused, the AI ​​generates answers that soften those emotions.

[1727] "Example 3"

[1728] Step 1: The emotion engine estimates the user's emotional state and reflects the result in a summary of the interaction in the metaverse.

[1729] Step 2: The summary is shared with the store. Specifically, information about the user's excitement about the product and dissatisfaction with the service is shared with the store.

[1730] Step 3: Stores use the shared information to predict product popularity and sales, and use it as a reference for improving services.

[1731] Example 1

[1732] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1733] Conventional user response systems in virtual space have difficulty generating appropriate answers to user questions and are unable to accurately grasp the user's emotional state. As a result, there were problems with poor user experience, poor guidance to physical stores, and ...

Claims

1. A method to install AI in a virtual space that can learn CS data and explain the company's services, After initial user interaction, a means to contact a physical store representative or schedule a visit; and A means for summarizing interactions within the virtual space and enabling them to be shared for the physical store; means for receiving a question from the user and analyzing the question using natural language processing techniques; A means for inputting a prompt sentence related to the question analyzed by the analyzing means into a generating AI model and generating an appropriate answer; A means for providing the answer generated by the generating means to the user through the AI; A means for recording and summarizing interaction data between the user and the AI; A system including:

2. The system of claim 1 , wherein the generative AI model further comprises means for automatically generating an answer to a user's question and providing the answer to the user.

3. The system of claim 1 , further comprising means for sharing information including user questions and AI answers in the summary of the interaction in the virtual space with the physical store.

Images