system

Abstract

Provide a system. A method for receiving a user request includes: a means for initializing the virtual reality environment; A means to initialize the synthetic voice AI, A way to initialize the conversation AI, means for generating an avatar of a specific character based on a request from the user; means for acquiring the user data and reflecting it in the virtual reality environment; means for managing real-time interactions with said user; a means for processing billing; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] At conventional shops, when users go through procedures such as signing up for a new contract, upgrading their model, or learning how to use their device, it takes time, which results in a poor user experience. There is also a lack of entertaining services, such as those that involve interacting with specific characters or celebrities. Furthermore, there is also the issue of users having to visit the shop to receive face-to-face support, which is time-consuming and labor-intensive. [Means for solving the problem]

[0005] The present invention relates to a system that provides guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. The present invention is solved by a system including the following means.

[0006] The system includes a means for receiving a user request, a means for initializing a virtual reality environment, a means for initializing a synthetic voice AI, a means for initializing a conversation AI, a means for generating an avatar of a specific character based on the user request, a means for acquiring the user data and reflecting it in the virtual reality environment, a means for managing real-time dialogue with the user, and a means for performing billing processing.It is also preferable that the system includes a means for selecting a service in the virtual reality environment, a means for connecting to a server and acquiring data, and a means for interacting through the user interface.

[0007] This system allows users to efficiently complete procedures through interactions with specific characters and celebrities in a virtual reality environment, improving the user experience. It also allows users to complete procedures remotely without visiting a store, saving time and effort.

[0008] "User Request" means a request sent by a user to use a service provided within a virtual reality environment.

[0009] "Virtual reality environment" means a simulated environment generated using computer technology with which a user can interact in real time.

[0010] "Synthetic voice AI" is an artificial intelligence technology that converts text data into something that sounds like human speech.

[0011] "Conversational AI" is an artificial intelligence technology that enables natural language dialogue with users.

[0012] A "particular character avatar" is a virtual alter ego, such as a particular celebrity or historical figure, that is displayed within a virtual reality environment and with which a user can interact.

[0013] "User Data" is a collective term for information about a user, including individual settings and history within a virtual reality environment.

[0014] "Real-time interaction" refers to an interaction between a user and a system that occurs immediately, without any time delay.

[0015] "Billing" means the process of applying charges to a User's account when using a Paid Service.

[0016] "Interaction" refers to the interaction between a user and objects or characters in a virtual reality environment.

[0017] "User interface" is a general term for the screen display and input means that allow users to interact with a system.

[0018] "Service type" refers to the type of individual service selected by the user, such as a new contract, a model change, or terminal operation guidance. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using synthetic voice AI and conversational AI.

[0041] Server side

[0042] Request received

[0043] The server receives a request sent by the user. The request includes the user's identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0044] Initializing the virtual reality environment

[0045] The server sets up the virtual reality environment based on the user's requests, including creating a VR space tailored to the user's preferences and creating an avatar for a specified character (e.g., a celebrity or historical figure).

[0046] Initializing the synthetic voice AI and conversation AI

[0047] During the initialization phase, instances of synthetic speech AI and conversation AI are generated, which prepares functions such as converting text to speech and realizing natural dialogue with the user.

[0048] Acquiring and updating user data

[0049] The server retrieves user data from the database and reflects it in the virtual reality environment. User data includes user profile information and past usage history.

[0050] Managing real-time interactions

[0051] The server receives input from the user in real time and generates a response using conversational AI. The response is converted into voice through synthetic speech AI and provided to the user. For example, if the user says, "I'd like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[0052] Billing

[0053] When a paid service such as a terminal operation guide is used, the server executes billing processing to the user's account, allowing the user to easily pay after receiving the service in the virtual reality environment.

[0054] Terminal side

[0055] Launching a VR app

[0056] The user launches the app on their VR device, which initiates a connection with the server and loads the virtual reality environment.

[0057] Connecting to the Server

[0058] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[0059] Service Selection

[0060] Within the virtual reality environment, the user selects the service they wish to use (new contract, model change, terminal operation guide, etc.). Based on the selected service, the server instructs them to take the appropriate action.

[0061] User Interaction

[0062] The terminal manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. The terminal sends input from the user to the server, and receives and displays responses from the server in real time.

[0063] User side

[0064] Wearing a VR device

[0065] Users put on the VR device and prepare to use the service. By equipping the VR headset and handheld controllers, they can enter the virtual reality environment.

[0066] Service Selection

[0067] Users use an interface in the virtual reality environment to select services such as new contracts, model changes, and operation guides, allowing them to efficiently complete the procedures they desire.

[0068] Start a dialogue

[0069] The user interacts with a specific character avatar and receives services. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[0070] Payment process

[0071] When using paid services, users complete payment procedures within the virtual reality environment, including entering credit card information and QR code payment.

[0072] VR Shop closed

[0073] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0074] Through these steps, the user can receive efficient and entertaining services within the virtual reality environment. The system of the present invention improves user convenience and solves the problems of the prior art.

[0075] The processing flow will be explained below.

[0076] Server side

[0077] Processing steps from receiving a request to generating a response

[0078] Step 1:

[0079] The server receives a user request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0080] Step 2:

[0081] The server initializes the virtual reality environment based on the request, creating a VR space tailored to the user's preferences.

[0082] Step 3:

[0083] The server initializes the synthetic voice AI and the conversation AI. The synthetic voice AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input.

[0084] Step 4:

[0085] The server retrieves user data from a database, including user profile information and past usage history.

[0086] Step 5:

[0087] The server reflects the user data in the virtual reality environment, allowing the avatar of the character set to provide appropriate services to the user.

[0088] Step 6:

[0089] The server receives input from the user in real time and generates a response using conversational AI, which is then converted into speech using synthetic speech AI and provided to the user.

[0090] Step 7:

[0091] When a paid service such as terminal operation guidance is used, the server executes billing processing, which charges the user's account.

[0092] Terminal side

[0093] Processing steps from app launch to user interface management

[0094] Step 1:

[0095] The user launches the VR app on their device, which initiates a connection with the server.

[0096] Step 2:

[0097] The device connects to the server, sends the user's request, receives the VR environment data returned from the server, and displays the virtual reality environment.

[0098] Step 3:

[0099] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[0100] Step 4:

[0101] The terminal manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[0102] Step 5:

[0103] Users interact with characters in a virtual reality environment through gestures and voice input, and the device transmits this input to a server, which receives and displays responses.

[0104] User side

[0105] Processing steps from device installation to payment procedure

[0106] Step 1:

[0107] The user puts on the VR device and prepares to use the service, which allows them to enter the virtual reality environment.

[0108] Step 2:

[0109] The user selects the desired service using an interface provided within the virtual reality environment.

[0110] Step 3:

[0111] Based on the service selected, the user initiates a dialogue with a specific character avatar, which allows the user to proceed with the required procedure.

[0112] Step 4:

[0113] When using paid services, users can complete payment procedures within the virtual reality environment, using methods such as credit card information or QR code payment.

[0114] Step 5:

[0115] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0116] Example 1

[0117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] Currently, there is no system in existence that allows users to smoothly and efficiently receive guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. In particular, major challenges remain in real-time dialogue, high operability of the user interface, natural communication using synthesized speech and conversational agents, and simple payment procedures.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0120] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice generating means, means for initializing a dialogue agent, means for generating a virtual avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing payments, and means for generating and displaying data for the virtual reality environment. This allows the user to smoothly receive guidance on contracts, model changes, and terminal operations within the VR environment and complete operations and procedures through real-time dialogue.

[0121] The "means for receiving a user request" is a function for receiving an information request sent from a user terminal.

[0122] "Means for initializing a virtual reality environment" is a function that configures and creates a VR space based on the user's requests.

[0123] "Synthetic voice generation means" is a technology for converting text data into voice data.

[0124] The "means for initializing a conversational agent" is a function for preparing an artificial intelligence to have a natural conversation with a user.

[0125] The "means for generating a virtual avatar of a specific character" is a function for creating a 3D model of a specified character in response to a user request.

[0126] "Means of acquiring user data and reflecting it in a virtual reality environment" refers to technology that acquires user information from a database and applies it to a VR environment.

[0127] The "means for managing real-time dialogue with the user" is a function that responds immediately to input from the user and provides an appropriate response.

[0128] The "payment processing means" is a system that charges the user for the service usage and completes the settlement.

[0129] The "means for generating and displaying data for a virtual reality environment" is a function for generating data necessary for a virtual reality environment and displaying it on a user terminal.

[0130] The "means for selecting a service" is an interface that allows a user to select the required service within the virtual reality environment.

[0131] "Means for connecting to a server and obtaining data" refers to a technique by which a user terminal accesses a server and obtains the necessary information.

[0132] A "means of interaction through a user interface" is an interface that a user uses to operate and interact.

[0133] The "means for providing guidance in accordance with the type of service to be used" is a function for providing appropriate instructions and guidance to the user in accordance with the selected service.

[0134] "Means for wearing a virtual reality device and starting a service" refers to the steps a user takes to start using a service using a VR device.

[0135] The "means for exiting the virtual reality shop after service usage has ended" is a function that allows the user to exit the virtual reality environment after finishing using the service.

[0136] "Means for interacting through gestures and voice input" refers to technologies that allow users to communicate with the system using hand movements and voice.

[0137] The present invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using a synthetic speech generation means and a dialogue agent.

[0138] Server side

[0139] Request received

[0140] The server receives requests sent from users using the HTTPS protocol. The request contains the user's identification information (user ID) and the type of service desired (new contract, model change, device operation guide, etc.). The received request is parsed in JSON format and the appropriate processing is carried out.

[0141] Initializing the virtual reality environment

[0142] The server uses Unity to set up a virtual reality environment based on the user's request. This includes generating a VR space tailored to the user's preferences and placing a specific character avatar created using Blender. The virtual reality environment data is sent to the user's device and displayed.

[0143] Initialization of synthetic speech generator and dialogue agent

[0144] The server initializes the synthetic speech generation means using Google® Cloud's Text-to-Speech API and the dialogue agent using Dialogflow, which prepares the functions of converting text to speech and realizing natural dialogue with the user.

[0145] Acquiring and updating user data

[0146] The server retrieves user data from a MySQL® database and reflects it in the virtual reality environment. The user data includes user profile information and past usage history. This allows the system to provide appropriate services for each individual user.

[0147] Managing real-time interactions

[0148] The server receives voice input from the user and generates an appropriate response using Dialogflow. The generated response is converted into audio using Google Cloud's Text-to-Speech API and provided to the user. For example, if a user types "I would like to sign a new contract," the conversational agent will suggest an appropriate contract plan and explain its details in audio.

[0149] Payment Processing

[0150] The server processes payments using the Stripe API. For example, if a paid service such as a terminal operation guide is used, the server will charge the user's account. This allows the user to easily pay after receiving the service in the virtual reality environment.

[0151] Terminal side

[0152] Launching a VR app

[0153] A user launches an app on a VR device (e.g., Oculus Quest), which initiates a connection with the server and loads the virtual reality environment. VR apps run using the Oculus VR SDK.

[0154] Connecting to the Server

[0155] The device connects to the server and sends the user's request to the server. The device receives the VR environment data returned from the server and displays it. The device connects using Wi-Fi or 5G.

[0156] Service Selection

[0157] Within the virtual reality environment, users select the service they want to use, such as signing up for a new contract, upgrading their device, or receiving device operation instructions. Service selection is performed using an interactive menu within the VR app.

[0158] User Interaction

[0159] The device manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the device's operation procedures.

[0160] User side

[0161] Wearing a VR device

[0162] Users put on a VR device (such as Oculus Quest) and prepare to use the service, which allows them to enter an immersive virtual reality environment.

[0163] Service Selection

[0164] Users can select services such as new contracts, model changes, and operation guidance within the virtual reality environment, allowing them to efficiently complete the procedures they desire.

[0165] Start a dialogue

[0166] Users interact with a specific character avatar to receive services, including obtaining necessary information through natural conversation. For example, a user can begin the process by entering "I would like to sign a new contract."

[0167] Payment process

[0168] When using paid services, users complete payment procedures within the virtual reality environment, including entering credit card information or using a QR code. Payments are processed by the server via the Stripe API.

[0169] VR Shop closed

[0170] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0171] Examples of concrete examples and prompts

[0172] For example, if a user wants to sign up for a new smartphone model, the following steps are taken.

[0173] 1. The user puts on the Oculus Quest and launches the VR app.

[0174] 2. Select "New Contract" within the virtual reality environment.

[0175] 3. A specific character will appear and ask questions about the contract in a dialogue format.

[0176] 4. The server generates the appropriate contract plan through Dialogflow and converts it into audio using Google Cloud's Text-to-Speech API.

[0177] 5. The user follows the character's instructions to complete the contract procedure.

[0178] Prompt Sentence Examples

[0179] "Enter a scenario where a user signs up for a new smartphone:

[0180] 1. First, the user puts on the VR device and launches the VR app.

[0181] 2. Select a service menu in the virtual reality environment

[0182] 3. Check the details of the new contract through dialogue with the character

[0183] 4. Payment Procedure

[0184] 5. Close the VR shop.

[0185] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0186] Step 1: Receiving a request

[0187] The server receives a request from the user via the HTTPS protocol. This request includes the user ID and the details of the desired service (e.g., new contract, model change, terminal operation guide). The user's request data is input, and this data is analyzed to extract the necessary information. The analyzed user ID and service content data are obtained as output. Specifically, the server uses Apache (registered trademark) HTTP Server to analyze the received JSON data.

[0188] Step 2: Initializing the Virtual Reality Environment

[0189] The server uses Unity to set up a virtual reality environment based on the user's request. The input is the user ID and service content data, and based on this, it configures the VR space and generates a specific character avatar. The output is the generated VR space data. Specifically, it calls the Unity API and places a VR space tailored to the user's preferences and a character avatar generated using Blender.

[0190] Step 3: Initialize the synthetic speech generator and the dialogue agent

[0191] The server initializes the synthetic speech generation means using Google Cloud's Text-to-Speech API and the dialogue agent using Dialogflow. User data and service content data are used as input, and each API is configured based on this. Synthetic speech and dialogue agent instances are obtained as output. Specifically, the server accesses the API key and settings via the Google Cloud Console and generates the required instances.

[0192] Step 4: Obtaining and updating user data

[0193] The server retrieves user data from a MySQL database and reflects it in the virtual reality environment. It executes a database query using the user ID as input, and obtains user profile information and usage history as output. Specifically, it executes a MySQL query to retrieve the necessary data and applies it to the VR environment settings.

[0194] Step 5: Managing real-time interactions

[0195] The server receives voice input from the user and generates an appropriate response using Dialogflow. It then converts the voice input into audio using Google Cloud's Text-to-Speech API and provides it to the user. The input is the user's voice input data, which is analyzed and converted into text, and a response is generated using a conversational agent. The output is audio response data. Specifically, the voice data is temporarily saved to a WAV file, sent to Dialogflow, and the returned text is passed to the synthetic speech API to generate audio.

[0196] Step 6: Payment Processing

[0197] After a paid service is used, the server processes the payment through the Stripe API. The input is the user's payment information and service information, and billing is performed based on this. The output is payment confirmation data. Specifically, the server calls the Stripe API, processes the user's payment information, and completes the payment.

[0198] Step 7: Launch the VR app

[0199] A user launches an app on a VR device such as Oculus Quest. The input is a command to launch the VR device, and the output is to start accessing the VR environment. Specifically, the app launches using the Oculus VR SDK and establishes a network connection.

[0200] Step 8: Connect to the Server

[0201] The device connects to the server and sends the user's request to the server. The input is the user's request data, and the output is the virtual reality environment data. Specifically, the device sends an HTTP request to the server and receives the VR environment data returned from the server.

[0202] Step 9: Select a service

[0203] The user selects the service they want to use in the virtual reality environment. The input is the user's service selection, and the output is instructions based on the selected service returned from the server. Specifically, the user selects the service using an interactive menu within the VR app.

[0204] Step 10: User Interaction

[0205] The device manages interactions with the user. Inputs include user gestures and voice inputs, and output displays response data from the server. Specifically, it uses WebSocket to capture user gestures and voice commands and send them to the server.

[0206] Step 11: Put on the VR device

[0207] The user puts on the VR device and prepares to use the service. The input is a command to put on the VR device, and the output is to start accessing the virtual reality environment. Specifically, the user puts on the Oculus Quest and prepares the handheld controller.

[0208] Step 12: Select a service

[0209] Within the virtual reality environment, the user selects services such as new contracts, model changes, and operation guidance. The input is the user's service selection, and the output is instructions based on the selected service provided by the server. Specific operations involve using a menu provided within the VR environment to select a service that suits the purpose.

[0210] Step 13: Start a conversation

[0211] A user initiates a dialogue with a specific character avatar. The input is the user's voice input, and the dialogue agent generates a response accordingly. The output is a voice response from the dialogue agent. Specifically, the user enters a voice command, which is processed by the dialogue agent to generate a response.

[0212] Step 14: Payment Process

[0213] When using a paid service, the user completes the payment process within the virtual reality environment. The input is the user's payment information, and the output is payment confirmation data. Specifically, the user enters their credit card information within the virtual reality environment and completes the payment via the Stripe API.

[0214] Step 15: Finish the VR Shop

[0215] After completing the service, the user removes the VR device and exits the virtual reality shop. The input is the user's exit command, and the output is the ability to return to the real world. The specific actions are to close the VR app and remove the Oculus Quest.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] Services provided within conventional virtual reality environments have not fully satisfied user convenience. In particular, it has been difficult to provide information on new contracts, model upgrades, and device operation methods quickly and effectively within a virtual reality environment. Furthermore, user interaction has been limited, and there has been a lack of means for real-time dialogue, resulting in a poor user experience. Furthermore, there has been a demand for a method that allows more users to easily access services by using virtual reality devices connected to smartphones.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice artificial intelligence, means for initializing a conversational artificial intelligence, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for realizing a dialogue system for providing guidance on new contracts, model changes, and device operation within the virtual reality environment, and means for the user to interact using a virtual reality headset and controller connected to a smartphone. This allows the user to easily and quickly receive more convenient and effective guidance on new contracts, model changes, and device operation within the virtual reality environment.

[0221] 1. "Means for receiving user requests" refers to the methods and functions by which the system receives operations and requests from users.

[0222] 2. "Means for initializing a virtual reality environment" means methods and functionality for configuring and building a user-accessible virtual reality environment.

[0223] 3. "Means for initializing a synthetic speech artificial intelligence" means methods and functions for configuring and activating an artificial intelligence that converts text data into speech data.

[0224] 4. "Means for initializing conversational AI" refers to methods and functions for configuring and launching an AI that communicates in natural language.

[0225] 5. "Means for generating an avatar of a specific character" means methods and functions for displaying a specific character within a virtual reality environment based on a user's request.

[0226] 6. "Means for acquiring user data and reflecting it in the virtual reality environment" refers to the methods and functions for acquiring user information from the database and applying it within the virtual reality environment.

[0227] 7. "Means for managing real-time interactions with users" refers to methods and functions that enable users and systems to interact with each other in real time.

[0228] 8. "Means for billing processing" refers to the methods and functions for managing and processing payment procedures when a user uses a paid service.

[0229] 9. "Means for realizing an interactive system for providing guidance on new contracts, model changes, and device operation within a virtual reality environment" refers to methods and functions that provide users with interactive guidance on new contracts, model changes, and device operation within a virtual reality environment.

[0230] 10. "Means for user interaction using a virtual reality headset and controllers connected to a smartphone" means methods and functionality that allow a user to navigate and interact within a virtual reality environment using a virtual reality headset and controllers connected to a smartphone.

[0231] The present invention is a system that allows users to receive service information such as new contracts, model changes, and terminal operation methods within a virtual reality environment. This system is composed of three parties: a server, a terminal, and a user, and uses detailed hardware and software to fulfill each role.

[0232] Server-side implementation

[0233] The server includes the following means:

[0234] 1. Means of receiving a user request:

[0235] The server receives a request sent by the user, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0236] 2. Means of initializing the virtual reality environment:

[0237] Based on the user's request, the server initializes the virtual reality environment using a VR toolkit (e.g., Unity or Unreal Engine), including creating a VR space tailored to the user's preferences and generating an avatar for a specific character.

[0238] 3. Methods for initializing the synthetic voice AI:

[0239] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly) that converts text data into voice data.

[0240] 4. Methods for initializing conversational AI:

[0241] The server initializes a conversational AI (e.g., Dialogflow or Microsoft® Bot Framework) to enable natural dialogue with the user.

[0242] 5. How user data is acquired and reflected in the virtual reality environment:

[0243] The server retrieves the user's profile information and past usage history from the database and reflects this in the virtual reality environment.

[0244] 6. Means of managing real-time user interactions:

[0245] The server receives voice input from the user in real time and uses conversational AI to generate a response, which is then converted into speech by synthetic speech AI and provided to the user.

[0246] 7. Means of processing claims:

[0247] When a paid service such as terminal operation guidance is used, the server processes the billing to the user's account, allowing the user to easily make payment after receiving the service.

[0248] Terminal side embodiment

[0249] The terminal includes the following means:

[0250] 1. Launch the VR app:

[0251] The user puts on a VR headset and launches a VR application on their smartphone, which initiates a connection with the server.

[0252] 2. Connect to the server:

[0253] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[0254] 3. Service Selection:

[0255] The user uses an interface within the virtual reality environment to select services such as new contracts, model changes, and terminal operation guidance.

[0256] 4. User Interaction:

[0257] The terminal manages interactions with the user, who interacts with specific characters through gestures and voice input to receive necessary procedures and instructions.

[0258] User-Side Embodiment

[0259] The user includes the following means:

[0260] 1. Wearing the VR device:

[0261] The user prepares to enter the virtual reality environment by putting on a VR headset and preparing the handheld controllers.

[0262] 2. Service Selection:

[0263] The user operates the interface provided within the virtual reality environment to select services such as new contracts, model changes, and operation guidance.

[0264] 3. Start a conversation:

[0265] The user can start a conversation with a specific character and receive the necessary information or procedures. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[0266] 4. Payment Procedure:

[0267] When using a paid service, users can complete payment procedures within the virtual reality environment by entering credit card information or using a QR code.

[0268] 5. VR Shop Closure:

[0269] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0270] Specific examples

[0271] For example, if a user wishes to sign a new contract, they can say "I would like to sign a new contract" by voice within the VR environment, and a character avatar will respond by saying, "Here are our new contract plans. Which plan would you like?" In this way, each time the user asks a question, the conversational AI generates an appropriate response and replies audibly through the synthetic voice AI.

[0272] Prompt Sentence Examples

[0273] Describe a dialogue system with an AI character that realizes a scenario in which a user wishes to make a new contract in a VR environment. Explain in detail the response and behavior of the character avatar when the user says, "I want to make a new contract."

[0274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0275] Step 1:

[0276] The server receives the user's request, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.). Based on this input, the server analyzes the request and begins preparing an appropriate virtual reality environment.

[0277] Step 2:

[0278] The server initializes the virtual reality environment. This is the process of generating a VR space tailored to the user's preferences using a VR toolkit (e.g., Unity or Unreal Engine) based on the user's request. An avatar for the specified character is also generated and configured. As a result, a virtual reality environment tailored to the user is constructed.

[0279] Step 3:

[0280] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly). At this stage, an instance of the synthetic voice AI is created and prepared to convert text data into speech data. This allows speech output for subsequent user interactions.

[0281] Step 4:

[0282] The server initializes the conversational AI (e.g., Dialogflow or Microsoft Bot Framework), creates an instance of the conversational AI, and configures it to enable natural dialogue with the user. This configuration includes preparing a model to generate appropriate responses to user input.

[0283] Step 5:

[0284] The server retrieves user data from the database and reflects it in the virtual reality environment. The user data includes profile information and usage history, and personalizes the VR space based on that information. This information is used to improve the user experience.

[0285] Step 6:

[0286] The server manages real-time dialogue with the user. In this step, the user's voice input is converted and analyzed by conversational AI. Based on the analysis results, a response is generated through synthetic voice AI and provided to the user. This process is carried out in real time, ensuring smooth interaction between the user and the system.

[0287] Step 7:

[0288] The device launches the VR app and establishes a connection with the server, allowing the user to access the virtual reality environment and receive data sent from the server, enabling the user to view the virtual reality environment and operate the interface.

[0289] Step 8:

[0290] The device provides an interface for the user to select services. The user uses a VR headset and handheld controller to select services such as new contracts, model changes, and device operation guidance. This input causes the server to prepare the next response.

[0291] Step 9:

[0292] The user operates the interface and begins a dialogue with a specific character. For example, if the user voice-inputs, "I'd like to sign a new contract," the character will respond with, "Here are our new contract plans. Which plan would you like?" This dialogue progresses in real time, with responses generated by conversational AI.

[0293] Step 10:

[0294] When a user uses a paid service, the server handles the billing process. Specifically, the charge is sent to the user's account via credit card information entry or QR code payment, allowing the user to complete payment easily.

[0295] Step 11:

[0296] After completing their use of the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world and re-prepare the device for their next use.

[0297] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0298] This invention relates to a system that provides users with guidance on new contracts, model changes, and terminal operation methods in a virtual reality environment, while also recognizing the user's emotions and generating adaptive responses based on them. This system receives user requests, initializes the virtual reality environment, and interacts with the user using a synthetic voice AI, a conversational AI, and an emotion engine.

[0299] Server side

[0300] Processing from receiving a request to generating a response

[0301] The server receives the user's request. This request includes the user ID and the type of service desired (new contract, model change, device operation guide, etc.). The server then initializes the virtual reality environment based on the request. This includes generating a VR space tailored to the user's preferences and creating an avatar for the specified character.

[0302] Initialize instances of the synthetic speech AI and conversation AI. The synthetic speech AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input. The emotion engine is also initialized. The emotion engine analyzes the user's facial expressions and voice and recognizes emotions.

[0303] User data is retrieved from the database and reflected in the virtual reality environment, allowing the character avatar to provide appropriate services to the user.

[0304] The server receives input from the user in real time and generates a response using conversational AI. The generated response is converted into voice using synthetic voice AI and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[0305] The emotion engine analyzes the user's facial expressions and voice to detect emotions. Based on the detected emotions, the conversational AI generates a response that is adapted to the user. For example, if the user expresses surprise, the conversational AI will generate a response such as, "Did I surprise you?"

[0306] When a paid service such as a terminal operation guide is used, a billing process is executed, and the fee is charged to the user's account.

[0307] Terminal side

[0308] Processing from app launch to user interface management

[0309] The user launches the VR app on their device, which initiates a connection with the server and loads the virtual reality environment. The device then connects to the server and sends the user's request. The server returns the VR environment data, which is then displayed.

[0310] The user selects the services they wish to use within the virtual reality environment (new contract, model change, device operation guide, etc.). The selection is sent to the server. The device manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[0311] The user interacts with the character in the virtual reality environment through gestures and voice input. The device sends this input to the server, receives responses, and displays them. The emotion engine analyzes the user's facial expressions and voice and sends emotional data to the server.

[0312] User side

[0313] Processing from device installation to payment procedures

[0314] The user puts on the VR device and prepares to use the service. This allows the user to enter the virtual reality environment. The user selects the desired service using the interface provided within the virtual reality environment.

[0315] Based on the service selected, the user can start a conversation with a specific character avatar, which will then guide them through the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the operation procedure.

[0316] When using paid services, users make payments within the virtual reality environment. Payment methods include credit card entry and QR code payment. After completing the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0317] Through these steps, users can enjoy efficient and entertaining services within the virtual reality environment. Furthermore, the emotion engine recognizes the user's emotions and provides adaptive responses, further enhancing the user experience. The system of the present invention improves user convenience and solves the problems of the past.

[0318] The processing flow will be explained below.

[0319] Server side

[0320] Processing steps from receiving a request to generating a response

[0321] Step 1:

[0322] The server receives the user's request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0323] Step 2:

[0324] The server initializes the virtual reality environment based on the request, generates a VR space according to the user's preferences, and prepares to generate an avatar for the specified character.

[0325] Step 3:

[0326] The server initializes instances of a synthetic speech AI and a conversational AI. The synthetic speech AI has the function of converting text to speech, and the conversational AI generates appropriate responses to user input.

[0327] Step 4:

[0328] The server retrieves user data from a database, including profile information and past usage history.

[0329] Step 5:

[0330] The server reflects the acquired user data in the virtual reality environment and applies the user's individual settings.

[0331] Step 6:

[0332] The server initiates a real-time dialogue with the user using conversational AI, taking the user's input and generating the optimal response.

[0333] Step 7:

[0334] The server uses synthetic voice AI to convert the generated response into speech, which is then provided to the user.

[0335] Step 8:

[0336] The server analyzes the user's facial expressions and voice through an emotion engine to recognize their emotions.

[0337] Step 9:

[0338] The server adjusts the conversational AI's responses based on the emotions it recognizes, for example, providing a follow-up message such as "Don't worry" if the user expresses anxiety.

[0339] Step 10:

[0340] The server performs billing when terminal operation guidance or other paid services are used, and charges are applied to the user's account.

[0341] Terminal side

[0342] Processing steps from app launch to user interface management

[0343] Step 1:

[0344] The user launches the VR app on their device, which initiates a connection with the server.

[0345] Step 2:

[0346] The device connects to the server and sends the user's request, which includes information such as the user ID and the type of service.

[0347] Step 3:

[0348] The device receives the VR environment data returned from the server and displays the virtual reality environment.

[0349] Step 4:

[0350] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[0351] Step 5:

[0352] The terminal manages the user interface within the virtual reality environment, displaying menus and handling interactions.

[0353] Step 6:

[0354] The emotion engine analyzes the user's facial expressions and voice, and sends the emotional data to the server.

[0355] Step 7:

[0356] The device receives responses from the server and displays them both audibly and visually, and adaptive responses are provided based on data from the emotion engine.

[0357] User side

[0358] Processing steps from device installation to payment procedure

[0359] Step 1:

[0360] The user puts on the VR device and prepares to use the service, allowing them to enter the virtual reality environment.

[0361] Step 2:

[0362] The user uses an interface provided within the virtual reality environment to select the service they want, for example, to sign up for a new contract or upgrade their device.

[0363] Step 3:

[0364] Based on the service selected, the user begins a conversation with a specific character avatar, who guides the user through specific procedures.

[0365] Step 4:

[0366] The user follows the character's instructions as they interact with the character, while the emotion engine recognizes the user's emotions and adjusts responses accordingly.

[0367] Step 5:

[0368] When using paid services, users can complete payment procedures within the virtual reality environment, such as entering credit card information or using a QR code.

[0369] Step 6:

[0370] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0371] Example 2

[0372] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0373] Conventional virtual reality environments have difficulty understanding user emotions and providing adaptive responses. They also lack real-time dialogue management and services tailored to user needs. This can limit the user experience and reduce user satisfaction. Another issue is the complicated billing and payment procedures for paid services, which places a burden on users.

[0374] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0375] In this invention, the server includes a means for analyzing a user's facial expressions and voice to recognize emotions, a means for generating an adaptive response based on the emotion recognition, and a means for performing billing processing. This makes it possible to understand a user's emotions in real time and provide an adaptive response based on the emotion. It also realizes appropriate billing processing and payment procedures, improving the user experience and convenience.

[0376] The "means for receiving a user request" refers to a function or module for acquiring a request from a user.

[0377] A "means for initializing a virtual reality environment" is a system or program for setting up and starting up a virtual environment.

[0378] The "means for initializing the synthetic speech generation means" is a function for starting and setting up a speech generation system that has the function of converting text data into speech data.

[0379] The "means for initializing a conversation generation model" is a function for starting and setting up a conversation model that generates an appropriate response to a user's input.

[0380] A "means for generating an avatar of a specific character" is a system or program that generates the appearance and behavior of a virtual character based on the user's requests and settings.

[0381] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring user information from a database, etc., and applying it to the virtual reality space.

[0382] "Means for managing real-time user interactions" refers to functions and modules that monitor and control real-time communication between users and the system.

[0383] The "means for analyzing the user's facial expressions and voice and recognizing emotions" is an analysis system for detecting the user's emotional state from facial expression and voice data.

[0384] The "means for generating adaptive responses based on emotion recognition" is a function or module that generates an appropriate response that matches the recognized emotional state of the user.

[0385] The "means for billing" refers to a system or program that calculates fees for the use of paid services and bills the user account.

[0386] "Means for interaction through a user interface" refers to functions or modules that provide an interface that users can visually operate and communicate through it.

[0387] "Means for wearing a virtual reality device and starting the service" refers to the procedures and modules for starting to use the system after wearing the VR device.

[0388] "Means for selecting services within a virtual reality environment" refers to an interface or system that allows users to select the desired service from the various services provided within the virtual space.

[0389] The "means for starting a dialogue with a character" refers to a function or module for starting a conversation with a generated virtual character and advancing the dialogue.

[0390] "Means for carrying out payment procedures when using a paid service" refers to a system or module for completing payment when a user selects a paid service.

[0391] "Means for exiting the virtual reality shop after using the service" refers to the procedures and functions that allow users to safely exit the virtual reality environment after they have finished using the service.

[0392] This invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and also has the function of recognizing the user's emotions and generating adaptive responses. An embodiment of this system will be described in detail below.

[0393] To receive a user request, the server acquires the user ID and desired service details via an HTTP request or WebSocket. This request includes the type of service, such as a new contract, a model change, or device operation instructions.

[0394] The server then initializes the virtual reality environment, which involves using the Unreal Engine to generate a VR space tailored to the user's preferences and an avatar for the specified character.

[0395] The server then initializes the synthetic speech generator (Google Cloud Text-to-Speech) and the conversation generation model (OpenAI® GPT-3®). It also initializes the emotion engine (Microsoft Azure® Emotion API) that analyzes the user's facial expressions and voice to recognize their emotions.

[0396] The server retrieves user data from the database and uses this information to reflect in the virtual reality environment, a process that allows the character avatar to provide appropriate services to the user.

[0397] For real-time dialogue management, the server receives user input and generates an appropriate response using a conversation generation model. The generated response is converted into speech using a synthetic speech generation means and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversation generation model will suggest an appropriate contract plan and explain its details in voice.

[0398] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Based on the recognized emotions, the conversation generation model generates adaptive responses. For example, if the user expresses surprise, the response generated will be, "Did I surprise you?"

[0399] If a paid service is used, the server executes a billing process and charges the fee to the user's account, completing the use of the service.

[0400] On the device side, the user launches a VR app and connects to the server using HTTP or WebSocket communication. The device retrieves the virtual reality environment data returned from the server and displays it using Unity. Within the virtual reality environment, the user selects the service they wish to use and sends that selection to the server. The device also manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[0401] Finally, the user puts on the VR device, enters the virtual reality environment, and selects a service using the provided interface. They then begin interacting with a specific character avatar within the virtual reality environment and carry out the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the steps. When using a paid service, the user completes the payment procedure within the virtual reality environment, and once the service is complete, the user removes the VR device and exits the virtual reality shop to return to the real world.

[0402] As a concrete example, the following prompt sentences can be used by the conversation generation model to smoothly engage the user:

[0403] User: I'd like to sign up. What plans are available?

[0404] System: There are several plans available for new subscribers: Basic Plan, Premium Plan, and Family Plan. Want to learn more about each plan?

[0405] In this way, the embodiments of the invention involve the use of specific hardware and software, and include detailed data processing and data calculation processing, making it possible to provide users with efficient and entertaining services.

[0406] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0407] The flow of this system's program processing

[0408] Server-side processing

[0409] Step 1: Receiving a user request

[0410] How it works: The server receives the user's ID and desired service via an HTTP request or WebSocket.

[0411] Input: User ID, desired service (new contract, model change, terminal operation guide, etc.)

[0412] Output: Received data (user ID, service details)

[0413] Step 2: Initializing the Virtual Reality Environment

[0414] How it works: The server generates a VR space using Unreal Engine based on the received user data and generates an avatar for a specific character.

[0415] Input: User data, service details

[0416] Output: Initialized VR space, character avatar

[0417] Step 3: Initialize the synthetic speech generator and the speech generation model

[0418] How it works: The server initializes Google Cloud Text-to-Speech and OpenAI GPT-3.

[0419] Input: None (at system startup)

[0420] Output: Initialized synthetic speech generator, speech generation model

[0421] Step 4: Obtaining and updating user data

[0422] Operation: The server retrieves user data from the database and reflects it in the virtual reality environment.

[0423] Input: User ID

[0424] Output: Updated VR environment

[0425] Step 5: Receiving user input and generating a response

[0426] How it works: The server receives voice or text input from the user, uses a speech generation model to generate an appropriate response, and converts the response into audio using Google Cloud Text-to-Speech.

[0427] Input: User input (voice or text)

[0428] Output: Response voice

[0429] Step 6: Emotion recognition and adaptive response generation

[0430] How it works: The server uses the Microsoft Azure Emotion API to analyze the user's facial expressions and voice to recognize emotions, and then uses a conversation generation model to generate adaptive responses.

[0431] Input: User's facial expression data, voice data

[0432] Output: Adaptive response

[0433] Step 7: Billing for paid services

[0434] How it works: The server calculates the fee when a paid service is used and bills the user's account.

[0435] Input: Service information used

[0436] Output: Billing information, updated user account

[0437] Terminal side processing

[0438] Step 1: Launch the VR app

[0439] How it works: The user launches the VR app on their device and connects to the server.

[0440] Input: VR app launch command

[0441] Output: Server connection started

[0442] Step 2: Acquiring and displaying data in a VR environment

[0443] How it works: The device retrieves VR environment data from the server and displays the virtual reality environment using Unity.

[0444] Input: VR data from the server

[0445] Output: The displayed VR environment

[0446] Step 3: Select service and submit

[0447] Operation: The user selects the service to use within the virtual reality environment, and the device sends the selection to the server.

[0448] Input: Service selection (user input)

[0449] Output: Selected service details (sent from server)

[0450] Step 4: Managing the User Interface

[0451] Operation: The terminal manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[0452] Input: User interaction

[0453] Output: Updated user interface

[0454] Step 5: Sending User Input and Displaying Responses

[0455] How it works: The user's voice or gesture input is sent to the server, and the response from the server is displayed.

[0456] Input: User input (voice and gestures)

[0457] Output: Response from the server (visual and audio output)

[0458] Step 6: Sending Emotion Data

[0459] How it works: The emotion engine analyzes the user's facial expressions and voice and sends the data to the server.

[0460] Input: User's facial expression data, voice data

[0461] Output: Emotion data (sent to server)

[0462] User processing

[0463] Step 1: Put on your VR device

[0464] Action: The user puts on the VR device and prepares to use the service.

[0465] Input: None (wearing operation)

[0466] Output: VR device is installed

[0467] Step 2: Select a service

[0468] Operation: The user selects the desired service using the interface provided within the virtual reality environment.

[0469] Input: Service selection by user

[0470] Output: Selected service content

[0471] Step 3: Interact with the character

[0472] Action: The user initiates a dialogue with a specific character avatar and proceeds with the required procedures.

[0473] Input: User instruction to start a dialogue

[0474] Output: Started conversation

[0475] Step 4: Complete the payment process

[0476] Operation: When a user uses a paid service, the payment process is carried out within the VR environment.

[0477] Input: Payment information (credit card information or QR code)

[0478] Output: Completed payment

[0479] Step 5: Completing the service and returning to the real world

[0480] Operation: After completing the service, the user removes the VR device and exits the virtual reality shop to return to the real world.

[0481] Input: None (removal operation)

[0482] Output: Return to the real world

[0483] (Application example 2)

[0484] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0485] The problem to be solved by this invention is to provide a system that improves the user experience in a virtual reality environment and generates responses that adapt to the user's emotions. Conventional systems lack the ability to analyze the user's facial expressions and voice in real time and provide adaptive responses based on the analysis, which can lead to reduced user satisfaction. Furthermore, the means for processing prompts using generative AI models and converting them into synthetic speech are insufficient, resulting in a lack of naturalness in the interaction.

[0486] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0487] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic speech generation device, means for initializing a conversational dialogue device, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for initializing an emotion recognition device, means for analyzing the user's facial expressions and voice data and generating a response based on the detected emotion, means for generating a response according to a prompt sentence using a generative AI model, and means for converting the prompt sentence into synthetic speech. This makes interaction with the user more natural and personal, thereby improving user satisfaction.

[0488] "Means for receiving user requests" refers to a function for receiving service requests and operation instructions sent by users and processing them within the system.

[0489] The "means for initializing a virtual reality environment" is a function for constructing and starting a virtual reality space based on a user's request.

[0490] The "means for initializing the synthetic speech generation device" is a function for preparing the synthetic speech technology that converts text data into natural speech and setting it into an operational state.

[0491] The "means for initializing the conversation dialogue device" is a function for constructing a conversation engine for realizing a dialogue with a user and for starting response generation.

[0492] The "means for generating an avatar of a specific character" is a function for generating an avatar to be displayed in a virtual reality space based on a user request or user data.

[0493] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring individual user data from a database and customizing the virtual reality environment based on that information.

[0494] The "means for managing real-time interactions with the user" is a function for processing input from the user in real time and providing an appropriate response.

[0495] The "means for performing billing processing" is a function for carrying out payment procedures associated with the use of paid services.

[0496] The "means for initializing the emotion recognition device" is a function for analyzing the user's voice and facial expression data and setting the engine for recognizing the emotion into an operational state.

[0497] "Means for analyzing the user's facial expression and voice data and generating a response based on the detected emotion" is a function for analyzing the emotion from the user's facial expression and voice and generating an adaptive response according to that emotion.

[0498] "Means for generating a response in accordance with a prompt sentence using a generative AI model" is a function that uses a generative AI model to generate an appropriate response in accordance with a prompt sentence from a user.

[0499] The "means for converting a prompt sentence into synthetic speech" is a function for converting text data generated as a prompt sentence into speech and providing it to the user.

[0500] This invention is a system for providing users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and its operation is characterized by generating adaptive responses based on the user's emotions. This system is composed of both a server and a terminal, and is realized by combining various hardware and software.

[0501] Server side

[0502] The server receives the user's request and initializes the virtual reality environment based on the request. Specifically, it generates a VR space and a specific character avatar tailored to the user's preferences. During the initialization process, the synthetic speech generator, conversational dialogue device, and emotion recognition device are activated.

[0503] The server retrieves user data from a database, such as MySQL or MongoDB, and then imports it into the virtual reality environment. The user data includes preferences and settings, and the virtual space is customized based on this information.

[0504] User input is sent to the server in real time, and a conversational dialogue system (e.g., OpenAI's GPT-3) generates an appropriate response to the user's input. This response is then provided as an adaptive response based on the analysis results of an emotion recognition system (e.g., Microsoft Azure Emotion AI or Affectiva).

[0505] Terminal side

[0506] On the device, the user puts on a virtual reality device (e.g., Oculus Rift or HTC Vive) and launches an application. At this time, a connection is established with the server and the virtual reality environment is loaded. On the device, the user interacts with the virtual environment through gestures and voice input.

[0507] The device manages the virtual reality environment based on data received from the server. The user selects services (new contract, model change, device operation guide, etc.) within the virtual reality environment, and the selection is sent to the server in real time. On the device side, the user interface is dynamically managed, and menus and guides are provided according to the user's actions.

[0508] User side

[0509] Users put on a VR device and enter a virtual reality environment to use the service. As a specific example of operation, if a user wishes to sign a new contract, a virtual character will propose a contract plan and provide an explanation using synthesized voice. Furthermore, if the user reacts emotionally (for example, if they show a surprised expression), the emotion recognition device will analyze this and generate an appropriate response such as, "Did I surprise you?"

[0510] Example prompt

[0511] "I want to sign a new smartphone contract."

[0512] Please tell me the features of this model.

[0513] "How much is the monthly fee?"

[0514] This system allows users to experience more natural and personal interactions within a virtual reality environment, and improves user satisfaction by generating responses using generative AI models and prompts.

[0515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0516] Step 1:

[0517] The server receives the user's request. When the user puts on the virtual reality device and starts the application, the device sends a request containing service details (new contract, model change, device operation instructions, etc.) to the server. Based on this input data, the server prepares to proceed to the next process.

[0518] Step 2:

[0519] The server initializes the virtual reality environment. Specifically, it generates a VR space tailored to the user's desired services and preferences, and prepares a specific character avatar. During this process, it reads the user's preferences and settings from a user database (e.g., MySQL or MongoDB) and reflects them in the virtual reality space. This operation is performed using a VRSDK (e.g., Unity or Unreal Engine).

[0520] Step 3:

[0521] The server initializes a synthetic speech generator, a conversational dialogue device, and an emotion recognition device. The synthetic speech generator (e.g., Google Text-to-Speech API) converts text into speech, the conversational dialogue device (e.g., OpenAI's GPT-3) generates appropriate responses to user input, and the emotion recognition device (e.g., Microsoft Azure Emotion AI or Affectiva) is used to analyze the user's facial expressions and voice data and recognize emotions.

[0522] Step 4:

[0523] The terminal displays the virtual reality environment to the user and manages the user interface. The user selects services within the virtual reality environment through gestures or voice input. The selected service content is transmitted from the terminal to the server. Based on this input data, the server prepares an appropriate response.

[0524] Step 5:

[0525] The server responds to user input in real time. The conversational dialogue device generates a text response based on the user's prompt (e.g., "I want to sign up for a new smartphone contract," "What are the features of this model?", "How much is the monthly fee?"). This text response is converted into speech by a synthetic speech generator and sent to the terminal as voice data.

[0526] Step 6:

[0527] The server generates an adaptive response based on the user's emotional data analyzed by the emotion recognition device. This emotional data is obtained from the user's facial expressions and voice data, and if the user looks surprised, for example, a response such as "Did I surprise you?" is generated. The generated response is converted into voice by a synthetic voice generation device and sent back to the terminal.

[0528] Step 7:

[0529] The device displays adaptive responses to the user and continues the interaction. The user continues to interact with the character in the virtual reality environment using gestures and voice input again. This interaction is repeated and continues until the user terminates the service.

[0530] Step 8:

[0531] The server processes billing when a paid service is used. When the user completes the service use, the terminal displays the billing procedure (e.g., inputting credit card information). When the user completes the payment procedure, the server charges the fee through a payment processing system (e.g., Stripe or PayPal).

[0532] Step 9:

[0533] The user removes the virtual reality device and ends the service. Once the interaction in the virtual reality environment is complete and the user exits the virtual reality shop, they return to the real world. This process allows the user to receive efficient and entertaining services within the virtual reality environment.

[0534] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0535] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0536] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0537] [Second embodiment]

[0538] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0539] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0540] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0541] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0542] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0543] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0544] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0545] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0546] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0547] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0548] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0549] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0550] This invention relates to a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using synthetic voice AI and conversational AI.

[0551] Server side

[0552] Request received

[0553] The server receives a request sent by the user. The request includes the user's identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0554] Initializing the virtual reality environment

[0555] The server sets up the virtual reality environment based on the user's requests, including creating a VR space tailored to the user's preferences and creating an avatar for a specified character (e.g., a celebrity or historical figure).

[0556] Initializing the synthetic voice AI and conversation AI

[0557] During the initialization phase, instances of synthetic speech AI and conversation AI are generated, which prepares functions such as converting text to speech and realizing natural dialogue with the user.

[0558] Acquiring and updating user data

[0559] The server retrieves user data from the database and reflects it in the virtual reality environment. User data includes user profile information and past usage history.

[0560] Managing real-time interactions

[0561] The server receives input from the user in real time and generates a response using conversational AI. The response is converted into voice through synthetic speech AI and provided to the user. For example, if the user says, "I'd like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[0562] Billing

[0563] When a paid service such as a terminal operation guide is used, the server executes billing processing to the user's account, allowing the user to easily pay after receiving the service in the virtual reality environment.

[0564] Terminal side

[0565] Launching a VR app

[0566] The user launches the app on their VR device, which initiates a connection with the server and loads the virtual reality environment.

[0567] Connecting to the Server

[0568] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[0569] Service Selection

[0570] Within the virtual reality environment, the user selects the service they wish to use (new contract, model change, terminal operation guide, etc.). Based on the selected service, the server instructs them to take the appropriate action.

[0571] User Interaction

[0572] The terminal manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. The terminal sends input from the user to the server, and receives and displays responses from the server in real time.

[0573] User side

[0574] Wearing a VR device

[0575] Users put on the VR device and prepare to use the service. By equipping the VR headset and handheld controllers, they can enter the virtual reality environment.

[0576] Service Selection

[0577] Users use an interface in the virtual reality environment to select services such as new contracts, model changes, and operation guides, allowing them to efficiently complete the procedures they desire.

[0578] Start a dialogue

[0579] The user interacts with a specific character avatar and receives services. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[0580] Payment process

[0581] For paid services, users complete payment procedures within the virtual reality environment, including entering credit card information and QR code payments.

[0582] VR Shop closed

[0583] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0584] Through these steps, the user can receive efficient and entertaining services within the virtual reality environment. The system of the present invention improves user convenience and solves the problems of the prior art.

[0585] The processing flow will be explained below.

[0586] Server side

[0587] Processing steps from receiving a request to generating a response

[0588] Step 1:

[0589] The server receives a user request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0590] Step 2:

[0591] The server initializes the virtual reality environment based on the request, creating a VR space tailored to the user's preferences.

[0592] Step 3:

[0593] The server initializes the synthetic voice AI and the conversation AI. The synthetic voice AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input.

[0594] Step 4:

[0595] The server retrieves user data from a database, including user profile information and past usage history.

[0596] Step 5:

[0597] The server reflects the user data in the virtual reality environment, allowing the avatar of the character set to provide appropriate services to the user.

[0598] Step 6:

[0599] The server receives input from the user in real time and generates a response using conversational AI, which is then converted into speech using synthetic speech AI and provided to the user.

[0600] Step 7:

[0601] When a paid service such as terminal operation guidance is used, the server executes billing processing, which charges the user's account.

[0602] Terminal side

[0603] Processing steps from app launch to user interface management

[0604] Step 1:

[0605] The user launches the VR app on their device, which initiates a connection with the server.

[0606] Step 2:

[0607] The device connects to the server, sends the user's request, receives the VR environment data returned from the server, and displays the virtual reality environment.

[0608] Step 3:

[0609] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[0610] Step 4:

[0611] The terminal manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[0612] Step 5:

[0613] Users interact with characters in a virtual reality environment through gestures and voice input, and the device transmits this input to a server, which receives and displays responses.

[0614] User side

[0615] Processing steps from device installation to payment procedure

[0616] Step 1:

[0617] The user puts on the VR device and prepares to use the service, which allows them to enter the virtual reality environment.

[0618] Step 2:

[0619] The user selects the desired service using an interface provided within the virtual reality environment.

[0620] Step 3:

[0621] Based on the service selected, the user initiates a dialogue with a specific character avatar, which allows the user to proceed with the required procedure.

[0622] Step 4:

[0623] When using paid services, users can complete payment procedures within the virtual reality environment, using methods such as credit card information or QR code payment.

[0624] Step 5:

[0625] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0626] Example 1

[0627] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0628] Currently, there is no system in existence that allows users to smoothly and efficiently receive guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. In particular, major challenges remain in real-time dialogue, high operability of the user interface, natural communication using synthesized speech and conversational agents, and simple payment procedures.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0630] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice generating means, means for initializing a dialogue agent, means for generating a virtual avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing payments, and means for generating and displaying data for the virtual reality environment. This allows the user to smoothly receive guidance on contracts, model changes, and terminal operations within the VR environment and complete operations and procedures through real-time dialogue.

[0631] The "means for receiving a user request" is a function for receiving an information request sent from a user terminal.

[0632] "Means for initializing a virtual reality environment" is a function that configures and creates a VR space based on the user's requests.

[0633] "Synthetic voice generation means" is a technology for converting text data into voice data.

[0634] The "means for initializing a conversational agent" is a function for preparing an artificial intelligence to have a natural conversation with a user.

[0635] The "means for generating a virtual avatar of a specific character" is a function for creating a 3D model of a specified character in response to a user request.

[0636] "Means of acquiring user data and reflecting it in a virtual reality environment" refers to technology that acquires user information from a database and applies it to a VR environment.

[0637] The "means for managing real-time dialogue with the user" is a function that responds immediately to input from the user and provides an appropriate response.

[0638] The "payment processing means" is a system that charges the user for the service usage and completes the settlement.

[0639] The "means for generating and displaying data for a virtual reality environment" is a function for generating data necessary for a virtual reality environment and displaying it on a user terminal.

[0640] The "means for selecting a service" is an interface that allows a user to select the required service within the virtual reality environment.

[0641] "Means for connecting to a server and obtaining data" refers to a technique by which a user terminal accesses a server and obtains the necessary information.

[0642] A "means of interaction through a user interface" is an interface that a user uses to operate and interact.

[0643] The "means for providing guidance in accordance with the type of service to be used" is a function for providing appropriate instructions and guidance to the user in accordance with the selected service.

[0644] "Means for wearing a virtual reality device and starting a service" refers to the steps a user takes to start using a service using a VR device.

[0645] The "means for exiting the virtual reality shop after service usage has ended" is a function that allows the user to exit the virtual reality environment after finishing using the service.

[0646] "Means for interacting through gestures and voice input" refers to technologies that allow users to communicate with the system using hand movements and voice.

[0647] The present invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using a synthetic speech generation means and a dialogue agent.

[0648] Server side

[0649] Request received

[0650] The server receives requests sent from users using the HTTPS protocol. The request contains the user's identification information (user ID) and the type of service desired (new contract, model change, device operation guide, etc.). The received request is parsed in JSON format and the appropriate processing is carried out.

[0651] Initializing the virtual reality environment

[0652] The server uses Unity to set up a virtual reality environment based on the user's request. This includes generating a VR space tailored to the user's preferences and placing a specific character avatar created using Blender. The virtual reality environment data is sent to the user's device and displayed.

[0653] Initialization of synthetic speech generator and dialogue agent

[0654] The server initializes the synthetic speech generator using Google Cloud's Text-to-Speech API and the conversational agent using Dialogflow, which prepares the text-to-speech conversion function and the ability to have natural conversations with the user.

[0655] Acquiring and updating user data

[0656] The server retrieves user data from the MySQL database and reflects it in the virtual reality environment. The user data includes user profile information and past usage history. This allows the server to provide appropriate services for each individual user.

[0657] Managing real-time interactions

[0658] The server receives voice input from the user and generates an appropriate response using Dialogflow. The generated response is converted into audio using Google Cloud's Text-to-Speech API and provided to the user. For example, if a user types "I would like to sign a new contract," the conversational agent will suggest an appropriate contract plan and explain its details in audio.

[0659] Payment Processing

[0660] The server processes payments using the Stripe API. For example, if a paid service such as a terminal operation guide is used, the server will charge the user's account. This allows the user to easily pay after receiving the service in the virtual reality environment.

[0661] Terminal side

[0662] Launching a VR app

[0663] A user launches an app on a VR device (e.g., Oculus Quest), which initiates a connection with the server and loads the virtual reality environment. VR apps run using the Oculus VR SDK.

[0664] Connecting to the Server

[0665] The device connects to the server and sends the user's request to the server. The device receives the VR environment data returned from the server and displays it. The device connects using Wi-Fi or 5G.

[0666] Service Selection

[0667] Within the virtual reality environment, users select the service they want to use, such as signing up for a new contract, upgrading their device, or receiving device operation instructions. Service selection is performed using an interactive menu within the VR app.

[0668] User Interaction

[0669] The device manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the device's operation procedures.

[0670] User side

[0671] Wearing a VR device

[0672] Users put on a VR device (such as Oculus Quest) and prepare to use the service, which allows them to enter an immersive virtual reality environment.

[0673] Service Selection

[0674] Users can select services such as new contracts, model changes, and operation guidance within the virtual reality environment, allowing them to efficiently complete the procedures they desire.

[0675] Start a dialogue

[0676] Users interact with a specific character avatar to receive services, including obtaining necessary information through natural conversation. For example, a user can begin the process by entering "I would like to sign a new contract."

[0677] Payment process

[0678] When using paid services, users complete payment procedures within the virtual reality environment, including entering credit card information or using a QR code. Payments are processed by the server via the Stripe API.

[0679] VR Shop closed

[0680] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0681] Examples of concrete examples and prompts

[0682] For example, if a user wants to sign up for a new smartphone model, the following steps are taken.

[0683] 1. The user puts on the Oculus Quest and launches the VR app.

[0684] 2. Select "New Contract" within the virtual reality environment.

[0685] 3. A specific character will appear and ask questions about the contract in a dialogue format.

[0686] 4. The server generates the appropriate contract plan through Dialogflow and converts it into audio using Google Cloud's Text-to-Speech API.

[0687] 5. The user follows the character's instructions to complete the contract procedure.

[0688] Prompt Sentence Examples

[0689] "Enter a scenario where a user signs up for a new smartphone:

[0690] 1. First, the user puts on the VR device and launches the VR app.

[0691] 2. Select a service menu in the virtual reality environment

[0692] 3. Check the details of the new contract through dialogue with the character

[0693] 4. Payment Procedure

[0694] 5. Close the VR shop.

[0695] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0696] Step 1: Receiving a request

[0697] The server receives a request from the user via the HTTPS protocol. This request includes the user ID and the details of the desired service (e.g., new contract, model change, terminal operation guide). The input is the user's request data, which is analyzed to extract the necessary information. The output is the analyzed user ID and service content data. Specifically, the server uses Apache HTTP Server to analyze the received JSON data.

[0698] Step 2: Initializing the Virtual Reality Environment

[0699] The server uses Unity to set up a virtual reality environment based on the user's request. The input is the user ID and service content data, and based on this, it configures the VR space and generates a specific character avatar. The output is the generated VR space data. Specifically, it calls the Unity API and places a VR space tailored to the user's preferences and a character avatar generated using Blender.

[0700] Step 3: Initialize the synthetic speech generator and the dialogue agent

[0701] The server initializes the synthetic speech generation means using Google Cloud's Text-to-Speech API and the dialogue agent using Dialogflow. User data and service content data are used as input, and each API is configured based on this. Synthetic speech and dialogue agent instances are obtained as output. Specifically, the server accesses the API key and settings via the Google Cloud Console and generates the required instances.

[0702] Step 4: Obtaining and updating user data

[0703] The server retrieves user data from a MySQL database and reflects it in the virtual reality environment. It executes a database query using the user ID as input, and obtains user profile information and usage history as output. Specifically, it executes a MySQL query to retrieve the necessary data and applies it to the VR environment settings.

[0704] Step 5: Managing real-time interactions

[0705] The server receives voice input from the user and generates an appropriate response using Dialogflow. It then converts the voice input into audio using Google Cloud's Text-to-Speech API and provides it to the user. The input is the user's voice input data, which is analyzed and converted into text, and a response is generated using a conversational agent. The output is audio response data. Specifically, the voice data is temporarily saved to a WAV file, sent to Dialogflow, and the returned text is passed to the synthetic speech API to generate audio.

[0706] Step 6: Payment Processing

[0707] After a paid service is used, the server processes the payment through the Stripe API. The input is the user's payment information and service information, and billing is performed based on this. The output is payment confirmation data. Specifically, the server calls the Stripe API, processes the user's payment information, and completes the payment.

[0708] Step 7: Launch the VR app

[0709] A user launches an app on a VR device such as Oculus Quest. The input is a command to launch the VR device, and the output is to start accessing the VR environment. Specifically, the app launches using the Oculus VR SDK and establishes a network connection.

[0710] Step 8: Connect to the Server

[0711] The device connects to the server and sends the user's request to the server. The input is the user's request data, and the output is the virtual reality environment data. Specifically, the device sends an HTTP request to the server and receives the VR environment data returned from the server.

[0712] Step 9: Select a service

[0713] The user selects the service they want to use in the virtual reality environment. The input is the user's service selection, and the output is instructions based on the selected service returned from the server. Specifically, the user selects the service using an interactive menu within the VR app.

[0714] Step 10: User Interaction

[0715] The device manages interactions with the user. Inputs include user gestures and voice inputs, and output displays response data from the server. Specifically, it uses WebSocket to capture user gestures and voice commands and send them to the server.

[0716] Step 11: Put on the VR device

[0717] The user puts on the VR device and prepares to use the service. The input is a command to put on the VR device, and the output is to start accessing the virtual reality environment. Specifically, the user puts on the Oculus Quest and prepares the handheld controller.

[0718] Step 12: Select a service

[0719] Within the virtual reality environment, the user selects services such as new contracts, model changes, and operation guidance. The input is the user's service selection, and the output is instructions based on the selected service provided by the server. Specific operations involve using a menu provided within the VR environment to select a service that suits the purpose.

[0720] Step 13: Start a conversation

[0721] A user initiates a dialogue with a specific character avatar. The input is the user's voice input, and the dialogue agent generates a response accordingly. The output is a voice response from the dialogue agent. Specifically, the user enters a voice command, which is processed by the dialogue agent to generate a response.

[0722] Step 14: Payment Process

[0723] When using a paid service, the user completes the payment process within the virtual reality environment. The input is the user's payment information, and the output is payment confirmation data. Specifically, the user enters their credit card information within the virtual reality environment and completes the payment via the Stripe API.

[0724] Step 15: Finish the VR Shop

[0725] After completing the service, the user removes the VR device and exits the virtual reality shop. The input is the user's exit command, and the output is the ability to return to the real world. The specific actions are to close the VR app and remove the Oculus Quest.

[0726] (Application example 1)

[0727] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] Services provided within conventional virtual reality environments have not fully satisfied user convenience. In particular, it has been difficult to provide information on new contracts, model upgrades, and device operation methods quickly and effectively within a virtual reality environment. Furthermore, user interaction has been limited, and there has been a lack of means for real-time dialogue, resulting in a poor user experience. Furthermore, there has been a demand for a method that allows more users to easily access services by using virtual reality devices connected to smartphones.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0730] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice artificial intelligence, means for initializing a conversational artificial intelligence, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for realizing a dialogue system for providing guidance on new contracts, model changes, and device operation within the virtual reality environment, and means for the user to interact using a virtual reality headset and controller connected to a smartphone. This allows the user to easily and quickly receive more convenient and effective guidance on new contracts, model changes, and device operation within the virtual reality environment.

[0731] 1. "Means for receiving user requests" refers to the methods and functions by which the system receives operations and requests from users.

[0732] 2. "Means for initializing a virtual reality environment" means methods and functionality for configuring and building a user-accessible virtual reality environment.

[0733] 3. "Means for initializing a synthetic speech artificial intelligence" means methods and functions for configuring and activating an artificial intelligence that converts text data into speech data.

[0734] 4. "Means for initializing conversational AI" refers to methods and functions for configuring and launching an AI that communicates in natural language.

[0735] 5. "Means for generating an avatar of a specific character" means methods and functions for displaying a specific character within a virtual reality environment based on a user's request.

[0736] 6. "Means for acquiring user data and reflecting it in the virtual reality environment" refers to the methods and functions for acquiring user information from the database and applying it within the virtual reality environment.

[0737] 7. "Means for managing real-time interactions with users" refers to methods and functions that enable users and systems to interact with each other in real time.

[0738] 8. "Means for billing processing" refers to the methods and functions for managing and processing payment procedures when a user uses a paid service.

[0739] 9. "Means for realizing an interactive system for providing guidance on new contracts, model changes, and device operation within a virtual reality environment" refers to methods and functions that provide users with interactive guidance on new contracts, model changes, and device operation within a virtual reality environment.

[0740] 10. "Means for user interaction using a virtual reality headset and controllers connected to a smartphone" means methods and functionality that allow a user to navigate and interact within a virtual reality environment using a virtual reality headset and controllers connected to a smartphone.

[0741] The present invention is a system that allows users to receive service information such as new contracts, model changes, and terminal operation methods within a virtual reality environment. This system is composed of three parties: a server, a terminal, and a user, and uses detailed hardware and software to fulfill each role.

[0742] Server-side implementation

[0743] The server includes the following means:

[0744] 1. Means of receiving a user request:

[0745] The server receives a request sent by the user, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0746] 2. Means of initializing the virtual reality environment:

[0747] Based on the user's request, the server initializes the virtual reality environment using a VR toolkit (e.g., Unity or Unreal Engine), including creating a VR space tailored to the user's preferences and generating an avatar for a specific character.

[0748] 3. Methods for initializing the synthetic voice AI:

[0749] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly) that converts text data into voice data.

[0750] 4. Methods for initializing conversational AI:

[0751] The server initializes a conversational AI (e.g., Dialogflow or Microsoft Bot Framework) to enable natural dialogue with the user.

[0752] 5. How user data is acquired and reflected in the virtual reality environment:

[0753] The server retrieves the user's profile information and past usage history from the database and reflects this in the virtual reality environment.

[0754] 6. Means of managing real-time user interactions:

[0755] The server receives voice input from the user in real time and uses conversational AI to generate a response, which is then converted into speech by synthetic speech AI and provided to the user.

[0756] 7. Means of processing claims:

[0757] When a paid service such as terminal operation guidance is used, the server processes the billing to the user's account, allowing the user to easily make payment after receiving the service.

[0758] Terminal side embodiment

[0759] The terminal includes the following means:

[0760] 1. Launch the VR app:

[0761] The user puts on a VR headset and launches a VR application on their smartphone, which initiates a connection with the server.

[0762] 2. Connect to the server:

[0763] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[0764] 3. Service Selection:

[0765] The user uses an interface within the virtual reality environment to select services such as new contracts, model changes, and terminal operation guidance.

[0766] 4. User Interaction:

[0767] The terminal manages interactions with the user, who interacts with specific characters through gestures and voice input to receive necessary procedures and instructions.

[0768] User-Side Embodiment

[0769] The user includes the following means:

[0770] 1. Wearing the VR device:

[0771] The user prepares to enter the virtual reality environment by putting on a VR headset and preparing the handheld controllers.

[0772] 2. Service Selection:

[0773] The user operates the interface provided within the virtual reality environment to select services such as new contracts, model changes, and operation guidance.

[0774] 3. Start a conversation:

[0775] The user can start a conversation with a specific character and receive the necessary information or procedures. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[0776] 4. Payment Procedure:

[0777] When using a paid service, users can complete payment procedures within the virtual reality environment by entering credit card information or using a QR code.

[0778] 5. VR Shop Closure:

[0779] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0780] Specific examples

[0781] For example, if a user wishes to sign a new contract, they can say "I would like to sign a new contract" by voice within the VR environment, and a character avatar will respond by saying, "Here are our new contract plans. Which plan would you like?" In this way, each time the user asks a question, the conversational AI generates an appropriate response and replies audibly through the synthetic voice AI.

[0782] Prompt Sentence Examples

[0783] Describe a dialogue system with an AI character that realizes a scenario in which a user wishes to make a new contract in a VR environment. Explain in detail the response and behavior of the character avatar when the user says, "I want to make a new contract."

[0784] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0785] Step 1:

[0786] The server receives the user's request, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.). Based on this input, the server analyzes the request and begins preparing an appropriate virtual reality environment.

[0787] Step 2:

[0788] The server initializes the virtual reality environment. This is the process of generating a VR space tailored to the user's preferences using a VR toolkit (e.g., Unity or Unreal Engine) based on the user's request. An avatar for the specified character is also generated and configured. As a result, a virtual reality environment tailored to the user is constructed.

[0789] Step 3:

[0790] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly). At this stage, an instance of the synthetic voice AI is created and prepared to convert text data into speech data. This allows speech output for subsequent user interactions.

[0791] Step 4:

[0792] The server initializes the conversational AI (e.g., Dialogflow or Microsoft Bot Framework), creates an instance of the conversational AI, and configures it to enable natural dialogue with the user. This configuration includes preparing a model to generate appropriate responses to user input.

[0793] Step 5:

[0794] The server retrieves user data from the database and reflects it in the virtual reality environment. The user data includes profile information and usage history, and personalizes the VR space based on that information. This information is used to improve the user experience.

[0795] Step 6:

[0796] The server manages real-time dialogue with the user. In this step, the user's voice input is converted and analyzed by conversational AI. Based on the analysis results, a response is generated through synthetic voice AI and provided to the user. This process is carried out in real time, ensuring smooth interaction between the user and the system.

[0797] Step 7:

[0798] The device launches the VR app and establishes a connection with the server, allowing the user to access the virtual reality environment and receive data sent from the server, enabling the user to view the virtual reality environment and operate the interface.

[0799] Step 8:

[0800] The device provides an interface for the user to select services. The user uses a VR headset and handheld controller to select services such as new contracts, model changes, and device operation guidance. This input causes the server to prepare the next response.

[0801] Step 9:

[0802] The user operates the interface and begins a dialogue with a specific character. For example, if the user voice-inputs, "I'd like to sign a new contract," the character will respond with, "Here are our new contract plans. Which plan would you like?" This dialogue progresses in real time, with responses generated by conversational AI.

[0803] Step 10:

[0804] When a user uses a paid service, the server handles the billing process. Specifically, the charge is sent to the user's account via credit card information entry or QR code payment, allowing the user to complete payment easily.

[0805] Step 11:

[0806] After completing their use of the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world and re-prepare the device for their next use.

[0807] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0808] This invention relates to a system that provides users with guidance on new contracts, model changes, and terminal operation methods in a virtual reality environment, while also recognizing the user's emotions and generating adaptive responses based on them. This system receives user requests, initializes the virtual reality environment, and interacts with the user using a synthetic voice AI, a conversational AI, and an emotion engine.

[0809] Server side

[0810] Processing from receiving a request to generating a response

[0811] The server receives the user's request. This request includes the user ID and the type of service desired (new contract, model change, device operation guide, etc.). The server then initializes the virtual reality environment based on the request. This includes generating a VR space tailored to the user's preferences and creating an avatar for the specified character.

[0812] Initialize instances of the synthetic speech AI and conversation AI. The synthetic speech AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input. The emotion engine is also initialized. The emotion engine analyzes the user's facial expressions and voice and recognizes emotions.

[0813] User data is retrieved from the database and reflected in the virtual reality environment, allowing the character avatar to provide appropriate services to the user.

[0814] The server receives input from the user in real time and generates a response using conversational AI. The generated response is converted into voice using synthetic voice AI and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[0815] The emotion engine analyzes the user's facial expressions and voice to detect emotions. Based on the detected emotions, the conversational AI generates a response that is adapted to the user. For example, if the user expresses surprise, the conversational AI will generate a response such as, "Did I surprise you?"

[0816] When a paid service such as a terminal operation guide is used, a billing process is executed, and the fee is charged to the user's account.

[0817] Terminal side

[0818] Processing from app launch to user interface management

[0819] The user launches the VR app on their device, which initiates a connection with the server and loads the virtual reality environment. The device then connects to the server and sends the user's request. The server returns the VR environment data, which is then displayed.

[0820] The user selects the services they wish to use within the virtual reality environment (new contract, model change, device operation guide, etc.). The selection is sent to the server. The device manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[0821] The user interacts with the character in the virtual reality environment through gestures and voice input. The device sends this input to the server, receives responses, and displays them. The emotion engine analyzes the user's facial expressions and voice and sends emotional data to the server.

[0822] User side

[0823] Processing from device installation to payment procedures

[0824] The user puts on the VR device and prepares to use the service. This allows the user to enter the virtual reality environment. The user selects the desired service using the interface provided within the virtual reality environment.

[0825] Based on the service selected, the user can start a conversation with a specific character avatar, which will then guide them through the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the operation procedure.

[0826] When using paid services, users make payments within the virtual reality environment. Payment methods include credit card entry and QR code payment. After completing the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0827] Through these steps, users can enjoy efficient and entertaining services within the virtual reality environment. Furthermore, the emotion engine recognizes the user's emotions and provides adaptive responses, further enhancing the user experience. The system of the present invention improves user convenience and solves the problems of the past.

[0828] The processing flow will be explained below.

[0829] Server side

[0830] Processing steps from receiving a request to generating a response

[0831] Step 1:

[0832] The server receives the user's request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[0833] Step 2:

[0834] The server initializes the virtual reality environment based on the request, generates a VR space according to the user's preferences, and prepares to generate an avatar for the specified character.

[0835] Step 3:

[0836] The server initializes instances of a synthetic speech AI and a conversational AI. The synthetic speech AI has the function of converting text to speech, and the conversational AI generates appropriate responses to user input.

[0837] Step 4:

[0838] The server retrieves user data from a database, including profile information and past usage history.

[0839] Step 5:

[0840] The server reflects the acquired user data in the virtual reality environment and applies the user's individual settings.

[0841] Step 6:

[0842] The server initiates a real-time dialogue with the user using conversational AI, taking the user's input and generating the optimal response.

[0843] Step 7:

[0844] The server uses synthetic voice AI to convert the generated response into speech, which is then provided to the user.

[0845] Step 8:

[0846] The server analyzes the user's facial expressions and voice through an emotion engine to recognize their emotions.

[0847] Step 9:

[0848] The server adjusts the conversational AI's responses based on the emotions it recognizes, for example, providing a follow-up message such as "Don't worry" if the user expresses anxiety.

[0849] Step 10:

[0850] The server performs billing when terminal operation guidance or other paid services are used, and charges are applied to the user's account.

[0851] Terminal side

[0852] Processing steps from app launch to user interface management

[0853] Step 1:

[0854] The user launches the VR app on their device, which initiates a connection with the server.

[0855] Step 2:

[0856] The device connects to the server and sends the user's request, which includes information such as the user ID and the type of service.

[0857] Step 3:

[0858] The device receives the VR environment data returned from the server and displays the virtual reality environment.

[0859] Step 4:

[0860] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[0861] Step 5:

[0862] The terminal manages the user interface within the virtual reality environment, displaying menus and handling interactions.

[0863] Step 6:

[0864] The emotion engine analyzes the user's facial expressions and voice, and sends the emotional data to the server.

[0865] Step 7:

[0866] The device receives responses from the server and displays them both audibly and visually, and adaptive responses are provided based on data from the emotion engine.

[0867] User side

[0868] Processing steps from device installation to payment procedure

[0869] Step 1:

[0870] The user puts on the VR device and prepares to use the service, allowing them to enter the virtual reality environment.

[0871] Step 2:

[0872] The user uses an interface provided within the virtual reality environment to select the service they want, for example, to sign up for a new contract or upgrade their device.

[0873] Step 3:

[0874] Based on the service selected, the user begins a conversation with a specific character avatar, who guides the user through specific procedures.

[0875] Step 4:

[0876] The user follows the character's instructions as they interact with the character, while the emotion engine recognizes the user's emotions and adjusts responses accordingly.

[0877] Step 5:

[0878] When using paid services, users can complete payment procedures within the virtual reality environment, such as entering credit card information or using a QR code.

[0879] Step 6:

[0880] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[0881] Example 2

[0882] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] Conventional virtual reality environments have difficulty understanding user emotions and providing adaptive responses. They also lack real-time dialogue management and services tailored to user needs. This can limit the user experience and reduce user satisfaction. Another issue is the complicated billing and payment procedures for paid services, which places a burden on users.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0885] In this invention, the server includes a means for analyzing a user's facial expressions and voice to recognize emotions, a means for generating an adaptive response based on the emotion recognition, and a means for performing billing processing. This makes it possible to understand a user's emotions in real time and provide an adaptive response based on the emotion. It also realizes appropriate billing processing and payment procedures, improving the user experience and convenience.

[0886] The "means for receiving a user request" refers to a function or module for acquiring a request from a user.

[0887] A "means for initializing a virtual reality environment" is a system or program for setting up and starting up a virtual environment.

[0888] The "means for initializing the synthetic speech generation means" is a function for starting and setting up a speech generation system that has the function of converting text data into speech data.

[0889] The "means for initializing a conversation generation model" is a function for starting and setting up a conversation model that generates an appropriate response to a user's input.

[0890] A "means for generating an avatar of a specific character" is a system or program that generates the appearance and behavior of a virtual character based on the user's requests and settings.

[0891] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring user information from a database, etc., and applying it to the virtual reality space.

[0892] "Means for managing real-time user interactions" refers to functions and modules that monitor and control real-time communication between users and the system.

[0893] The "means for analyzing the user's facial expressions and voice and recognizing emotions" is an analysis system for detecting the user's emotional state from facial expression and voice data.

[0894] The "means for generating adaptive responses based on emotion recognition" is a function or module that generates an appropriate response that matches the recognized emotional state of the user.

[0895] The "means for billing" refers to a system or program that calculates fees for the use of paid services and bills the user account.

[0896] "Means for interaction through a user interface" refers to functions or modules that provide an interface that users can visually operate and communicate through it.

[0897] "Means for wearing a virtual reality device and starting the service" refers to the procedures and modules for starting to use the system after wearing the VR device.

[0898] "Means for selecting services within a virtual reality environment" refers to an interface or system that allows users to select the desired service from the various services provided within the virtual space.

[0899] The "means for starting a dialogue with a character" refers to a function or module for starting a conversation with a generated virtual character and advancing the dialogue.

[0900] "Means for carrying out payment procedures when using a paid service" refers to a system or module for completing payment when a user selects a paid service.

[0901] "Means for exiting the virtual reality shop after using the service" refers to the procedures and functions that allow users to safely exit the virtual reality environment after they have finished using the service.

[0902] This invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and also has the function of recognizing the user's emotions and generating adaptive responses. An embodiment of this system will be described in detail below.

[0903] To receive a user request, the server acquires the user ID and desired service details via an HTTP request or WebSocket. This request includes the type of service, such as a new contract, a model change, or device operation instructions.

[0904] The server then initializes the virtual reality environment, which involves using the Unreal Engine to generate a VR space tailored to the user's preferences and an avatar for the specified character.

[0905] After that, the server initializes the synthetic speech generator (Google Cloud Text-to-Speech) and the conversation generation model (OpenAI GPT-3), as well as the emotion engine (Microsoft Azure Emotion API) that analyzes the user's facial expressions and voice to recognize emotions.

[0906] The server retrieves user data from the database and uses this information to reflect in the virtual reality environment, a process that allows the character avatar to provide appropriate services to the user.

[0907] For real-time dialogue management, the server receives user input and generates an appropriate response using a conversation generation model. The generated response is converted into speech using a synthetic speech generation means and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversation generation model will suggest an appropriate contract plan and explain its details in voice.

[0908] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Based on the recognized emotions, the conversation generation model generates adaptive responses. For example, if the user expresses surprise, the response generated will be, "Did I surprise you?"

[0909] If a paid service is used, the server executes a billing process and charges the fee to the user's account, completing the use of the service.

[0910] On the device side, the user launches a VR app and connects to the server using HTTP or WebSocket communication. The device retrieves the virtual reality environment data returned from the server and displays it using Unity. Within the virtual reality environment, the user selects the service they wish to use and sends that selection to the server. The device also manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[0911] Finally, the user puts on the VR device, enters the virtual reality environment, and selects a service using the provided interface. They then begin interacting with a specific character avatar within the virtual reality environment and carry out the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the steps. When using a paid service, the user completes the payment procedure within the virtual reality environment, and once the service is complete, the user removes the VR device and exits the virtual reality shop to return to the real world.

[0912] As a concrete example, the following prompt sentences can be used by the conversation generation model to smoothly engage the user:

[0913] User: I'd like to sign up. What plans are available?

[0914] System: There are several plans available for new subscribers: Basic Plan, Premium Plan, and Family Plan. Want to learn more about each plan?

[0915] In this way, the embodiments of the invention involve the use of specific hardware and software, and include detailed data processing and data calculation processing, making it possible to provide users with efficient and entertaining services.

[0916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0917] The flow of this system's program processing

[0918] Server-side processing

[0919] Step 1: Receiving a user request

[0920] How it works: The server receives the user's ID and desired service via an HTTP request or WebSocket.

[0921] Input: User ID, desired service (new contract, model change, terminal operation guide, etc.)

[0922] Output: Received data (user ID, service details)

[0923] Step 2: Initializing the Virtual Reality Environment

[0924] How it works: The server generates a VR space using Unreal Engine based on the received user data and generates an avatar for a specific character.

[0925] Input: User data, service details

[0926] Output: Initialized VR space, character avatar

[0927] Step 3: Initialize the synthetic speech generator and the speech generation model

[0928] How it works: The server initializes Google Cloud Text-to-Speech and OpenAI GPT-3.

[0929] Input: None (at system startup)

[0930] Output: Initialized synthetic speech generator, speech generation model

[0931] Step 4: Obtaining and updating user data

[0932] Operation: The server retrieves user data from the database and reflects it in the virtual reality environment.

[0933] Input: User ID

[0934] Output: Updated VR environment

[0935] Step 5: Receiving user input and generating a response

[0936] How it works: The server receives voice or text input from the user, uses a speech generation model to generate an appropriate response, and converts the response into audio using Google Cloud Text-to-Speech.

[0937] Input: User input (voice or text)

[0938] Output: Response voice

[0939] Step 6: Emotion recognition and adaptive response generation

[0940] How it works: The server uses the Microsoft Azure Emotion API to analyze the user's facial expressions and voice to recognize emotions, and then uses a conversation generation model to generate adaptive responses.

[0941] Input: User's facial expression data, voice data

[0942] Output: Adaptive response

[0943] Step 7: Billing for paid services

[0944] How it works: The server calculates the fee when a paid service is used and bills the user's account.

[0945] Input: Service information used

[0946] Output: Billing information, updated user account

[0947] Terminal side processing

[0948] Step 1: Launch the VR app

[0949] How it works: The user launches the VR app on their device and connects to the server.

[0950] Input: VR app launch command

[0951] Output: Server connection started

[0952] Step 2: Acquiring and displaying data in a VR environment

[0953] How it works: The device retrieves VR environment data from the server and displays the virtual reality environment using Unity.

[0954] Input: VR data from the server

[0955] Output: The displayed VR environment

[0956] Step 3: Select service and submit

[0957] Operation: The user selects the service to use within the virtual reality environment, and the device sends the selection to the server.

[0958] Input: Service selection (user input)

[0959] Output: Selected service details (sent from server)

[0960] Step 4: Managing the User Interface

[0961] Operation: The terminal manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[0962] Input: User interaction

[0963] Output: Updated user interface

[0964] Step 5: Sending User Input and Displaying Responses

[0965] How it works: The user's voice or gesture input is sent to the server, and the response from the server is displayed.

[0966] Input: User input (voice and gestures)

[0967] Output: Response from the server (visual and audio output)

[0968] Step 6: Sending Emotion Data

[0969] How it works: The emotion engine analyzes the user's facial expressions and voice and sends the data to the server.

[0970] Input: User's facial expression data, voice data

[0971] Output: Emotion data (sent to server)

[0972] User processing

[0973] Step 1: Put on your VR device

[0974] Action: The user puts on the VR device and prepares to use the service.

[0975] Input: None (wearing operation)

[0976] Output: VR device is installed

[0977] Step 2: Select a service

[0978] Operation: The user selects the desired service using the interface provided within the virtual reality environment.

[0979] Input: Service selection by user

[0980] Output: Selected service content

[0981] Step 3: Interact with the character

[0982] Action: The user initiates a dialogue with a specific character avatar and proceeds with the required procedures.

[0983] Input: User instruction to start a dialogue

[0984] Output: Started conversation

[0985] Step 4: Complete the payment process

[0986] Operation: When a user uses a paid service, the payment process is carried out within the VR environment.

[0987] Input: Payment information (credit card information or QR code)

[0988] Output: Completed payment

[0989] Step 5: Completing the service and returning to the real world

[0990] Operation: After completing the service, the user removes the VR device and exits the virtual reality shop to return to the real world.

[0991] Input: None (removal operation)

[0992] Output: Return to the real world

[0993] (Application example 2)

[0994] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0995] The problem to be solved by this invention is to provide a system that improves the user experience in a virtual reality environment and generates responses that adapt to the user's emotions. Conventional systems lack the ability to analyze the user's facial expressions and voice in real time and provide adaptive responses based on the analysis, which can lead to reduced user satisfaction. Furthermore, the means for processing prompts using generative AI models and converting them into synthetic speech are insufficient, resulting in a lack of naturalness in the interaction.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0997] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic speech generation device, means for initializing a conversational dialogue device, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for initializing an emotion recognition device, means for analyzing the user's facial expressions and voice data and generating a response based on the detected emotion, means for generating a response according to a prompt sentence using a generative AI model, and means for converting the prompt sentence into synthetic speech. This makes interaction with the user more natural and personal, thereby improving user satisfaction.

[0998] "Means for receiving user requests" refers to a function for receiving service requests and operation instructions sent by users and processing them within the system.

[0999] The "means for initializing a virtual reality environment" is a function for constructing and starting a virtual reality space based on a user's request.

[1000] The "means for initializing the synthetic speech generation device" is a function for preparing the synthetic speech technology that converts text data into natural speech and setting it into an operational state.

[1001] The "means for initializing the conversation dialogue device" is a function for constructing a conversation engine for realizing a dialogue with a user and for starting response generation.

[1002] The "means for generating an avatar of a specific character" is a function for generating an avatar to be displayed in a virtual reality space based on a user request or user data.

[1003] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring individual user data from a database and customizing the virtual reality environment based on that information.

[1004] The "means for managing real-time interactions with the user" is a function for processing input from the user in real time and providing an appropriate response.

[1005] The "means for performing billing processing" is a function for carrying out payment procedures associated with the use of paid services.

[1006] The "means for initializing the emotion recognition device" is a function for analyzing the user's voice and facial expression data and setting the engine for recognizing the emotion into an operational state.

[1007] "Means for analyzing the user's facial expression and voice data and generating a response based on the detected emotion" is a function for analyzing the emotion from the user's facial expression and voice and generating an adaptive response according to that emotion.

[1008] "Means for generating a response in accordance with a prompt sentence using a generative AI model" is a function that uses a generative AI model to generate an appropriate response in accordance with a prompt sentence from a user.

[1009] The "means for converting a prompt sentence into synthetic speech" is a function for converting text data generated as a prompt sentence into speech and providing it to the user.

[1010] This invention is a system for providing users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and its operation is characterized by generating adaptive responses based on the user's emotions. This system is composed of both a server and a terminal, and is realized by combining various hardware and software.

[1011] Server side

[1012] The server receives the user's request and initializes the virtual reality environment based on the request. Specifically, it generates a VR space and a specific character avatar tailored to the user's preferences. During the initialization process, the synthetic speech generator, conversational dialogue device, and emotion recognition device are activated.

[1013] The server retrieves user data from a database, such as MySQL or MongoDB, and then imports it into the virtual reality environment. The user data includes preferences and settings, and the virtual space is customized based on this information.

[1014] User input is sent to the server in real time, and a conversational dialogue system (e.g., OpenAI's GPT-3) generates an appropriate response to the user's input. This response is then provided as an adaptive response based on the analysis results of an emotion recognition system (e.g., Microsoft Azure Emotion AI or Affectiva).

[1015] Terminal side

[1016] On the device, the user puts on a virtual reality device (e.g., Oculus Rift or HTC Vive) and launches an application. At this time, a connection is established with the server and the virtual reality environment is loaded. On the device, the user interacts with the virtual environment through gestures and voice input.

[1017] The device manages the virtual reality environment based on data received from the server. The user selects services (new contract, model change, device operation guide, etc.) within the virtual reality environment, and the selection is sent to the server in real time. On the device side, the user interface is dynamically managed, and menus and guides are provided according to the user's actions.

[1018] User side

[1019] Users put on a VR device and enter a virtual reality environment to use the service. As a specific example of operation, if a user wishes to sign a new contract, a virtual character will propose a contract plan and provide an explanation using synthesized voice. Furthermore, if the user reacts emotionally (for example, if they show a surprised expression), the emotion recognition device will analyze this and generate an appropriate response such as, "Did I surprise you?"

[1020] Example prompt

[1021] "I want to sign a new smartphone contract."

[1022] Please tell me the features of this model.

[1023] "How much is the monthly fee?"

[1024] This system allows users to experience more natural and personal interactions within a virtual reality environment, and improves user satisfaction by generating responses using generative AI models and prompts.

[1025] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1026] Step 1:

[1027] The server receives the user's request. When the user puts on the virtual reality device and starts the application, the device sends a request containing service details (new contract, model change, device operation instructions, etc.) to the server. Based on this input data, the server prepares to proceed to the next process.

[1028] Step 2:

[1029] The server initializes the virtual reality environment. Specifically, it generates a VR space tailored to the user's desired services and preferences, and prepares a specific character avatar. During this process, it reads the user's preferences and settings from a user database (e.g., MySQL or MongoDB) and reflects them in the virtual reality space. This operation is performed using a VRSDK (e.g., Unity or Unreal Engine).

[1030] Step 3:

[1031] The server initializes a synthetic speech generator, a conversational dialogue device, and an emotion recognition device. The synthetic speech generator (e.g., Google Text-to-Speech API) converts text into speech, the conversational dialogue device (e.g., OpenAI's GPT-3) generates appropriate responses to user input, and the emotion recognition device (e.g., Microsoft Azure Emotion AI or Affectiva) is used to analyze the user's facial expressions and voice data and recognize emotions.

[1032] Step 4:

[1033] The terminal displays the virtual reality environment to the user and manages the user interface. The user selects services within the virtual reality environment through gestures or voice input. The selected service content is transmitted from the terminal to the server. Based on this input data, the server prepares an appropriate response.

[1034] Step 5:

[1035] The server responds to user input in real time. The conversational dialogue device generates a text response based on the user's prompt (e.g., "I want to sign up for a new smartphone contract," "What are the features of this model?", "How much is the monthly fee?"). This text response is converted into speech by a synthetic speech generator and sent to the terminal as voice data.

[1036] Step 6:

[1037] The server generates an adaptive response based on the user's emotional data analyzed by the emotion recognition device. This emotional data is obtained from the user's facial expressions and voice data, and if the user looks surprised, for example, a response such as "Did I surprise you?" is generated. The generated response is converted into voice by a synthetic voice generation device and sent back to the terminal.

[1038] Step 7:

[1039] The device displays adaptive responses to the user and continues the interaction. The user continues to interact with the character in the virtual reality environment using gestures and voice input again. This interaction is repeated and continues until the user terminates the service.

[1040] Step 8:

[1041] The server processes billing when a paid service is used. When the user completes the service use, the terminal displays the billing procedure (e.g., inputting credit card information). When the user completes the payment procedure, the server charges the fee through a payment processing system (e.g., Stripe or PayPal).

[1042] Step 9:

[1043] The user removes the virtual reality device and ends the service. Once the interaction in the virtual reality environment is complete and the user exits the virtual reality shop, they return to the real world. This process allows the user to receive efficient and entertaining services within the virtual reality environment.

[1044] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1046] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1047] [Third embodiment]

[1048] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1049] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1050] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1051] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1052] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1053] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1054] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1055] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1056] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1057] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1058] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1059] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1060] This invention relates to a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using synthetic voice AI and conversational AI.

[1061] Server side

[1062] Request received

[1063] The server receives a request sent by the user. The request includes the user's identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1064] Initializing the virtual reality environment

[1065] The server sets up the virtual reality environment based on the user's requests, including creating a VR space tailored to the user's preferences and creating an avatar for a specified character (e.g., a celebrity or historical figure).

[1066] Initializing the synthetic voice AI and conversation AI

[1067] During the initialization phase, instances of synthetic speech AI and conversation AI are generated, which prepares functions such as converting text to speech and realizing natural dialogue with the user.

[1068] Acquiring and updating user data

[1069] The server retrieves user data from the database and reflects it in the virtual reality environment. User data includes user profile information and past usage history.

[1070] Managing real-time interactions

[1071] The server receives input from the user in real time and generates a response using conversational AI. The response is converted into voice through synthetic speech AI and provided to the user. For example, if the user says, "I'd like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[1072] Billing

[1073] When a paid service such as a terminal operation guide is used, the server executes billing processing to the user's account, allowing the user to easily pay after receiving the service in the virtual reality environment.

[1074] Terminal side

[1075] Launching a VR app

[1076] The user launches the app on their VR device, which initiates a connection with the server and loads the virtual reality environment.

[1077] Connecting to the Server

[1078] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[1079] Service Selection

[1080] Within the virtual reality environment, the user selects the service they wish to use (new contract, model change, terminal operation guide, etc.). Based on the selected service, the server instructs them to take the appropriate action.

[1081] User Interaction

[1082] The terminal manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. The terminal sends input from the user to the server, and receives and displays responses from the server in real time.

[1083] User side

[1084] Wearing a VR device

[1085] Users put on the VR device and prepare to use the service. By equipping the VR headset and handheld controllers, they can enter the virtual reality environment.

[1086] Service Selection

[1087] Users use an interface in the virtual reality environment to select services such as new contracts, model changes, and operation guides, allowing them to efficiently complete the procedures they desire.

[1088] Start a dialogue

[1089] The user interacts with a specific character avatar and receives services. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[1090] Payment process

[1091] For paid services, users complete payment procedures within the virtual reality environment, including entering credit card information and QR code payments.

[1092] VR Shop closed

[1093] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1094] Through these steps, the user can receive efficient and entertaining services within the virtual reality environment. The system of the present invention improves user convenience and solves the problems of the prior art.

[1095] The processing flow will be explained below.

[1096] Server side

[1097] Processing steps from receiving a request to generating a response

[1098] Step 1:

[1099] The server receives a user request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1100] Step 2:

[1101] The server initializes the virtual reality environment based on the request, creating a VR space tailored to the user's preferences.

[1102] Step 3:

[1103] The server initializes the synthetic voice AI and the conversation AI. The synthetic voice AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input.

[1104] Step 4:

[1105] The server retrieves user data from a database, including user profile information and past usage history.

[1106] Step 5:

[1107] The server reflects the user data in the virtual reality environment, allowing the avatar of the character set to provide appropriate services to the user.

[1108] Step 6:

[1109] The server receives input from the user in real time and generates a response using conversational AI, which is then converted into speech using synthetic speech AI and provided to the user.

[1110] Step 7:

[1111] When a paid service such as terminal operation guidance is used, the server executes billing processing, which charges the user's account.

[1112] Terminal side

[1113] Processing steps from app launch to user interface management

[1114] Step 1:

[1115] The user launches the VR app on their device, which initiates a connection with the server.

[1116] Step 2:

[1117] The device connects to the server, sends the user's request, receives the VR environment data returned from the server, and displays the virtual reality environment.

[1118] Step 3:

[1119] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[1120] Step 4:

[1121] The terminal manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[1122] Step 5:

[1123] Users interact with characters in a virtual reality environment through gestures and voice input, and the device transmits this input to a server, which receives and displays responses.

[1124] User side

[1125] Processing steps from device installation to payment procedure

[1126] Step 1:

[1127] The user puts on the VR device and prepares to use the service, which allows them to enter the virtual reality environment.

[1128] Step 2:

[1129] The user selects the desired service using an interface provided within the virtual reality environment.

[1130] Step 3:

[1131] Based on the service selected, the user initiates a dialogue with a specific character avatar, which allows the user to proceed with the required procedure.

[1132] Step 4:

[1133] When using paid services, users can complete payment procedures within the virtual reality environment, using methods such as credit card information or QR code payment.

[1134] Step 5:

[1135] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1136] Example 1

[1137] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1138] Currently, there is no system in existence that allows users to smoothly and efficiently receive guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. In particular, major challenges remain in real-time dialogue, high operability of the user interface, natural communication using synthesized speech and conversational agents, and simple payment procedures.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1140] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice generating means, means for initializing a dialogue agent, means for generating a virtual avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing payments, and means for generating and displaying data for the virtual reality environment. This allows the user to smoothly receive guidance on contracts, model changes, and terminal operations within the VR environment and complete operations and procedures through real-time dialogue.

[1141] The "means for receiving a user request" is a function for receiving an information request sent from a user terminal.

[1142] "Means for initializing a virtual reality environment" is a function that configures and creates a VR space based on the user's requests.

[1143] "Synthetic voice generation means" is a technology for converting text data into voice data.

[1144] The "means for initializing a conversational agent" is a function for preparing an artificial intelligence to have a natural conversation with a user.

[1145] The "means for generating a virtual avatar of a specific character" is a function for creating a 3D model of a specified character in response to a user request.

[1146] "Means of acquiring user data and reflecting it in a virtual reality environment" refers to technology that acquires user information from a database and applies it to a VR environment.

[1147] The "means for managing real-time dialogue with the user" is a function that responds immediately to input from the user and provides an appropriate response.

[1148] The "payment processing means" is a system that charges the user for the service usage and completes the settlement.

[1149] The "means for generating and displaying data for a virtual reality environment" is a function for generating data necessary for a virtual reality environment and displaying it on a user terminal.

[1150] The "means for selecting a service" is an interface that allows a user to select the required service within the virtual reality environment.

[1151] "Means for connecting to a server and obtaining data" refers to a technique by which a user terminal accesses a server and obtains the necessary information.

[1152] A "means of interaction through a user interface" is an interface that a user uses to operate and interact.

[1153] The "means for providing guidance in accordance with the type of service to be used" is a function for providing appropriate instructions and guidance to the user in accordance with the selected service.

[1154] "Means for wearing a virtual reality device and starting a service" refers to the steps a user takes to start using a service using a VR device.

[1155] The "means for exiting the virtual reality shop after service usage has ended" is a function that allows the user to exit the virtual reality environment after finishing using the service.

[1156] "Means for interacting through gestures and voice input" refers to technologies that allow users to communicate with the system using hand movements and voice.

[1157] The present invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using a synthetic speech generation means and a dialogue agent.

[1158] Server side

[1159] Request received

[1160] The server receives requests sent from users using the HTTPS protocol. The request contains the user's identification information (user ID) and the type of service desired (new contract, model change, device operation guide, etc.). The received request is parsed in JSON format and the appropriate processing is carried out.

[1161] Initializing the virtual reality environment

[1162] The server uses Unity to set up a virtual reality environment based on the user's request. This includes generating a VR space tailored to the user's preferences and placing a specific character avatar created using Blender. The virtual reality environment data is sent to the user's device and displayed.

[1163] Initialization of synthetic speech generator and dialogue agent

[1164] The server initializes the synthetic speech generator using Google Cloud's Text-to-Speech API and the conversational agent using Dialogflow, which prepares the text-to-speech conversion function and the ability to have natural conversations with the user.

[1165] Acquiring and updating user data

[1166] The server retrieves user data from the MySQL database and reflects it in the virtual reality environment. The user data includes user profile information and past usage history. This allows the server to provide appropriate services for each individual user.

[1167] Managing real-time interactions

[1168] The server receives voice input from the user and generates an appropriate response using Dialogflow. The generated response is converted into audio using Google Cloud's Text-to-Speech API and provided to the user. For example, if a user types "I would like to sign a new contract," the conversational agent will suggest an appropriate contract plan and explain its details in audio.

[1169] Payment Processing

[1170] The server processes payments using the Stripe API. For example, if a paid service such as a terminal operation guide is used, the server will charge the user's account. This allows the user to easily pay after receiving the service in the virtual reality environment.

[1171] Terminal side

[1172] Launching a VR app

[1173] A user launches an app on a VR device (e.g., Oculus Quest), which initiates a connection with the server and loads the virtual reality environment. VR apps run using the Oculus VR SDK.

[1174] Connecting to the Server

[1175] The device connects to the server and sends the user's request to the server. The device receives the VR environment data returned from the server and displays it. The device connects using Wi-Fi or 5G.

[1176] Service Selection

[1177] Within the virtual reality environment, users select the service they want to use, such as signing up for a new contract, upgrading their device, or receiving device operation instructions. Service selection is performed using an interactive menu within the VR app.

[1178] User Interaction

[1179] The device manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the device's operation procedures.

[1180] User side

[1181] Wearing a VR device

[1182] Users put on a VR device (such as Oculus Quest) and prepare to use the service, which allows them to enter an immersive virtual reality environment.

[1183] Service Selection

[1184] Users can select services such as new contracts, model changes, and operation guidance within the virtual reality environment, allowing them to efficiently complete the procedures they desire.

[1185] Start a dialogue

[1186] Users interact with a specific character avatar to receive services, including obtaining necessary information through natural conversation. For example, a user can begin the process by entering "I would like to sign a new contract."

[1187] Payment process

[1188] When using paid services, users complete payment procedures within the virtual reality environment, including entering credit card information or using a QR code. Payments are processed by the server via the Stripe API.

[1189] VR Shop closed

[1190] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1191] Examples of concrete examples and prompts

[1192] For example, if a user wants to sign up for a new smartphone model, the following steps are taken.

[1193] 1. The user puts on the Oculus Quest and launches the VR app.

[1194] 2. Select "New Contract" within the virtual reality environment.

[1195] 3. A specific character will appear and ask questions about the contract in a dialogue format.

[1196] 4. The server generates the appropriate contract plan through Dialogflow and converts it into audio using Google Cloud's Text-to-Speech API.

[1197] 5. The user follows the character's instructions to complete the contract procedure.

[1198] Prompt Sentence Examples

[1199] "Enter a scenario where a user signs up for a new smartphone:

[1200] 1. First, the user puts on the VR device and launches the VR app.

[1201] 2. Select a service menu in the virtual reality environment

[1202] 3. Check the details of the new contract through dialogue with the character

[1203] 4. Payment Procedure

[1204] 5. Close the VR shop.

[1205] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1206] Step 1: Receiving a request

[1207] The server receives a request from the user via the HTTPS protocol. This request includes the user ID and the details of the desired service (e.g., new contract, model change, terminal operation guide). The input is the user's request data, which is analyzed to extract the necessary information. The output is the analyzed user ID and service content data. Specifically, the server uses Apache HTTP Server to analyze the received JSON data.

[1208] Step 2: Initializing the Virtual Reality Environment

[1209] The server uses Unity to set up a virtual reality environment based on the user's request. The input is the user ID and service content data, and based on this, it configures the VR space and generates a specific character avatar. The output is the generated VR space data. Specifically, it calls the Unity API and places a VR space tailored to the user's preferences and a character avatar generated using Blender.

[1210] Step 3: Initialize the synthetic speech generator and the dialogue agent

[1211] The server initializes the synthetic speech generation means using Google Cloud's Text-to-Speech API and the dialogue agent using Dialogflow. User data and service content data are used as input, and each API is configured based on this. Synthetic speech and dialogue agent instances are obtained as output. Specifically, the server accesses the API key and settings via the Google Cloud Console and generates the required instances.

[1212] Step 4: Obtaining and updating user data

[1213] The server retrieves user data from a MySQL database and reflects it in the virtual reality environment. It executes a database query using the user ID as input, and obtains user profile information and usage history as output. Specifically, it executes a MySQL query to retrieve the necessary data and applies it to the VR environment settings.

[1214] Step 5: Managing real-time interactions

[1215] The server receives voice input from the user and generates an appropriate response using Dialogflow. It then converts the voice input into audio using Google Cloud's Text-to-Speech API and provides it to the user. The input is the user's voice input data, which is analyzed and converted into text, and a response is generated using a conversational agent. The output is audio response data. Specifically, the voice data is temporarily saved to a WAV file, sent to Dialogflow, and the returned text is passed to the synthetic speech API to generate audio.

[1216] Step 6: Payment Processing

[1217] After a paid service is used, the server processes the payment through the Stripe API. The input is the user's payment information and service information, and billing is performed based on this. The output is payment confirmation data. Specifically, the server calls the Stripe API, processes the user's payment information, and completes the payment.

[1218] Step 7: Launch the VR app

[1219] A user launches an app on a VR device such as Oculus Quest. The input is a command to launch the VR device, and the output is to start accessing the VR environment. Specifically, the app launches using the Oculus VR SDK and establishes a network connection.

[1220] Step 8: Connect to the Server

[1221] The device connects to the server and sends the user's request to the server. The input is the user's request data, and the output is the virtual reality environment data. Specifically, the device sends an HTTP request to the server and receives the VR environment data returned from the server.

[1222] Step 9: Select a service

[1223] The user selects the service they want to use in the virtual reality environment. The input is the user's service selection, and the output is instructions based on the selected service returned from the server. Specifically, the user selects the service using an interactive menu within the VR app.

[1224] Step 10: User Interaction

[1225] The device manages interactions with the user. Inputs include user gestures and voice inputs, and output displays response data from the server. Specifically, it uses WebSocket to capture user gestures and voice commands and send them to the server.

[1226] Step 11: Put on the VR device

[1227] The user puts on the VR device and prepares to use the service. The input is a command to put on the VR device, and the output is to start accessing the virtual reality environment. Specifically, the user puts on the Oculus Quest and prepares the handheld controller.

[1228] Step 12: Select a service

[1229] Within the virtual reality environment, the user selects services such as new contracts, model changes, and operation guidance. The input is the user's service selection, and the output is instructions based on the selected service provided by the server. Specific operations involve using a menu provided within the VR environment to select a service that suits the purpose.

[1230] Step 13: Start a conversation

[1231] A user initiates a dialogue with a specific character avatar. The input is the user's voice input, and the dialogue agent generates a response accordingly. The output is a voice response from the dialogue agent. Specifically, the user enters a voice command, which is processed by the dialogue agent to generate a response.

[1232] Step 14: Payment Process

[1233] When using a paid service, the user completes the payment process within the virtual reality environment. The input is the user's payment information, and the output is payment confirmation data. Specifically, the user enters their credit card information within the virtual reality environment and completes the payment via the Stripe API.

[1234] Step 15: Finish the VR Shop

[1235] After completing the service, the user removes the VR device and exits the virtual reality shop. The input is the user's exit command, and the output is the ability to return to the real world. The specific actions are to close the VR app and remove the Oculus Quest.

[1236] (Application example 1)

[1237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1238] Services provided within conventional virtual reality environments have not fully satisfied user convenience. In particular, it has been difficult to provide information on new contracts, model upgrades, and device operation methods quickly and effectively within a virtual reality environment. Furthermore, user interaction has been limited, and there has been a lack of means for real-time dialogue, resulting in a poor user experience. Furthermore, there has been a demand for a method that allows more users to easily access services by using virtual reality devices connected to smartphones.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1240] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice artificial intelligence, means for initializing a conversational artificial intelligence, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for realizing a dialogue system for providing guidance on new contracts, model changes, and device operation within the virtual reality environment, and means for the user to interact using a virtual reality headset and controller connected to a smartphone. This allows the user to easily and quickly receive more convenient and effective guidance on new contracts, model changes, and device operation within the virtual reality environment.

[1241] 1. "Means for receiving user requests" refers to the methods and functions by which the system receives operations and requests from users.

[1242] 2. "Means for initializing a virtual reality environment" means methods and functionality for configuring and building a user-accessible virtual reality environment.

[1243] 3. "Means for initializing a synthetic speech artificial intelligence" means methods and functions for configuring and activating an artificial intelligence that converts text data into speech data.

[1244] 4. "Means for initializing conversational AI" refers to methods and functions for configuring and launching an AI that communicates in natural language.

[1245] 5. "Means for generating an avatar of a specific character" means methods and functions for displaying a specific character within a virtual reality environment based on a user's request.

[1246] 6. "Means for acquiring user data and reflecting it in the virtual reality environment" refers to the methods and functions for acquiring user information from the database and applying it within the virtual reality environment.

[1247] 7. "Means for managing real-time interactions with users" refers to methods and functions that enable users and systems to interact with each other in real time.

[1248] 8. "Means for billing processing" refers to the methods and functions for managing and processing payment procedures when a user uses a paid service.

[1249] 9. "Means for realizing an interactive system for providing guidance on new contracts, model changes, and device operation within a virtual reality environment" refers to methods and functions that provide users with interactive guidance on new contracts, model changes, and device operation within a virtual reality environment.

[1250] 10. "Means for user interaction using a virtual reality headset and controllers connected to a smartphone" means methods and functionality that allow a user to navigate and interact within a virtual reality environment using a virtual reality headset and controllers connected to a smartphone.

[1251] The present invention is a system that allows users to receive service information such as new contracts, model changes, and terminal operation methods within a virtual reality environment. This system is composed of three parties: a server, a terminal, and a user, and uses detailed hardware and software to fulfill each role.

[1252] Server-side implementation

[1253] The server includes the following means:

[1254] 1. Means of receiving a user request:

[1255] The server receives a request sent by the user, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1256] 2. Means of initializing the virtual reality environment:

[1257] Based on the user's request, the server initializes the virtual reality environment using a VR toolkit (e.g., Unity or Unreal Engine), including creating a VR space tailored to the user's preferences and generating an avatar for a specific character.

[1258] 3. Methods for initializing the synthetic voice AI:

[1259] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly) that converts text data into voice data.

[1260] 4. Methods for initializing conversational AI:

[1261] The server initializes a conversational AI (e.g., Dialogflow or Microsoft Bot Framework) to enable natural dialogue with the user.

[1262] 5. How user data is acquired and reflected in the virtual reality environment:

[1263] The server retrieves the user's profile information and past usage history from the database and reflects this in the virtual reality environment.

[1264] 6. Means of managing real-time user interactions:

[1265] The server receives voice input from the user in real time and uses conversational AI to generate a response, which is then converted into speech by synthetic speech AI and provided to the user.

[1266] 7. Means of processing claims:

[1267] When a paid service such as terminal operation guidance is used, the server processes the billing to the user's account, allowing the user to easily make payment after receiving the service.

[1268] Terminal side embodiment

[1269] The terminal includes the following means:

[1270] 1. Launch the VR app:

[1271] The user puts on a VR headset and launches a VR application on their smartphone, which initiates a connection with the server.

[1272] 2. Connect to the server:

[1273] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[1274] 3. Service Selection:

[1275] The user uses an interface within the virtual reality environment to select services such as new contracts, model changes, and terminal operation guidance.

[1276] 4. User Interaction:

[1277] The terminal manages interactions with the user, who interacts with specific characters through gestures and voice input to receive necessary procedures and instructions.

[1278] User-Side Embodiment

[1279] The user includes the following means:

[1280] 1. Wearing the VR device:

[1281] The user prepares to enter the virtual reality environment by putting on a VR headset and preparing the handheld controllers.

[1282] 2. Service Selection:

[1283] The user operates the interface provided within the virtual reality environment to select services such as new contracts, model changes, and operation guidance.

[1284] 3. Start a conversation:

[1285] The user can start a conversation with a specific character and receive the necessary information or procedures. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[1286] 4. Payment Procedure:

[1287] When using a paid service, users can complete payment procedures within the virtual reality environment by entering credit card information or using a QR code.

[1288] 5. VR Shop Closure:

[1289] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1290] Specific examples

[1291] For example, if a user wishes to sign a new contract, they can say "I would like to sign a new contract" by voice within the VR environment, and a character avatar will respond by saying, "Here are our new contract plans. Which plan would you like?" In this way, each time the user asks a question, the conversational AI generates an appropriate response and replies audibly through the synthetic voice AI.

[1292] Prompt Sentence Examples

[1293] Describe a dialogue system with an AI character that realizes a scenario in which a user wishes to make a new contract in a VR environment. Explain in detail the response and behavior of the character avatar when the user says, "I want to make a new contract."

[1294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1295] Step 1:

[1296] The server receives the user's request, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.). Based on this input, the server analyzes the request and begins preparing an appropriate virtual reality environment.

[1297] Step 2:

[1298] The server initializes the virtual reality environment. This is the process of generating a VR space tailored to the user's preferences using a VR toolkit (e.g., Unity or Unreal Engine) based on the user's request. An avatar for the specified character is also generated and configured. As a result, a virtual reality environment tailored to the user is constructed.

[1299] Step 3:

[1300] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly). At this stage, an instance of the synthetic voice AI is created and prepared to convert text data into speech data. This allows speech output for subsequent user interactions.

[1301] Step 4:

[1302] The server initializes the conversational AI (e.g., Dialogflow or Microsoft Bot Framework), creates an instance of the conversational AI, and configures it to enable natural dialogue with the user. This configuration includes preparing a model to generate appropriate responses to user input.

[1303] Step 5:

[1304] The server retrieves user data from the database and reflects it in the virtual reality environment. The user data includes profile information and usage history, and personalizes the VR space based on that information. This information is used to improve the user experience.

[1305] Step 6:

[1306] The server manages real-time dialogue with the user. In this step, the user's voice input is converted and analyzed by conversational AI. Based on the analysis results, a response is generated through synthetic voice AI and provided to the user. This process is carried out in real time, ensuring smooth interaction between the user and the system.

[1307] Step 7:

[1308] The device launches the VR app and establishes a connection with the server, allowing the user to access the virtual reality environment and receive data sent from the server, enabling the user to view the virtual reality environment and operate the interface.

[1309] Step 8:

[1310] The device provides an interface for the user to select services. The user uses a VR headset and handheld controller to select services such as new contracts, model changes, and device operation guidance. This input causes the server to prepare the next response.

[1311] Step 9:

[1312] The user operates the interface and begins a dialogue with a specific character. For example, if the user voice-inputs, "I'd like to sign a new contract," the character will respond with, "Here are our new contract plans. Which plan would you like?" This dialogue progresses in real time, with responses generated by conversational AI.

[1313] Step 10:

[1314] When a user uses a paid service, the server handles the billing process. Specifically, the charge is sent to the user's account via credit card information entry or QR code payment, allowing the user to complete payment easily.

[1315] Step 11:

[1316] After completing their use of the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world and re-prepare the device for their next use.

[1317] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1318] This invention relates to a system that provides users with guidance on new contracts, model changes, and terminal operation methods in a virtual reality environment, while also recognizing the user's emotions and generating adaptive responses based on them. This system receives user requests, initializes the virtual reality environment, and interacts with the user using a synthetic voice AI, a conversational AI, and an emotion engine.

[1319] Server side

[1320] Processing from receiving a request to generating a response

[1321] The server receives the user's request. This request includes the user ID and the type of service desired (new contract, model change, device operation guide, etc.). The server then initializes the virtual reality environment based on the request. This includes generating a VR space tailored to the user's preferences and creating an avatar for the specified character.

[1322] Initialize instances of the synthetic speech AI and conversation AI. The synthetic speech AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input. The emotion engine is also initialized. The emotion engine analyzes the user's facial expressions and voice and recognizes emotions.

[1323] User data is retrieved from the database and reflected in the virtual reality environment, allowing the character avatar to provide appropriate services to the user.

[1324] The server receives input from the user in real time and generates a response using conversational AI. The generated response is converted into voice using synthetic voice AI and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[1325] The emotion engine analyzes the user's facial expressions and voice to detect emotions. Based on the detected emotions, the conversational AI generates a response that is adapted to the user. For example, if the user expresses surprise, the conversational AI will generate a response such as, "Did I surprise you?"

[1326] When a paid service such as a terminal operation guide is used, a billing process is executed, and the fee is charged to the user's account.

[1327] Terminal side

[1328] Processing from app launch to user interface management

[1329] The user launches the VR app on their device, which initiates a connection with the server and loads the virtual reality environment. The device then connects to the server and sends the user's request. The server returns the VR environment data, which is then displayed.

[1330] The user selects the services they wish to use within the virtual reality environment (new contract, model change, device operation guide, etc.). The selection is sent to the server. The device manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[1331] The user interacts with the character in the virtual reality environment through gestures and voice input. The device sends this input to the server, receives responses, and displays them. The emotion engine analyzes the user's facial expressions and voice and sends emotional data to the server.

[1332] User side

[1333] Processing from device installation to payment procedures

[1334] The user puts on the VR device and prepares to use the service. This allows the user to enter the virtual reality environment. The user selects the desired service using the interface provided within the virtual reality environment.

[1335] Based on the service selected, the user can start a conversation with a specific character avatar, which will then guide them through the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the operation procedure.

[1336] When using paid services, users make payments within the virtual reality environment. Payment methods include credit card entry and QR code payment. After completing the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1337] Through these steps, users can enjoy efficient and entertaining services within the virtual reality environment. Furthermore, the emotion engine recognizes the user's emotions and provides adaptive responses, further enhancing the user experience. The system of the present invention improves user convenience and solves the problems of the past.

[1338] The processing flow will be explained below.

[1339] Server side

[1340] Processing steps from receiving a request to generating a response

[1341] Step 1:

[1342] The server receives the user's request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1343] Step 2:

[1344] The server initializes the virtual reality environment based on the request, generates a VR space according to the user's preferences, and prepares to generate an avatar for the specified character.

[1345] Step 3:

[1346] The server initializes instances of a synthetic speech AI and a conversational AI. The synthetic speech AI has the function of converting text to speech, and the conversational AI generates appropriate responses to user input.

[1347] Step 4:

[1348] The server retrieves user data from a database, including profile information and past usage history.

[1349] Step 5:

[1350] The server reflects the acquired user data in the virtual reality environment and applies the user's individual settings.

[1351] Step 6:

[1352] The server initiates a real-time dialogue with the user using conversational AI, taking the user's input and generating the optimal response.

[1353] Step 7:

[1354] The server uses synthetic voice AI to convert the generated response into speech, which is then provided to the user.

[1355] Step 8:

[1356] The server analyzes the user's facial expressions and voice through an emotion engine to recognize their emotions.

[1357] Step 9:

[1358] The server adjusts the conversational AI's responses based on the emotions it recognizes, for example, providing a follow-up message such as "Don't worry" if the user expresses anxiety.

[1359] Step 10:

[1360] The server performs billing when terminal operation guidance or other paid services are used, and charges are applied to the user's account.

[1361] Terminal side

[1362] Processing steps from app launch to user interface management

[1363] Step 1:

[1364] The user launches the VR app on their device, which initiates a connection with the server.

[1365] Step 2:

[1366] The device connects to the server and sends the user's request, which includes information such as the user ID and the type of service.

[1367] Step 3:

[1368] The device receives the VR environment data returned from the server and displays the virtual reality environment.

[1369] Step 4:

[1370] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[1371] Step 5:

[1372] The terminal manages the user interface within the virtual reality environment, displaying menus and handling interactions.

[1373] Step 6:

[1374] The emotion engine analyzes the user's facial expressions and voice, and sends the emotional data to the server.

[1375] Step 7:

[1376] The device receives responses from the server and displays them both audibly and visually, and adaptive responses are provided based on data from the emotion engine.

[1377] User side

[1378] Processing steps from device installation to payment procedure

[1379] Step 1:

[1380] The user puts on the VR device and prepares to use the service, allowing them to enter the virtual reality environment.

[1381] Step 2:

[1382] The user uses an interface provided within the virtual reality environment to select the service they want, for example, to sign up for a new contract or upgrade their device.

[1383] Step 3:

[1384] Based on the service selected, the user begins a conversation with a specific character avatar, who guides the user through specific procedures.

[1385] Step 4:

[1386] The user follows the character's instructions as they interact with the character, while the emotion engine recognizes the user's emotions and adjusts responses accordingly.

[1387] Step 5:

[1388] When using paid services, users can complete payment procedures within the virtual reality environment, such as entering credit card information or using a QR code.

[1389] Step 6:

[1390] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1391] Example 2

[1392] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1393] Conventional virtual reality environments have difficulty understanding user emotions and providing adaptive responses. They also lack real-time dialogue management and services tailored to user needs. This can limit the user experience and reduce user satisfaction. Another issue is the complicated billing and payment procedures for paid services, which places a burden on users.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1395] In this invention, the server includes a means for analyzing a user's facial expressions and voice to recognize emotions, a means for generating an adaptive response based on the emotion recognition, and a means for performing billing processing. This makes it possible to understand a user's emotions in real time and provide an adaptive response based on the emotion. It also realizes appropriate billing processing and payment procedures, improving the user experience and convenience.

[1396] The "means for receiving a user request" refers to a function or module for acquiring a request from a user.

[1397] A "means for initializing a virtual reality environment" is a system or program for setting up and starting up a virtual environment.

[1398] The "means for initializing the synthetic speech generation means" is a function for starting and setting up a speech generation system that has the function of converting text data into speech data.

[1399] The "means for initializing a conversation generation model" is a function for starting and setting up a conversation model that generates an appropriate response to a user's input.

[1400] A "means for generating an avatar of a specific character" is a system or program that generates the appearance and behavior of a virtual character based on the user's requests and settings.

[1401] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring user information from a database, etc., and applying it to the virtual reality space.

[1402] "Means for managing real-time user interactions" refers to functions and modules that monitor and control real-time communication between users and the system.

[1403] The "means for analyzing the user's facial expressions and voice and recognizing emotions" is an analysis system for detecting the user's emotional state from facial expression and voice data.

[1404] The "means for generating adaptive responses based on emotion recognition" is a function or module that generates an appropriate response that matches the recognized emotional state of the user.

[1405] The "means for billing" refers to a system or program that calculates fees for the use of paid services and bills the user account.

[1406] "Means for interaction through a user interface" refers to functions or modules that provide an interface that users can visually operate and communicate through it.

[1407] "Means for wearing a virtual reality device and starting the service" refers to the procedures and modules for starting to use the system after wearing the VR device.

[1408] "Means for selecting services within a virtual reality environment" refers to an interface or system that allows users to select the desired service from the various services provided within the virtual space.

[1409] The "means for starting a dialogue with a character" refers to a function or module for starting a conversation with a generated virtual character and advancing the dialogue.

[1410] "Means for carrying out payment procedures when using a paid service" refers to a system or module for completing payment when a user selects a paid service.

[1411] "Means for exiting the virtual reality shop after using the service" refers to the procedures and functions that allow users to safely exit the virtual reality environment after they have finished using the service.

[1412] This invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and also has the function of recognizing the user's emotions and generating adaptive responses. An embodiment of this system will be described in detail below.

[1413] To receive a user request, the server acquires the user ID and desired service details via an HTTP request or WebSocket. This request includes the type of service, such as a new contract, a model change, or device operation instructions.

[1414] The server then initializes the virtual reality environment, which involves using the Unreal Engine to generate a VR space tailored to the user's preferences and an avatar for the specified character.

[1415] After that, the server initializes the synthetic speech generator (Google Cloud Text-to-Speech) and the conversation generation model (OpenAI GPT-3), as well as the emotion engine (Microsoft Azure Emotion API) that analyzes the user's facial expressions and voice to recognize emotions.

[1416] The server retrieves user data from the database and uses this information to reflect in the virtual reality environment, a process that allows the character avatar to provide appropriate services to the user.

[1417] For real-time dialogue management, the server receives user input and generates an appropriate response using a conversation generation model. The generated response is converted into speech using a synthetic speech generation means and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversation generation model will suggest an appropriate contract plan and explain its details in voice.

[1418] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Based on the recognized emotions, the conversation generation model generates adaptive responses. For example, if the user expresses surprise, the response generated will be, "Did I surprise you?"

[1419] If a paid service is used, the server executes a billing process and charges the fee to the user's account, completing the use of the service.

[1420] On the device side, the user launches a VR app and connects to the server using HTTP or WebSocket communication. The device retrieves the virtual reality environment data returned from the server and displays it using Unity. Within the virtual reality environment, the user selects the service they wish to use and sends that selection to the server. The device also manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[1421] Finally, the user puts on the VR device, enters the virtual reality environment, and selects a service using the provided interface. They then begin interacting with a specific character avatar within the virtual reality environment and carry out the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the steps. When using a paid service, the user completes the payment procedure within the virtual reality environment, and once the service is complete, the user removes the VR device and exits the virtual reality shop to return to the real world.

[1422] As a concrete example, the following prompt sentences can be used by the conversation generation model to smoothly engage the user:

[1423] User: I'd like to sign up. What plans are available?

[1424] System: There are several plans available for new subscribers: Basic Plan, Premium Plan, and Family Plan. Want to learn more about each plan?

[1425] In this way, the embodiments of the invention involve the use of specific hardware and software, and include detailed data processing and data calculation processing, making it possible to provide users with efficient and entertaining services.

[1426] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1427] The flow of this system's program processing

[1428] Server-side processing

[1429] Step 1: Receiving a user request

[1430] How it works: The server receives the user's ID and desired service via an HTTP request or WebSocket.

[1431] Input: User ID, desired service (new contract, model change, terminal operation guide, etc.)

[1432] Output: Received data (user ID, service details)

[1433] Step 2: Initializing the Virtual Reality Environment

[1434] How it works: The server generates a VR space using Unreal Engine based on the received user data and generates an avatar for a specific character.

[1435] Input: User data, service details

[1436] Output: Initialized VR space, character avatar

[1437] Step 3: Initialize the synthetic speech generator and the speech generation model

[1438] How it works: The server initializes Google Cloud Text-to-Speech and OpenAI GPT-3.

[1439] Input: None (at system startup)

[1440] Output: Initialized synthetic speech generator, speech generation model

[1441] Step 4: Obtaining and updating user data

[1442] Operation: The server retrieves user data from the database and reflects it in the virtual reality environment.

[1443] Input: User ID

[1444] Output: Updated VR environment

[1445] Step 5: Receiving user input and generating a response

[1446] How it works: The server receives voice or text input from the user, uses a speech generation model to generate an appropriate response, and converts the response into audio using Google Cloud Text-to-Speech.

[1447] Input: User input (voice or text)

[1448] Output: Response voice

[1449] Step 6: Emotion recognition and adaptive response generation

[1450] How it works: The server uses the Microsoft Azure Emotion API to analyze the user's facial expressions and voice to recognize emotions, and then uses a conversation generation model to generate adaptive responses.

[1451] Input: User's facial expression data, voice data

[1452] Output: Adaptive response

[1453] Step 7: Billing for paid services

[1454] How it works: The server calculates the fee when a paid service is used and bills the user's account.

[1455] Input: Service information used

[1456] Output: Billing information, updated user account

[1457] Terminal side processing

[1458] Step 1: Launch the VR app

[1459] How it works: The user launches the VR app on their device and connects to the server.

[1460] Input: VR app launch command

[1461] Output: Server connection started

[1462] Step 2: Acquiring and displaying data in a VR environment

[1463] How it works: The device retrieves VR environment data from the server and displays the virtual reality environment using Unity.

[1464] Input: VR data from the server

[1465] Output: The displayed VR environment

[1466] Step 3: Select service and submit

[1467] Operation: The user selects the service to use within the virtual reality environment, and the device sends the selection to the server.

[1468] Input: Service selection (user input)

[1469] Output: Selected service details (sent from server)

[1470] Step 4: Managing the User Interface

[1471] Operation: The terminal manages the user interface within the virtual reality environment, displaying the necessary menus and handling interactions.

[1472] Input: User interaction

[1473] Output: Updated user interface

[1474] Step 5: Sending User Input and Displaying Responses

[1475] How it works: The user's voice or gesture input is sent to the server, and the response from the server is displayed.

[1476] Input: User input (voice and gestures)

[1477] Output: Response from the server (visual and audio output)

[1478] Step 6: Sending Emotion Data

[1479] How it works: The emotion engine analyzes the user's facial expressions and voice and sends the data to the server.

[1480] Input: User's facial expression data, voice data

[1481] Output: Emotion data (sent to server)

[1482] User processing

[1483] Step 1: Put on your VR device

[1484] Action: The user puts on the VR device and prepares to use the service.

[1485] Input: None (wearing operation)

[1486] Output: VR device is installed

[1487] Step 2: Select a service

[1488] Operation: The user selects the desired service using the interface provided within the virtual reality environment.

[1489] Input: Service selection by user

[1490] Output: Selected service content

[1491] Step 3: Interact with the character

[1492] Action: The user initiates a dialogue with a specific character avatar and proceeds with the required procedures.

[1493] Input: User instruction to start a dialogue

[1494] Output: Started conversation

[1495] Step 4: Complete the payment process

[1496] Operation: When a user uses a paid service, the payment process is carried out within the VR environment.

[1497] Input: Payment information (credit card information or QR code)

[1498] Output: Completed payment

[1499] Step 5: Completing the service and returning to the real world

[1500] Operation: After completing the service, the user removes the VR device and exits the virtual reality shop to return to the real world.

[1501] Input: None (removal operation)

[1502] Output: Return to the real world

[1503] (Application example 2)

[1504] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1505] The problem to be solved by this invention is to provide a system that improves the user experience in a virtual reality environment and generates responses that adapt to the user's emotions. Conventional systems lack the ability to analyze the user's facial expressions and voice in real time and provide adaptive responses based on the analysis, which can lead to reduced user satisfaction. Furthermore, the means for processing prompts using generative AI models and converting them into synthetic speech are insufficient, resulting in a lack of naturalness in the interaction.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1507] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic speech generation device, means for initializing a conversational dialogue device, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for initializing an emotion recognition device, means for analyzing the user's facial expressions and voice data and generating a response based on the detected emotion, means for generating a response according to a prompt sentence using a generative AI model, and means for converting the prompt sentence into synthetic speech. This makes interaction with the user more natural and personal, thereby improving user satisfaction.

[1508] "Means for receiving user requests" refers to a function for receiving service requests and operation instructions sent by users and processing them within the system.

[1509] The "means for initializing a virtual reality environment" is a function for constructing and starting a virtual reality space based on a user's request.

[1510] The "means for initializing the synthetic speech generation device" is a function for preparing the synthetic speech technology that converts text data into natural speech and setting it into an operational state.

[1511] The "means for initializing the conversation dialogue device" is a function for constructing a conversation engine for realizing a dialogue with a user and for starting response generation.

[1512] The "means for generating an avatar of a specific character" is a function for generating an avatar to be displayed in a virtual reality space based on a user request or user data.

[1513] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring individual user data from a database and customizing the virtual reality environment based on that information.

[1514] The "means for managing real-time interactions with the user" is a function for processing input from the user in real time and providing an appropriate response.

[1515] The "means for performing billing processing" is a function for carrying out payment procedures associated with the use of paid services.

[1516] The "means for initializing the emotion recognition device" is a function for analyzing the user's voice and facial expression data and setting the engine for recognizing the emotion into an operational state.

[1517] "Means for analyzing the user's facial expression and voice data and generating a response based on the detected emotion" is a function for analyzing the emotion from the user's facial expression and voice and generating an adaptive response according to that emotion.

[1518] "Means for generating a response in accordance with a prompt sentence using a generative AI model" is a function that uses a generative AI model to generate an appropriate response in accordance with a prompt sentence from a user.

[1519] The "means for converting a prompt sentence into synthetic speech" is a function for converting text data generated as a prompt sentence into speech and providing it to the user.

[1520] This invention is a system for providing users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment, and its operation is characterized by generating adaptive responses based on the user's emotions. This system is composed of both a server and a terminal, and is realized by combining various hardware and software.

[1521] Server side

[1522] The server receives the user's request and initializes the virtual reality environment based on the request. Specifically, it generates a VR space and a specific character avatar tailored to the user's preferences. During the initialization process, the synthetic speech generator, conversational dialogue device, and emotion recognition device are activated.

[1523] The server retrieves user data from a database, such as MySQL or MongoDB, and then imports it into the virtual reality environment. The user data includes preferences and settings, and the virtual space is customized based on this information.

[1524] User input is sent to the server in real time, and a conversational dialogue system (e.g., OpenAI's GPT-3) generates an appropriate response to the user's input. This response is then provided as an adaptive response based on the analysis results of an emotion recognition system (e.g., Microsoft Azure Emotion AI or Affectiva).

[1525] Terminal side

[1526] On the device, the user puts on a virtual reality device (e.g., Oculus Rift or HTC Vive) and launches an application. At this time, a connection is established with the server and the virtual reality environment is loaded. On the device, the user interacts with the virtual environment through gestures and voice input.

[1527] The device manages the virtual reality environment based on data received from the server. The user selects services (new contract, model change, device operation guide, etc.) within the virtual reality environment, and the selection is sent to the server in real time. On the device side, the user interface is dynamically managed, and menus and guides are provided according to the user's actions.

[1528] User side

[1529] Users put on a VR device and enter a virtual reality environment to use the service. As a specific example of operation, if a user wishes to sign a new contract, a virtual character will propose a contract plan and provide an explanation using synthesized voice. Furthermore, if the user reacts emotionally (for example, if they show a surprised expression), the emotion recognition device will analyze this and generate an appropriate response such as, "Did I surprise you?"

[1530] Example prompt

[1531] "I want to sign a new smartphone contract."

[1532] Please tell me the features of this model.

[1533] "How much is the monthly fee?"

[1534] This system allows users to experience more natural and personal interactions within a virtual reality environment, and improves user satisfaction by generating responses using generative AI models and prompts.

[1535] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1536] Step 1:

[1537] The server receives the user's request. When the user puts on the virtual reality device and starts the application, the device sends a request containing service details (new contract, model change, device operation instructions, etc.) to the server. Based on this input data, the server prepares to proceed to the next process.

[1538] Step 2:

[1539] The server initializes the virtual reality environment. Specifically, it generates a VR space tailored to the user's desired services and preferences, and prepares a specific character avatar. During this process, it reads the user's preferences and settings from a user database (e.g., MySQL or MongoDB) and reflects them in the virtual reality space. This operation is performed using a VRSDK (e.g., Unity or Unreal Engine).

[1540] Step 3:

[1541] The server initializes a synthetic speech generator, a conversational dialogue device, and an emotion recognition device. The synthetic speech generator (e.g., Google Text-to-Speech API) converts text into speech, the conversational dialogue device (e.g., OpenAI's GPT-3) generates appropriate responses to user input, and the emotion recognition device (e.g., Microsoft Azure Emotion AI or Affectiva) is used to analyze the user's facial expressions and voice data and recognize emotions.

[1542] Step 4:

[1543] The terminal displays the virtual reality environment to the user and manages the user interface. The user selects services within the virtual reality environment through gestures or voice input. The selected service content is transmitted from the terminal to the server. Based on this input data, the server prepares an appropriate response.

[1544] Step 5:

[1545] The server responds to user input in real time. The conversational dialogue device generates a text response based on the user's prompt (e.g., "I want to sign up for a new smartphone contract," "What are the features of this model?", "How much is the monthly fee?"). This text response is converted into speech by a synthetic speech generator and sent to the terminal as voice data.

[1546] Step 6:

[1547] The server generates an adaptive response based on the user's emotional data analyzed by the emotion recognition device. This emotional data is obtained from the user's facial expressions and voice data, and if the user looks surprised, for example, a response such as "Did I surprise you?" is generated. The generated response is converted into voice by a synthetic voice generation device and sent back to the terminal.

[1548] Step 7:

[1549] The device displays adaptive responses to the user and continues the interaction. The user continues to interact with the character in the virtual reality environment using gestures and voice input again. This interaction is repeated and continues until the user terminates the service.

[1550] Step 8:

[1551] The server processes billing when a paid service is used. When the user completes the service use, the terminal displays the billing procedure (e.g., inputting credit card information). When the user completes the payment procedure, the server charges the fee through a payment processing system (e.g., Stripe or PayPal).

[1552] Step 9:

[1553] The user removes the virtual reality device and ends the service. Once the interaction in the virtual reality environment is complete and the user exits the virtual reality shop, they return to the real world. This process allows the user to receive efficient and entertaining services within the virtual reality environment.

[1554] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1556] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1557] [Fourth embodiment]

[1558] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1559] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1560] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1561] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1562] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1564] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1565] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1566] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1569] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1570] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1571] This invention relates to a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using synthetic voice AI and conversational AI.

[1572] Server side

[1573] Request received

[1574] The server receives a request sent by the user. The request includes the user's identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1575] Initializing the virtual reality environment

[1576] The server sets up the virtual reality environment based on the user's requests, including creating a VR space tailored to the user's preferences and creating an avatar for a specified character (e.g., a celebrity or historical figure).

[1577] Initializing the synthetic voice AI and conversation AI

[1578] During the initialization phase, instances of synthetic speech AI and conversation AI are generated, which prepares functions such as converting text to speech and realizing natural dialogue with the user.

[1579] Acquiring and updating user data

[1580] The server retrieves user data from the database and reflects it in the virtual reality environment. User data includes user profile information and past usage history.

[1581] Managing real-time interactions

[1582] The server receives input from the user in real time and generates a response using conversational AI. The response is converted into voice through synthetic speech AI and provided to the user. For example, if the user says, "I'd like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[1583] Billing

[1584] When a paid service such as a terminal operation guide is used, the server executes billing processing to the user's account, allowing the user to easily pay after receiving the service in the virtual reality environment.

[1585] Terminal side

[1586] Launching a VR app

[1587] The user launches the app on their VR device, which initiates a connection with the server and loads the virtual reality environment.

[1588] Connecting to the Server

[1589] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[1590] Service Selection

[1591] Within the virtual reality environment, the user selects the service they wish to use (new contract, model change, terminal operation guide, etc.). Based on the selected service, the server instructs them to take the appropriate action.

[1592] User Interaction

[1593] The terminal manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. The terminal sends input from the user to the server, and receives and displays responses from the server in real time.

[1594] User side

[1595] Wearing a VR device

[1596] Users put on the VR device and prepare to use the service. By equipping the VR headset and handheld controllers, they can enter the virtual reality environment.

[1597] Service Selection

[1598] Users use an interface in the virtual reality environment to select services such as new contracts, model changes, and operation guides, allowing them to efficiently complete the procedures they desire.

[1599] Start a dialogue

[1600] The user interacts with a specific character avatar and receives services. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[1601] Payment process

[1602] For paid services, users complete payment procedures within the virtual reality environment, including entering credit card information and QR code payments.

[1603] VR Shop closed

[1604] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1605] Through these steps, the user can receive efficient and entertaining services within the virtual reality environment. The system of the present invention improves user convenience and solves the problems of the prior art.

[1606] The processing flow will be explained below.

[1607] Server side

[1608] Processing steps from receiving a request to generating a response

[1609] Step 1:

[1610] The server receives a user request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1611] Step 2:

[1612] The server initializes the virtual reality environment based on the request, creating a VR space tailored to the user's preferences.

[1613] Step 3:

[1614] The server initializes the synthetic voice AI and the conversation AI. The synthetic voice AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input.

[1615] Step 4:

[1616] The server retrieves user data from a database, including user profile information and past usage history.

[1617] Step 5:

[1618] The server reflects the user data in the virtual reality environment, allowing the avatar of the character set to provide appropriate services to the user.

[1619] Step 6:

[1620] The server receives input from the user in real time and generates a response using conversational AI, which is then converted into speech using synthetic speech AI and provided to the user.

[1621] Step 7:

[1622] When a paid service such as terminal operation guidance is used, the server executes billing processing, which charges the user's account.

[1623] Terminal side

[1624] Processing steps from app launch to user interface management

[1625] Step 1:

[1626] The user launches the VR app on their device, which initiates a connection with the server.

[1627] Step 2:

[1628] The device connects to the server, sends the user's request, receives the VR environment data returned from the server, and displays the virtual reality environment.

[1629] Step 3:

[1630] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[1631] Step 4:

[1632] The terminal manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[1633] Step 5:

[1634] Users interact with characters in a virtual reality environment through gestures and voice input, and the device transmits this input to a server, which receives and displays responses.

[1635] User side

[1636] Processing steps from device installation to payment procedure

[1637] Step 1:

[1638] The user puts on the VR device and prepares to use the service, which allows them to enter the virtual reality environment.

[1639] Step 2:

[1640] The user selects the desired service using an interface provided within the virtual reality environment.

[1641] Step 3:

[1642] Based on the service selected, the user initiates a dialogue with a specific character avatar, which allows the user to proceed with the required procedure.

[1643] Step 4:

[1644] When using paid services, users can complete payment procedures within the virtual reality environment, using methods such as credit card information or QR code payment.

[1645] Step 5:

[1646] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1647] Example 1

[1648] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1649] Currently, there is no system in existence that allows users to smoothly and efficiently receive guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. In particular, major challenges remain in real-time dialogue, high operability of the user interface, natural communication using synthesized speech and conversational agents, and simple payment procedures.

[1650] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1651] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice generating means, means for initializing a dialogue agent, means for generating a virtual avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing payments, and means for generating and displaying data for the virtual reality environment. This allows the user to smoothly receive guidance on contracts, model changes, and terminal operations within the VR environment and complete operations and procedures through real-time dialogue.

[1652] The "means for receiving a user request" is a function for receiving an information request sent from a user terminal.

[1653] "Means for initializing a virtual reality environment" is a function that configures and creates a VR space based on the user's requests.

[1654] "Synthetic voice generation means" is a technology for converting text data into voice data.

[1655] The "means for initializing a conversational agent" is a function for preparing an artificial intelligence to have a natural conversation with a user.

[1656] The "means for generating a virtual avatar of a specific character" is a function for creating a 3D model of a specified character in response to a user request.

[1657] "Means of acquiring user data and reflecting it in a virtual reality environment" refers to technology that acquires user information from a database and applies it to a VR environment.

[1658] The "means for managing real-time dialogue with the user" is a function that responds immediately to input from the user and provides an appropriate response.

[1659] The "payment processing means" is a system that charges the user for the service usage and completes the settlement.

[1660] The "means for generating and displaying data for a virtual reality environment" is a function for generating data necessary for a virtual reality environment and displaying it on a user terminal.

[1661] The "means for selecting a service" is an interface that allows a user to select the required service within the virtual reality environment.

[1662] "Means for connecting to a server and obtaining data" refers to a technique by which a user terminal accesses a server and obtains the necessary information.

[1663] A "means of interaction through a user interface" is an interface that a user uses to operate and interact.

[1664] The "means for providing guidance in accordance with the type of service to be used" is a function for providing appropriate instructions and guidance to the user in accordance with the selected service.

[1665] "Means for wearing a virtual reality device and starting a service" refers to the steps a user takes to start using a service using a VR device.

[1666] The "means for exiting the virtual reality shop after service usage has ended" is a function that allows the user to exit the virtual reality environment after finishing using the service.

[1667] "Means for interacting through gestures and voice input" refers to technologies that allow users to communicate with the system using hand movements and voice.

[1668] The present invention is a system that provides users with guidance on new contracts, model changes, terminal operation methods, etc. in a virtual reality environment. This system receives user requests, initializes the virtual reality environment, and realizes interaction with the user using a synthetic speech generation means and a dialogue agent.

[1669] Server side

[1670] Request received

[1671] The server receives requests sent from users using the HTTPS protocol. The request contains the user's identification information (user ID) and the type of service desired (new contract, model change, device operation guide, etc.). The received request is parsed in JSON format and the appropriate processing is carried out.

[1672] Initializing the virtual reality environment

[1673] The server uses Unity to set up a virtual reality environment based on the user's request. This includes generating a VR space tailored to the user's preferences and placing a specific character avatar created using Blender. The virtual reality environment data is sent to the user's device and displayed.

[1674] Initialization of synthetic speech generator and dialogue agent

[1675] The server initializes the synthetic speech generator using Google Cloud's Text-to-Speech API and the conversational agent using Dialogflow, which prepares the text-to-speech conversion function and the ability to have natural conversations with the user.

[1676] Acquiring and updating user data

[1677] The server retrieves user data from the MySQL database and reflects it in the virtual reality environment. The user data includes user profile information and past usage history. This allows the server to provide appropriate services for each individual user.

[1678] Managing real-time interactions

[1679] The server receives voice input from the user and generates an appropriate response using Dialogflow. The generated response is converted into audio using Google Cloud's Text-to-Speech API and provided to the user. For example, if a user types "I would like to sign a new contract," the conversational agent will suggest an appropriate contract plan and explain its details in audio.

[1680] Payment Processing

[1681] The server processes payments using the Stripe API. For example, if a paid service such as a terminal operation guide is used, the server will charge the user's account. This allows the user to easily pay after receiving the service in the virtual reality environment.

[1682] Terminal side

[1683] Launching a VR app

[1684] A user launches an app on a VR device (e.g., Oculus Quest), which initiates a connection with the server and loads the virtual reality environment. VR apps run using the Oculus VR SDK.

[1685] Connecting to the Server

[1686] The device connects to the server and sends the user's request to the server. The device receives the VR environment data returned from the server and displays it. The device connects using Wi-Fi or 5G.

[1687] Service Selection

[1688] Within the virtual reality environment, users select the service they want to use, such as signing up for a new contract, upgrading their device, or receiving device operation instructions. Service selection is performed using an interactive menu within the VR app.

[1689] User Interaction

[1690] The device manages interactions with the user. The user interacts with a specific character through gestures or voice input, and receives necessary procedures and guidance. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the device's operation procedures.

[1691] User side

[1692] Wearing a VR device

[1693] Users put on a VR device (such as Oculus Quest) and prepare to use the service, which allows them to enter an immersive virtual reality environment.

[1694] Service Selection

[1695] Users can select services such as new contracts, model changes, and operation guidance within the virtual reality environment, allowing them to efficiently complete the procedures they desire.

[1696] Start a dialogue

[1697] Users interact with a specific character avatar to receive services, including obtaining necessary information through natural conversation. For example, a user can begin the process by entering "I would like to sign a new contract."

[1698] Payment process

[1699] When using paid services, users complete payment procedures within the virtual reality environment, including entering credit card information or using a QR code. Payments are processed by the server via the Stripe API.

[1700] VR Shop closed

[1701] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1702] Examples of concrete examples and prompts

[1703] For example, if a user wants to sign up for a new smartphone model, the following steps are taken.

[1704] 1. The user puts on the Oculus Quest and launches the VR app.

[1705] 2. Select "New Contract" within the virtual reality environment.

[1706] 3. A specific character will appear and ask questions about the contract in a dialogue format.

[1707] 4. The server generates the appropriate contract plan through Dialogflow and converts it into audio using Google Cloud's Text-to-Speech API.

[1708] 5. The user follows the character's instructions to complete the contract procedure.

[1709] Prompt Sentence Examples

[1710] "Enter a scenario where a user signs up for a new smartphone:

[1711] 1. First, the user puts on the VR device and launches the VR app.

[1712] 2. Select a service menu in the virtual reality environment

[1713] 3. Check the details of the new contract through dialogue with the character

[1714] 4. Payment Procedure

[1715] 5. Close the VR shop.

[1716] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1717] Step 1: Receiving a request

[1718] The server receives a request from the user via the HTTPS protocol. This request includes the user ID and the details of the desired service (e.g., new contract, model change, terminal operation guide). The input is the user's request data, which is analyzed to extract the necessary information. The output is the analyzed user ID and service content data. Specifically, the server uses Apache HTTP Server to analyze the received JSON data.

[1719] Step 2: Initializing the Virtual Reality Environment

[1720] The server uses Unity to set up a virtual reality environment based on the user's request. The input is the user ID and service content data, and based on this, it configures the VR space and generates a specific character avatar. The output is the generated VR space data. Specifically, it calls the Unity API and places a VR space tailored to the user's preferences and a character avatar generated using Blender.

[1721] Step 3: Initialize the synthetic speech generator and the dialogue agent

[1722] The server initializes the synthetic speech generation means using Google Cloud's Text-to-Speech API and the dialogue agent using Dialogflow. User data and service content data are used as input, and each API is configured based on this. Synthetic speech and dialogue agent instances are obtained as output. Specifically, the server accesses the API key and settings via the Google Cloud Console and generates the required instances.

[1723] Step 4: Obtaining and updating user data

[1724] The server retrieves user data from a MySQL database and reflects it in the virtual reality environment. It executes a database query using the user ID as input, and obtains user profile information and usage history as output. Specifically, it executes a MySQL query to retrieve the necessary data and applies it to the VR environment settings.

[1725] Step 5: Managing real-time interactions

[1726] The server receives voice input from the user and generates an appropriate response using Dialogflow. It then converts the voice input into audio using Google Cloud's Text-to-Speech API and provides it to the user. The input is the user's voice input data, which is analyzed and converted into text, and a response is generated using a conversational agent. The output is audio response data. Specifically, the voice data is temporarily saved to a WAV file, sent to Dialogflow, and the returned text is passed to the synthetic speech API to generate audio.

[1727] Step 6: Payment Processing

[1728] After a paid service is used, the server processes the payment through the Stripe API. The input is the user's payment information and service information, and billing is performed based on this. The output is payment confirmation data. Specifically, the server calls the Stripe API, processes the user's payment information, and completes the payment.

[1729] Step 7: Launch the VR app

[1730] A user launches an app on a VR device such as Oculus Quest. The input is a command to launch the VR device, and the output is to start accessing the VR environment. Specifically, the app launches using the Oculus VR SDK and establishes a network connection.

[1731] Step 8: Connect to the Server

[1732] The device connects to the server and sends the user's request to the server. The input is the user's request data, and the output is the virtual reality environment data. Specifically, the device sends an HTTP request to the server and receives the VR environment data returned from the server.

[1733] Step 9: Select a service

[1734] The user selects the service they want to use in the virtual reality environment. The input is the user's service selection, and the output is instructions based on the selected service returned from the server. Specifically, the user selects the service using an interactive menu within the VR app.

[1735] Step 10: User Interaction

[1736] The device manages interactions with the user. Inputs include user gestures and voice inputs, and output displays response data from the server. Specifically, it uses WebSocket to capture user gestures and voice commands and send them to the server.

[1737] Step 11: Put on the VR device

[1738] The user puts on the VR device and prepares to use the service. The input is a command to put on the VR device, and the output is to start accessing the virtual reality environment. Specifically, the user puts on the Oculus Quest and prepares the handheld controller.

[1739] Step 12: Select a service

[1740] Within the virtual reality environment, the user selects services such as new contracts, model changes, and operation guidance. The input is the user's service selection, and the output is instructions based on the selected service provided by the server. Specific operations involve using a menu provided within the VR environment to select a service that suits the purpose.

[1741] Step 13: Start a conversation

[1742] A user initiates a dialogue with a specific character avatar. The input is the user's voice input, and the dialogue agent generates a response accordingly. The output is a voice response from the dialogue agent. Specifically, the user enters a voice command, which is processed by the dialogue agent to generate a response.

[1743] Step 14: Payment Process

[1744] When using a paid service, the user completes the payment process within the virtual reality environment. The input is the user's payment information, and the output is payment confirmation data. Specifically, the user enters their credit card information within the virtual reality environment and completes the payment via the Stripe API.

[1745] Step 15: Finish the VR Shop

[1746] After completing the service, the user removes the VR device and exits the virtual reality shop. The input is the user's exit command, and the output is the ability to return to the real world. The specific actions are to close the VR app and remove the Oculus Quest.

[1747] (Application example 1)

[1748] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1749] Services provided within conventional virtual reality environments have not fully satisfied user convenience. In particular, it has been difficult to provide information on new contracts, model upgrades, and device operation methods quickly and effectively within a virtual reality environment. Furthermore, user interaction has been limited, and there has been a lack of means for real-time dialogue, resulting in a poor user experience. Furthermore, there has been a demand for a method that allows more users to easily access services by using virtual reality devices connected to smartphones.

[1750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1751] In this invention, the server includes means for receiving a user request, means for initializing a virtual reality environment, means for initializing a synthetic voice artificial intelligence, means for initializing a conversational artificial intelligence, means for generating an avatar of a specific character based on the user request, means for acquiring the user data and reflecting it in the virtual reality environment, means for managing real-time dialogue with the user, means for processing billing, means for realizing a dialogue system for providing guidance on new contracts, model changes, and device operation within the virtual reality environment, and means for the user to interact using a virtual reality headset and controller connected to a smartphone. This allows the user to easily and quickly receive more convenient and effective guidance on new contracts, model changes, and device operation within the virtual reality environment.

[1752] 1. "Means for receiving user requests" refers to the methods and functions by which the system receives operations and requests from users.

[1753] 2. "Means for initializing a virtual reality environment" means methods and functionality for configuring and building a user-accessible virtual reality environment.

[1754] 3. "Means for initializing a synthetic speech artificial intelligence" means methods and functions for configuring and activating an artificial intelligence that converts text data into speech data.

[1755] 4. "Means for initializing conversational AI" refers to methods and functions for configuring and launching an AI that communicates in natural language.

[1756] 5. "Means for generating an avatar of a specific character" means methods and functions for displaying a specific character within a virtual reality environment based on a user's request.

[1757] 6. "Means for acquiring user data and reflecting it in the virtual reality environment" refers to the methods and functions for acquiring user information from the database and applying it within the virtual reality environment.

[1758] 7. "Means for managing real-time interactions with users" refers to methods and functions that enable users and systems to interact with each other in real time.

[1759] 8. "Means for billing processing" refers to the methods and functions for managing and processing payment procedures when a user uses a paid service.

[1760] 9. "Means for realizing an interactive system for providing guidance on new contracts, model changes, and device operation within a virtual reality environment" refers to methods and functions that provide users with interactive guidance on new contracts, model changes, and device operation within a virtual reality environment.

[1761] 10. "Means for user interaction using a virtual reality headset and controllers connected to a smartphone" means methods and functionality that allow a user to navigate and interact within a virtual reality environment using a virtual reality headset and controllers connected to a smartphone.

[1762] The present invention is a system that allows users to receive service information such as new contracts, model changes, and terminal operation methods within a virtual reality environment. This system is composed of three parties: a server, a terminal, and a user, and uses detailed hardware and software to fulfill each role.

[1763] Server-side implementation

[1764] The server includes the following means:

[1765] 1. Means of receiving a user request:

[1766] The server receives a request sent by the user, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1767] 2. Means of initializing the virtual reality environment:

[1768] Based on the user's request, the server initializes the virtual reality environment using a VR toolkit (e.g., Unity or Unreal Engine), including creating a VR space tailored to the user's preferences and generating an avatar for a specific character.

[1769] 3. Methods for initializing the synthetic voice AI:

[1770] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly) that converts text data into voice data.

[1771] 4. Methods for initializing conversational AI:

[1772] The server initializes a conversational AI (e.g., Dialogflow or Microsoft Bot Framework) to enable natural dialogue with the user.

[1773] 5. How user data is acquired and reflected in the virtual reality environment:

[1774] The server retrieves the user's profile information and past usage history from the database and reflects this in the virtual reality environment.

[1775] 6. Means of managing real-time user interactions:

[1776] The server receives voice input from the user in real time and uses conversational AI to generate a response, which is then converted into speech by synthetic speech AI and provided to the user.

[1777] 7. Means of processing claims:

[1778] When a paid service such as terminal operation guidance is used, the server processes the billing to the user's account, allowing the user to easily make payment after receiving the service.

[1779] Terminal side embodiment

[1780] The terminal includes the following means:

[1781] 1. Launch the VR app:

[1782] The user puts on a VR headset and launches a VR application on their smartphone, which initiates a connection with the server.

[1783] 2. Connect to the server:

[1784] The device connects to the server, sends the user's request to the server, receives the VR environment data returned from the server, and displays it.

[1785] 3. Service Selection:

[1786] The user uses an interface within the virtual reality environment to select services such as new contracts, model changes, and terminal operation guidance.

[1787] 4. User Interaction:

[1788] The terminal manages interactions with the user, who interacts with specific characters through gestures and voice input to receive necessary procedures and instructions.

[1789] User-Side Embodiment

[1790] The user includes the following means:

[1791] 1. Wearing the VR device:

[1792] The user prepares to enter the virtual reality environment by putting on a VR headset and preparing the handheld controllers.

[1793] 2. Service Selection:

[1794] The user operates the interface provided within the virtual reality environment to select services such as new contracts, model changes, and operation guidance.

[1795] 3. Start a conversation:

[1796] The user can start a conversation with a specific character and receive the necessary information or procedures. For example, if the user wants to know how to operate the terminal, the character will demonstrate and explain the terminal operation procedure.

[1797] 4. Payment Procedure:

[1798] When using a paid service, users can complete payment procedures within the virtual reality environment by entering credit card information or using a QR code.

[1799] 5. VR Shop Closure:

[1800] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1801] Specific examples

[1802] For example, if a user wishes to sign a new contract, they can say "I would like to sign a new contract" by voice within the VR environment, and a character avatar will respond by saying, "Here are our new contract plans. Which plan would you like?" In this way, each time the user asks a question, the conversational AI generates an appropriate response and replies audibly through the synthetic voice AI.

[1803] Prompt Sentence Examples

[1804] Describe a dialogue system with an AI character that realizes a scenario in which a user wishes to make a new contract in a VR environment. Explain in detail the response and behavior of the character avatar when the user says, "I want to make a new contract."

[1805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1806] Step 1:

[1807] The server receives the user's request, which includes identification information (user ID) and the type of service desired (new contract, model change, terminal operation guide, etc.). Based on this input, the server analyzes the request and begins preparing an appropriate virtual reality environment.

[1808] Step 2:

[1809] The server initializes the virtual reality environment. This is the process of generating a VR space tailored to the user's preferences using a VR toolkit (e.g., Unity or Unreal Engine) based on the user's request. An avatar for the specified character is also generated and configured. As a result, a virtual reality environment tailored to the user is constructed.

[1810] Step 3:

[1811] The server initializes a synthetic voice AI (e.g., Google Text-to-Speech API or Amazon Polly). At this stage, an instance of the synthetic voice AI is created and prepared to convert text data into speech data. This allows speech output for subsequent user interactions.

[1812] Step 4:

[1813] The server initializes the conversational AI (e.g., Dialogflow or Microsoft Bot Framework), creates an instance of the conversational AI, and configures it to enable natural dialogue with the user. This configuration includes preparing a model to generate appropriate responses to user input.

[1814] Step 5:

[1815] The server retrieves user data from the database and reflects it in the virtual reality environment. The user data includes profile information and usage history, and personalizes the VR space based on that information. This information is used to improve the user experience.

[1816] Step 6:

[1817] The server manages real-time dialogue with the user. In this step, the user's voice input is converted and analyzed by conversational AI. Based on the analysis results, a response is generated through synthetic voice AI and provided to the user. This process is carried out in real time, ensuring smooth interaction between the user and the system.

[1818] Step 7:

[1819] The device launches the VR app and establishes a connection with the server, allowing the user to access the virtual reality environment and receive data sent from the server, enabling the user to view the virtual reality environment and operate the interface.

[1820] Step 8:

[1821] The device provides an interface for the user to select services. The user uses a VR headset and handheld controller to select services such as new contracts, model changes, and device operation guidance. This input causes the server to prepare the next response.

[1822] Step 9:

[1823] The user operates the interface and begins a dialogue with a specific character. For example, if the user voice-inputs, "I'd like to sign a new contract," the character will respond with, "Here are our new contract plans. Which plan would you like?" This dialogue progresses in real time, with responses generated by conversational AI.

[1824] Step 10:

[1825] When a user uses a paid service, the server handles the billing process. Specifically, the charge is sent to the user's account via credit card information entry or QR code payment, allowing the user to complete payment easily.

[1826] Step 11:

[1827] After completing their use of the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world and re-prepare the device for their next use.

[1828] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1829] This invention relates to a system that provides users with guidance on new contracts, model changes, and terminal operation methods in a virtual reality environment, while also recognizing the user's emotions and generating adaptive responses based on them. This system receives user requests, initializes the virtual reality environment, and interacts with the user using a synthetic voice AI, a conversational AI, and an emotion engine.

[1830] Server side

[1831] Processing from receiving a request to generating a response

[1832] The server receives the user's request. This request includes the user ID and the type of service desired (new contract, model change, device operation guide, etc.). The server then initializes the virtual reality environment based on the request. This includes generating a VR space tailored to the user's preferences and creating an avatar for the specified character.

[1833] Initialize instances of the synthetic speech AI and conversation AI. The synthetic speech AI has the function of converting text to speech, and the conversation AI generates appropriate responses to user input. The emotion engine is also initialized. The emotion engine analyzes the user's facial expressions and voice and recognizes emotions.

[1834] User data is retrieved from the database and reflected in the virtual reality environment, allowing the character avatar to provide appropriate services to the user.

[1835] The server receives input from the user in real time and generates a response using conversational AI. The generated response is converted into voice using synthetic voice AI and provided to the user. For example, if the user says, "I would like to sign a new contract," the conversational AI will suggest an appropriate contract plan and explain the details in voice.

[1836] The emotion engine analyzes the user's facial expressions and voice to detect emotions. Based on the detected emotions, the conversational AI generates a response that is adapted to the user. For example, if the user expresses surprise, the conversational AI will generate a response such as, "Did I surprise you?"

[1837] When a paid service such as a terminal operation guide is used, a billing process is executed, and the fee is charged to the user's account.

[1838] Terminal side

[1839] Processing from app launch to user interface management

[1840] The user launches the VR app on their device, which initiates a connection with the server and loads the virtual reality environment. The device then connects to the server and sends the user's request. The server returns the VR environment data, which is then displayed.

[1841] The user selects the services they wish to use within the virtual reality environment (new contract, model change, device operation guide, etc.). The selection is sent to the server. The device manages the user interface within the virtual reality environment, including displaying menus and handling interactions.

[1842] The user interacts with the character in the virtual reality environment through gestures and voice input. The device sends this input to the server, receives responses, and displays them. The emotion engine analyzes the user's facial expressions and voice and sends emotional data to the server.

[1843] User side

[1844] Processing from device installation to payment procedures

[1845] The user puts on the VR device and prepares to use the service. This allows the user to enter the virtual reality environment. The user selects the desired service using the interface provided within the virtual reality environment.

[1846] Based on the service selected, the user can start a conversation with a specific character avatar, which will then guide them through the necessary procedures. For example, if the user wants to know how to operate the device, the character will demonstrate and explain the operation procedure.

[1847] When using paid services, users make payments within the virtual reality environment. Payment methods include credit card entry and QR code payment. After completing the service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1848] Through these steps, users can enjoy efficient and entertaining services within the virtual reality environment. Furthermore, the emotion engine recognizes the user's emotions and provides adaptive responses, further enhancing the user experience. The system of the present invention improves user convenience and solves the problems of the past.

[1849] The processing flow will be explained below.

[1850] Server side

[1851] Processing steps from receiving a request to generating a response

[1852] Step 1:

[1853] The server receives the user's request, which includes the user ID and the type of service desired (new contract, model change, terminal operation guide, etc.).

[1854] Step 2:

[1855] The server initializes the virtual reality environment based on the request, generates a VR space according to the user's preferences, and prepares to generate an avatar for the specified character.

[1856] Step 3:

[1857] The server initializes instances of a synthetic speech AI and a conversational AI. The synthetic speech AI has the function of converting text to speech, and the conversational AI generates appropriate responses to user input.

[1858] Step 4:

[1859] The server retrieves user data from a database, including profile information and past usage history.

[1860] Step 5:

[1861] The server reflects the acquired user data in the virtual reality environment and applies the user's individual settings.

[1862] Step 6:

[1863] The server initiates a real-time dialogue with the user using conversational AI, taking the user's input and generating the optimal response.

[1864] Step 7:

[1865] The server uses synthetic voice AI to convert the generated response into speech, which is then provided to the user.

[1866] Step 8:

[1867] The server analyzes the user's facial expressions and voice through an emotion engine to recognize their emotions.

[1868] Step 9:

[1869] The server adjusts the conversational AI's responses based on the emotions it recognizes, for example, providing a follow-up message such as "Don't worry" if the user expresses anxiety.

[1870] Step 10:

[1871] The server performs billing when terminal operation guidance or other paid services are used, and charges are applied to the user's account.

[1872] Terminal side

[1873] Processing steps from app launch to user interface management

[1874] Step 1:

[1875] The user launches the VR app on their device, which initiates a connection with the server.

[1876] Step 2:

[1877] The device connects to the server and sends the user's request, which includes information such as the user ID and the type of service.

[1878] Step 3:

[1879] The device receives the VR environment data returned from the server and displays the virtual reality environment.

[1880] Step 4:

[1881] The user selects the services they wish to use in the virtual reality environment (new contract, model change, terminal operation guide, etc.), and the selected content is sent to the server.

[1882] Step 5:

[1883] The terminal manages the user interface within the virtual reality environment, displaying menus and handling interactions.

[1884] Step 6:

[1885] The emotion engine analyzes the user's facial expressions and voice, and sends the emotional data to the server.

[1886] Step 7:

[1887] The device receives responses from the server and displays them both audibly and visually, and adaptive responses are provided based on data from the emotion engine.

[1888] User side

[1889] Processing steps from device installation to payment procedure

[1890] Step 1:

[1891] The user puts on the VR device and prepares to use the service, allowing them to enter the virtual reality environment.

[1892] Step 2:

[1893] The user uses an interface provided within the virtual reality environment to select the service they want, for example, to sign up for a new contract or upgrade their device.

[1894] Step 3:

[1895] Based on the service selected, the user begins a conversation with a specific character avatar, who guides the user through specific procedures.

[1896] Step 4:

[1897] The user follows the character's instructions as they interact with the character, while the emotion engine recognizes the user's emotions and adjusts responses accordingly.

[1898] Step 5:

[1899] When using paid services, users can complete payment procedures within the virtual reality environment, such as entering credit card information or using a QR code.

[1900] Step 6:

[1901] After completing their service, users remove the VR device and exit the virtual reality shop, allowing them to return to the real world.

[1902] Example 2

[1903] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] Conventional virtual reality environments have difficulty understanding user emotions and providing adaptive responses. They also lack real-time dialogue management and services tailored to user needs. This can limit the user experience and reduce user satisfaction. Another issue is the complicated billing and payment procedures for paid services, which places a burden on users.

[1905] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1906] In this invention, the server includes a means for analyzing a user's facial expressions and voice to recognize emotions, a means for generating an adaptive response based on the emotion recognition, and a means for performing billing processing. This makes it possible to understand a user's emotions in real time and provide an adaptive response based on the emotion. It also realizes appropriate billing processing and payment procedures, improving the user experience and convenience.

[1907] The "means for receiving a user request" refers to a function or module for acquiring a request from a user.

[1908] A "means for initializing a virtual reality environment" is a system or program for setting up and starting up a virtual environment.

[1909] The "means for initializing the synthetic speech generation means" is a function for starting and setting up a speech generation system that has the function of converting text data into speech data.

[1910] The "means for initializing a conversation generation model" is a function for starting and setting up a conversation model that generates an appropriate response to a user's input.

[1911] A "means for generating an avatar of a specific character" is a system or program that generates the appearance and behavior of a virtual character based on the user's requests and settings.

[1912] "Means for acquiring user data and reflecting it in the virtual reality environment" refers to a function for acquiring user information from a database, etc., and applying it to the virtual reality space.

[1913] "Means for managing real-time user interactions" refers to functions and modules that monitor and control real-time communication between users and the system.

[1914] The "means for analyzing the user's facial expressions and voice and recognizing emotions" is an analysis system for detecting the user's emotional state from facial expression and voice data.

[1915] The "means for generating adaptive responses based on emotion recognition" is a function or module that generates an appropriate response that matches the recognized emotional state of the user.

[1916] The "means for billing" refers to a system or program that calculates fees for the use of paid services and bills the user account.

[1917] "Means for interaction through a user interface" refers to functions or modules that provid...

Claims

1. means for receiving a user request; a means for initializing the virtual reality environment; A means to initialize the synthetic voice AI, A way to initialize the conversation AI, means for generating an avatar of a specific character based on a request from the user; means for acquiring the user data and reflecting it in the virtual reality environment; means for managing real-time interactions with said user; a means for processing billing; A system including:

2. means for selecting a service within the virtual reality environment; a means for connecting to the server and retrieving data; means for interacting through said user interface; means for providing a guide tailored to the type of service used by the user; The system of claim 1 , comprising:

3. means for the user to wear a virtual reality device and start the service; means for selecting a service within the virtual reality environment; means for initiating a dialogue with said character; A means for carrying out payment procedures when using the paid service; a means for closing the virtual reality shop after the end of the service; The system of claim 1 , comprising:

Images