System

Abstract

A system is provided.SOLUTION: A system comprising: means for generating a metaverse space and arranging a plurality of virtual offices in a virtual reality space; means for managing account information of a user and granting an access right to an authenticated user; means for generating an avatar of the user in the virtual reality space and operating the avatar to move to the virtual reality office; AI means for receiving consultation contents from an input of the user and analyzing the consultation contents to generate an appropriate answer; and means for storing a consultation history in a database and using the consultation history for future learning.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern companies, various problems occur, including harassment and compliance violations, but many employees are hesitant to directly consult with the human resources or compliance department. This puts problems at risk of going undetected and becoming more serious. Many employees also prefer to avoid face-to-face consultations, making it difficult to find timely advice and solutions. To resolve these issues, it is necessary to provide an environment where employees can consult more easily and anonymously. [Means for solving the problem]

[0005] The present invention is a system that includes a means for generating a metaverse space and arranging multiple virtual offices within the virtual space, a means for managing user account information and granting access privileges to authenticated users, a means for generating a user avatar within the virtual space and manipulating the avatar to move to the virtual office, an AI means for receiving consultation content from the user's input, analyzing it, and generating an appropriate answer, and a means for storing consultation history in a database and using it for future learning. This allows users to consult with peace of mind while avoiding face-to-face contact, enabling problems to be discovered and resolved early.

[0006] The "metaverse space" is a three-dimensional virtual space on the Internet that is constructed using virtual reality and augmented reality technologies.

[0007] A "virtual office" is a virtual office space set up within the metaverse space for specific purposes or tasks.

[0008] "Account Information" means IDs, passwords, and other related information used to identify users and manage access rights.

[0009] "Access privileges" are permission settings that determine whether a user can access specific digital content or services.

[0010] An "avatar" is a three-dimensional character that represents a user within the metaverse space and can be controlled.

[0011] "AI means" refers to artificial intelligence technologies used to analyze information received from users and provide appropriate answers or measures.

[0012] "Consultation content" refers to text data of issues and questions related to harassment and compliance that users submit to the AI ​​help desk in the metaverse space.

[0013] A "database" is an information management system designed to store data in an organized manner so that it can be retrieved and used quickly and efficiently. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail below.

[0036] Creating and managing metaverse spaces

[0037] server:

[0038] The server first generates a metaverse space and places multiple virtual offices within the virtual space. This allows users to access locations for various tasks and consultations. Each virtual office corresponds to a department (e.g., human resources, compliance), and each has its own unique interactive functions. The server also manages user account information and controls access privileges by authenticating users. This ensures that only appropriate users can access each virtual office.

[0039] User Account Settings and Login Authentication

[0040] Device:

[0041] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0042] Log in to the virtual space and start using it

[0043] server:

[0044] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0045] Device:

[0046] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0047] Consultation at AI help desk

[0048] User:

[0049] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation.

[0050] Device:

[0051] The device sends the consultation details entered by the user to the AI. The input method is primarily text input, but other methods such as voice input are also supported.

[0052] server:

[0053] The AI ​​on the server analyzes the received inquiry and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0054] User:

[0055] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[0056] Follow-up on consultation results and history management

[0057] server:

[0058] The server stores all consultation history in a database and uses it for future learning, which is an important step to improve the AI's response accuracy and customer satisfaction.

[0059] Specific examples

[0060] Scenario 1: Harassment consultation

[0061] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[0062] 2. The device displays a consultation form, and the user enters the specific details of the harassment they wish to discuss.

[0063] 3. The server receives the consultation content, and the AI ​​analyzes it and provides solutions.

[0064] 4. The device displays the AI's answer, and the user decides on the next action.

[0065] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, enabling early detection and response to problems.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The server creates a metaverse space and places multiple virtual offices within it, each containing multiple areas used for different purposes, such as the human resources department and the compliance department.

[0069] Step 2:

[0070] The server manages user account information and stores it in a database, including user IDs, passwords, and access privileges.

[0071] Step 3:

[0072] The terminal displays a screen for the user to log in and prompts the user to enter their ID and password.

[0073] Step 4:

[0074] The user enters login information (ID and password) and clicks the login button.

[0075] Step 5:

[0076] The terminal transmits the entered login information to the server.

[0077] Step 6:

[0078] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[0079] Step 7:

[0080] The server generates an avatar for the authenticated user and places it in the virtual space, as well as setting the user's initial position and viewpoint.

[0081] Step 8:

[0082] The device renders and displays the user's avatar and the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0083] Step 9:

[0084] The user controls the avatar and heads to the desired virtual office (e.g., the compliance department).

[0085] Step 10:

[0086] When the user arrives at the virtual office, the terminal displays a form for inputting the details of the consultation.

[0087] Step 11:

[0088] Users enter their complaint in text and click the send button, which includes details of the specific harassment and other compliance issues.

[0089] Step 12:

[0090] The terminal transmits the input consultation content to the server.

[0091] Step 13:

[0092] The AI ​​on the server analyzes the received consultation content and generates appropriate answers and measures, using natural language processing technology.

[0093] Step 14:

[0094] The server sends the generated response to the terminal.

[0095] Step 15:

[0096] The device displays the AI's answer to the user, who can then ask further questions or add new information based on the answer.

[0097] Step 16:

[0098] The server stores all consultation history in a database, which is used for future learning and analysis.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Conventional consultation systems do not adequately provide an environment for quickly and accurately resolving users' problems. In particular, they lack a flexible system that can handle a wide range of consultation content while maintaining anonymity. Furthermore, because consultation history is not accumulated, it is difficult to make continuous improvements based on past consultation content. This makes it difficult to improve user satisfaction, and efficient problem-solving is required.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices within the virtual space, means for managing user account information and granting access rights to authenticated users, means for generating a user avatar within the virtual space and manipulating the avatar to move to the virtual office, means for receiving consultation content from the user's input, analyzing it using a generative AI model, and generating an appropriate answer, and means for saving consultation history in a database and using it for future learning. This allows users to freely consult while maintaining anonymity, and makes it possible to accumulate history and improve the accuracy of the AI.

[0104] A "metaverse space" is a digital space created using virtual reality technology, where users can move freely beyond physical constraints.

[0105] A "virtual office" is a virtual work area located within the metaverse space that corresponds to a specific business department. Users can consult and process business matters within this area.

[0106] "User account information" is a series of data (e.g., username, password, job title, etc.) used for personal authentication of users and access control.

[0107] "Authentication" is the process of verifying that a user has valid authority based on the entered user information.

[0108] "Access Privileges" means the authorization for a User to access certain resources or functions within the System.

[0109] An "avatar" is a character that represents a user in a virtual space, and is a digital entity that reflects the user's actions and behavior.

[0110] A "generative AI model" is an artificial intelligence system that uses natural language processing and machine learning techniques to analyze input data from users and generate appropriate responses.

[0111] "Consultation content" is detailed information about a question or problem that a user inputs to a help desk in the virtual space.

[0112] The "consultation history" is a record of all consultations that a user has made with the system, including the contents of those consultations and the responses to those consultations, and is saved for future reference and analysis.

[0113] A "database" is a system for storing data in a structured digital format, allowing for efficient retrieval and management.

[0114] This invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail.

[0115] Creating and managing metaverse spaces

[0116] server:

[0117] The server generates a metaverse space using game engines such as Unity or Unreal Engine. Multiple virtual offices are placed within the metaverse space. Each virtual office corresponds to a specific business department, such as the human resources department or the compliance department, and each has its own interactive functions. The server also manages user account information and performs authentication to enforce access control.

[0118] User Account Settings and Login Authentication

[0119] Device:

[0120] When a user uses the system, they first launch a dedicated application on their device. The user enters their ID and password on the login screen, and the authentication information is sent to the server. The server verifies the authentication information using a database, and if authentication is successful, the device is given access to the metaverse space.

[0121] Log in to the virtual space and start using it

[0122] server:

[0123] Once the user has been authenticated, the server generates an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar and head to the desired virtual office.

[0124] Device:

[0125] The device renders and displays the virtual space in real time based on the data received from the server. The virtual space is created using game engines such as Unity and Unreal Engine, allowing users to explore seamlessly.

[0126] Consultation at AI help desk

[0127] User:

[0128] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and enters the details of the consultation.

[0129] Device:

[0130] The device sends the consultation details entered by the user to the generation AI model. Input methods supported include text and voice input.

[0131] server:

[0132] A generative AI model (e.g., OpenAI's GPT-3) on the server analyzes the prompt entered by the user and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0133] Follow-up on consultation results and history management

[0134] server:

[0135] The server stores all consultation history in a database, which is used for future training of the AI ​​model to improve response accuracy.

[0136] Specific examples

[0137] Scenario 1: Harassment consultation

[0138] 1. The user launches the dedicated app and enters their login information into the device.

[0139] 2. The server authenticates the login information and provides the user with access to the metaverse space.

[0140] 3. After logging in, the terminal displays the virtual space, and the user operates the avatar to move to the "Compliance Department."

[0141] 4. The user enters the details of their consultation at the consultation booth, and the device sends the data to the generative AI model.

[0142] 5. The server's generated AI model analyzes the consultation content and generates appropriate countermeasures.

[0143] 6. The device displays the generated answer, and the user decides on the next action based on it.

[0144] Example prompt sentence:

[0145] "I would like to consult about measures to deal with harassment. To be more specific, my boss has been verbally abusing me. Could you please give me some advice on how I should respond?"

[0146] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and supports early detection and response to problems.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] Metaverse space generation and virtual office layout

[0150] server:

[0151] The server generates the metaverse space using Unity or Unreal Engine. First, a base environment for the virtual space is set up, and then multiple virtual offices are placed within it. The virtual offices are placed based on predefined coordinate data. Different interactive elements are also placed in each virtual office. For example, the HR department can provide an employee consultation service, and the compliance department can provide legal advice.

[0152] Input: Virtual space base setting data, virtual office metadata

[0153] Output: Location information within the metaverse space

[0154] Specific behavior:

[0155] The server runs the virtual space generation script and loads the layout information for each virtual office, completing the basic structure of the virtual space.

[0156] Step 2:

[0157] User Account Settings and Login Authentication

[0158] Device:

[0159] The user starts the dedicated application and enters their ID and password on the login screen. This authentication information is sent to the server as an HTTP request.

[0160] server:

[0161] The server uses a database to verify the submitted credentials, and if successful, stores the user's account information in the session and grants them access to the metaverse space.

[0162] Input: User ID and password

[0163] Output: Authentication result, session ID

[0164] Specific behavior:

[0165] The server checks the authentication information against the database, and if authentication is successful, generates a session ID and sends it to the terminal.

[0166] Step 3:

[0167] Log in to the virtual space and generate an avatar

[0168] server:

[0169] Once a user has been authenticated, the server generates an avatar in the virtual space. This avatar is a character that reflects the user's actions, and its appearance and functions are set based on the user's profile data.

[0170] Device:

[0171] The device renders and displays the virtual space in real time based on the avatar and virtual space information received from the server, allowing the user to begin activities within the virtual space.

[0172] Input: User credentials, profile data

[0173] Output: Avatar information in virtual space, virtual space rendering information

[0174] Specific behavior:

[0175] The server runs an avatar generation script based on the user's profile information and sends rendering data to the device, which then receives the data and renders the virtual space.

[0176] Step 4:

[0177] Consultation at AI help desk

[0178] User:

[0179] The user controls an avatar to move to a specific virtual office, select a consultation booth, and enter the details of their consultation. The input method is primarily text, but voice input is also supported.

[0180] Device:

[0181] The device converts the consultation content entered by the user into a text prompt and sends it to the generative AI model.

[0182] server:

[0183] A generative AI model (e.g., GPT-3) on the server analyzes the received prompt and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0184] Input: User's inquiry (prompt text)

[0185] Output: Answer from the generative AI model

[0186] Specific behavior:

[0187] The user inputs the content of their inquiry using a keyboard or voice input, and the device sends this as an HTTP request to the server. The server then analyzes the content of the inquiry using a generative AI model, generates a response, and sends it to the device.

[0188] Step 5:

[0189] Follow-up on consultation results and history management

[0190] server:

[0191] The server stores all consultation history in a database, including the user's consultation content and the answers provided by the generative AI model. This data will be used as training data for future projects.

[0192] Input: User's inquiry, response from the generated AI model

[0193] Output: Consultation history data

[0194] Specific behavior:

[0195] The server stores the received inquiries and generated responses in a log file or database for future reference and analysis.

[0196] This system provides an environment where users can easily consult with the system, and also allows historical data to be used to improve the performance of the AI.

[0197] (Application example 1)

[0198] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0199] Maintenance and inspection work in factories takes a lot of time and effort, and responding to machine abnormalities and breakdowns in particular requires advanced expertise. Conventional systems make it difficult to provide immediate on-site responses or appropriate solutions. As a result, efficient production is often hindered. Furthermore, differences in the knowledge and experience of technicians can lead to variations in response quality, reducing overall operational efficiency. To solve these issues, a system that supports on-site maintenance and inspection work is needed.

[0200] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0201] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; generation AI means for receiving consultation content from the user, analyzing it, and generating an appropriate answer; means for storing the consultation history in a database and using it for future learning; AI means for guiding procedures to support maintenance and inspection work in factories and providing solutions when problems occur; and means for factory employees and engineers to perform maintenance and inspection work in the virtual space using smartphones, smart glasses, or head-mounted displays. This enables immediate response and provision of appropriate solutions on-site, improving work efficiency throughout the factory.

[0202] The "metaverse space" is a virtual space that exists on the Internet, where users can interact with each other through avatars.

[0203] A "virtual office" is a virtual work location located within the metaverse space that provides the same functions as a real office.

[0204] "User account information" is information for uniquely identifying a user, and includes authentication information such as a login ID and password.

[0205] "Access privileges" refer to the rights of an authenticated user to access specific resources or functions within a system.

[0206] An "avatar" is a character that represents a user, moves within a virtual space, and interacts with other users and the system.

[0207] "Generative AI means" refers to artificial intelligence technology that analyzes user input and automatically generates appropriate answers and countermeasures.

[0208] "Consultation history" refers to a record of the consultations users make through the system, as well as the responses and countermeasures taken.

[0209] "Procedures to support maintenance and inspection work" are guidelines to instruct and support procedures for efficiently carrying out maintenance and inspection of machinery and equipment within a factory.

[0210] A "smartphone" is a portable information terminal that can connect to the Internet and use a variety of applications.

[0211] "Smart glasses" are a wearable eyeglass-type device equipped with information display and interactive functions.

[0212] A "head-mounted display" is a display device worn on the head, used to display a virtual space to the user.

[0213] "Factory employees" are workers who operate machines and perform maintenance and inspection work within a factory.

[0214] An "engineer" is a technician who designs, develops, maintains, and inspects machinery and equipment.

[0215] This invention is an AI-enabled metaverse system to support maintenance and inspection work in factories. This system has the function of placing multiple virtual offices in the metaverse space, managing user account information, and granting appropriate access rights through authentication. The details are explained below.

[0216] Creating and managing metaverse spaces

[0217] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices correspond to areas where maintenance and inspection work is carried out in the factory, and users can access them to carry out their work efficiently.

[0218] User Account Settings and Login Authentication

[0219] When a user uses the system, they first launch the application on their device (smartphone, smart glasses, head-mounted display) and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0220] Log in to the virtual space and start using it

[0221] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The user controls this avatar to move to the area where maintenance and inspection work will be performed. The device renders and displays the virtual space in real time based on the data received from the server.

[0222] Supporting maintenance and inspection work with AI tools

[0223] Users can control an avatar to move near specific equipment or machinery and begin maintenance or inspection tasks. The server-based AI generator analyzes the user's questions and problem reports and presents appropriate solutions and procedures. The following are examples of specific prompts:

[0224] Engineer: "I'm hearing an abnormal noise coming from the machine. What measures should I take?"

[0225] AI: "Please first perform a frequency analysis of the machine to identify the cause of the abnormal noise. If that doesn't solve the problem, please try the steps below."

[0226] Consultation history storage and follow-up

[0227] The server stores all inquiries and answers in a database for future learning, and the generative AI method provides continuous follow-up questions and solutions based on past history, allowing users to solve their problems effectively.

[0228] Program processing explanation

[0229] The server is operated as a FastAPI-based API server on a cloud service (e.g., AWS, Azure). Internet-enabled devices (smartphones, smart glasses, head-mounted displays) communicate with the server and render data in the virtual space. The generative AI means on the server analyzes the user's input, generates appropriate answers, and sends the answers to the user's device in real time. Using a specific generative AI model (e.g., GPT-4) enables more accurate responses.

[0230] In this way, the system of the present invention effectively supports maintenance and inspection work within a factory and contributes to improving work efficiency.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The server generates a metaverse space and places multiple virtual offices within the virtual space. This sets up the business areas and consultation booths that users can access. The input is data on how to arrange the virtual offices and business areas as server configuration information, and the output is the metaverse space that users can access.

[0234] Step 2:

[0235] A user turns on a device (smartphone, smart glasses, or head-mounted display) and enters login information. The device sends this information to a server, which then authenticates the user's account information. The input is the user's authentication information (login ID and password), and the output is the success or failure of the authentication.

[0236] Step 3:

[0237] If authentication is successful, the server generates an avatar for the user in the virtual space. The device renders the virtual space based on the data received from the server and displays it to the user. The input is the user's account information and access privileges, and the output is the avatar display in the virtual space.

[0238] Step 4:

[0239] The user controls the avatar to move to a specific virtual office (maintenance and inspection area). The terminal reflects the user's operations in real time and updates the avatar's position. The input is the user's operation command (movement instruction), and the output is the avatar's new position information.

[0240] Step 5:

[0241] When a user moves an avatar near a specific machine or equipment to begin maintenance or inspection work, the device sends the user's question or problem details to the server. The input is the user's question or problem report, and the output is a receipt confirmation from the server.

[0242] Step 6:

[0243] The generative AI means on the server analyzes the received user question and generates an appropriate answer and response procedure. This uses a generative AI model such as GPT-4. The input is the user question, and the output is the generated answer and response procedure.

[0244] Step 7:

[0245] The server sends the generated answers and response procedures to the user's device. The device displays the received answers and response procedures to the user. The input is the answer data from the generation AI means, and the output is the content displayed on the user's device.

[0246] Step 8:

[0247] The user's consultation history and response procedures are stored in a database on the server. This allows them to be used for future consultations and problem solving. The input is the user's consultation content and the response results of the generated AI, and the output is the data stored in the database.

[0248] Step 9:

[0249] The server proposes continuous follow-up questions and solutions based on past history and provides them to the user. The input is past consultation history data, and the output is follow-up questions and solution suggestions.

[0250] The purpose of these processing steps is to ensure that maintenance and inspection work within the factory is carried out efficiently and to provide prompt support for users in resolving problems.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] The present invention combines an AI consultation system that uses the metaverse space with an emotion engine, and the embodiments thereof will be described in detail below.

[0253] Creating and managing metaverse spaces

[0254] server:

[0255] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access each virtual office.

[0256] User Account Settings and Login Authentication

[0257] Device:

[0258] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0259] Log in to the virtual space and start using it

[0260] server:

[0261] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0262] Device:

[0263] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0264] Consultation at AI help desk and use of emotion engine

[0265] User:

[0266] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation. The device is equipped with a video camera and microphone to recognize the user's facial expressions and voice.

[0267] Device:

[0268] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[0269] server:

[0270] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. It automatically generates follow-up questions and countermeasures according to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[0271] User:

[0272] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[0273] Emotion Log and Follow-up

[0274] server:

[0275] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0276] Specific examples

[0277] Scenario 1: Harassment consultation and sentiment analysis

[0278] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[0279] 2. The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[0280] 3. The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[0281] 4. The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional follow-up questions.

[0282] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and by using an emotion engine, more personalized responses are possible.

[0283] The processing flow will be explained below.

[0284] Step 1:

[0285] The server generates a metaverse space and places multiple virtual offices within it, each of which is set up as an area for accepting specific tasks or consultations.

[0286] Step 2:

[0287] The server registers user account information in a database and manages authentication information, including user ID, password, and access privileges.

[0288] Step 3:

[0289] The terminal displays a login screen for the user to use the system.

[0290] Step 4:

[0291] The user enters their ID and password on the login screen and clicks the login button.

[0292] Step 5:

[0293] The terminal transmits the entered login information to the server.

[0294] Step 6:

[0295] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[0296] Step 7:

[0297] The server generates an avatar for the authenticated user and places it in the virtual space, setting the avatar's initial position and viewpoint.

[0298] Step 8:

[0299] The device renders and displays the virtual space based on the data received from the server, and the user can control the avatar in real time and explore the virtual space.

[0300] Step 9:

[0301] The user controls the avatar to access the desired virtual office (e.g., the compliance department).

[0302] Step 10:

[0303] When the user arrives at the virtual office, the terminal activates the emotion engine, activates the video camera and microphone to transmit the user's facial expressions and voice to the emotion engine, and displays a form for inputting the details of the consultation.

[0304] Step 11:

[0305] The user enters the content of the consultation in text and clicks the send button.

[0306] Step 12:

[0307] The device sends the input consultation content and facial expression and voice data obtained from the emotion engine to the server.

[0308] Step 13:

[0309] The AI ​​on the server analyzes the received consultation content and emotional information, generates an appropriate response, and adjusts follow-up questions and countermeasures based on the emotional information according to the user's emotional state.

[0310] Step 14:

[0311] The server sends the generated answers and follow-up questions to the terminal.

[0312] Step 15:

[0313] The device displays the AI's answers and follow-up questions to the user.

[0314] Step 16:

[0315] The user reviews the AI's answers and enters additional questions or information as needed.

[0316] Step 17:

[0317] The server stores all consultation history and emotion logs in a database, which allows future learning and tracking of users' long-term emotional changes.

[0318] Through the above process, users can feel safe consulting within the metaverse space, and by utilizing the emotion engine, more personalized responses can be provided.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] In modern companies and organizations, it is important to provide an environment where employees can seek advice anonymously and with peace of mind. However, current systems make it difficult to accurately grasp the user's emotions based on the text of the consultation alone, making it difficult to respond appropriately. In addition, there is a lack of follow-up and personalized responses in response to changes in employees' emotions, so improvements are needed to increase employee satisfaction.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0323] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; AI means for receiving consultation content from the user, analyzing it, and generating an appropriate response; emotion engine means for recognizing and analyzing the user's facial expressions and voice; means for the AI ​​means to generate follow-up questions and countermeasures according to the emotion information obtained from the emotion engine; and means for storing consultation history and emotion logs in a database and using them for future learning. This allows for real-time analysis of user emotions and enables appropriate responses and follow-ups based on the analysis. This provides an environment where employees can consult with confidence and improves employee satisfaction.

[0324] The "metaverse space" is a digital virtual space generated using virtual reality technology, in which users can control avatars and carry out various activities.

[0325] A "virtual office" is a virtual office located within the metaverse space, where activities for specific purposes, such as the human resources department or compliance department, are carried out.

[0326] "User account information" refers to information for identifying a user, including a user name, password, email address, and the like.

[0327] The "means for granting access rights" is a mechanism that grants the right to access a specific virtual office or a specific area within the metaverse space when a user is authenticated.

[0328] An "avatar" is a virtual character that represents a user within the metaverse space, and moves and acts within the virtual space according to the user's instructions.

[0329] "AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of analyzing the content of inquiries from users and generating appropriate answers.

[0330] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to evaluate their current emotional state in real time.

[0331] The "means for generating follow-up questions and countermeasures" is a mechanism that continuously asks questions to the user and provides appropriate countermeasures based on information obtained from the emotion engine.

[0332] "Consultation history" is a record of the content of the consultation a user has had and the responses to those consultations, and is data that is saved for use in future learning and analysis.

[0333] An "emotion log" is data that records the user's emotional state during a consultation, and serves as the basis for providing a more personalized response the next time the user consults.

[0334] The present invention combines an AI consultation system that utilizes the metaverse space with an emotion engine, and the following describes its implementation in detail. The entire system is composed of the main elements of a server, a terminal, and a user.

[0335] Creating and managing metaverse spaces

[0336] server:

[0337] The server first generates a virtual metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access specific virtual offices.

[0338] User Account Settings and Login Authentication

[0339] Device:

[0340] When a user uses the system, they first launch a dedicated application on their device and enter their login information. The entered information is sent from the device to the server, and once authenticated by the server, the user is able to access the metaverse space.

[0341] Log in to the virtual space and start using it

[0342] server:

[0343] Once a user has been authenticated, an avatar is generated and displayed in the virtual space. The avatar is a character that represents the user and can move freely within the virtual space.

[0344] Device:

[0345] The device renders the virtual space based on the data received from the server and displays it to the user, allowing the user to explore the virtual space in real time.

[0346] Consultation at AI help desk and use of emotion engine

[0347] User:

[0348] The user controls an avatar to access a specific virtual office (e.g., the compliance department) and selects a consultation booth. A form for entering the details of the consultation is displayed. The terminal's video camera and microphone recognize the user's facial expressions and voice.

[0349] Device:

[0350] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[0351] server:

[0352] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. Follow-up questions and countermeasures are also automatically generated in response to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[0353] User:

[0354] Based on the answers provided by the AI, users can continue to ask questions or add new information, and can also move to another virtual office for another consultation if necessary.

[0355] Emotion Log and Follow-up

[0356] server:

[0357] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0358] Specific examples

[0359] Scenario 1: Harassment consultation and sentiment analysis

[0360] Users log in and access the "Compliance Department" within the virtual space.

[0361] The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[0362] The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[0363] The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional questions or follow-ups.

[0364] Prompt Sentence Examples

[0365] "Please tell me more about the harassment you recently experienced. Can you describe the specific situation and how it made you feel?"

[0366] In this way, according to the embodiment of the invention, it is possible to analyze the user's emotions in real time and provide appropriate responses and follow-ups based on the analysis. This system provides an environment where users can feel at ease when seeking advice, and realizes more personalized responses.

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] Device:

[0370] The user starts the application and enters account information (user name, password) on the login screen.

[0371] Input: Username, Password

[0372] Action: A user enters login information at a terminal.

[0373] Output: The entered login information

[0374] Step 2:

[0375] Device:

[0376] The entered login information is sent to the server.

[0377] Input: The login information entered

[0378] How it works: The device sends the login information to the server.

[0379] Output: Sending login information to the server

[0380] Step 3:

[0381] server:

[0382] The received account information is compared with information in the database to verify that the account exists.

[0383] Input: Login information sent from the device

[0384] Action: The server checks against the database.

[0385] Output: Authentication result (success / failure)

[0386] Step 4:

[0387] server:

[0388] If authentication is successful, the user's session information is generated and sent back to the terminal. If authentication fails, an error message is sent back to the terminal.

[0389] Input: Authentication result (success / failure), user account information

[0390] Action: The server generates session information or an error message.

[0391] Output: Session information (if successful), error message (if unsuccessful)

[0392] Step 5:

[0393] Device:

[0394] The terminal receives a message indicating successful authentication and provides the user with access to the metaverse space.

[0395] Input: Session information (if successful)

[0396] Action: The device displays the access interface to the metaverse space.

[0397] Output: Access rights to the metaverse space

[0398] Step 6:

[0399] server:

[0400] Once authenticated, an avatar is generated and displayed in the virtual space for the user. Multiple virtual offices are placed within the virtual space.

[0401] Input: User credentials, Virtual Office configuration data

[0402] How it works: The server generates an avatar and a virtual office.

[0403] Output: Avatar information, virtual office layout data

[0404] Step 7:

[0405] Device:

[0406] The virtual space is rendered based on the data received from the server and displayed to the user.

[0407] Input: Avatar information, virtual office layout data

[0408] How it works: The device renders the virtual space in real time.

[0409] Output: Visual information in virtual space

[0410] Step 8:

[0411] User:

[0412] The user operates an avatar, accesses a specific virtual office, selects a consultation booth, and inputs the details of the consultation.

[0413] Input: User operation (moving the avatar and selecting a consultation booth), consultation content

[0414] Operation: The user enters the details of the consultation at the consultation booth.

[0415] Output: Input consultation details

[0416] Step 9:

[0417] Device:

[0418] The system uses a video camera and microphone to capture the user's facial expressions and voice, and then sends the consultation details and the captured facial expression and voice data to the server.

[0419] Input: consultation details, facial expressions, and voice data

[0420] How it works: The device captures facial expressions and voice and sends them to the server.

[0421] Output: Send data to the server

[0422] Step 10:

[0423] server:

[0424] The received consultation content and user emotional information are sent to the emotion engine for analysis. The emotion engine analyzes the user's emotions in real time and obtains the results.

[0425] Input: consultation details, emotional information

[0426] Operation: The server sends data to the emotion engine and obtains the analysis results.

[0427] Output: Emotion analysis results

[0428] Step 11:

[0429] server:

[0430] Based on the emotional information and the content of the consultation, a generative AI model is used to generate an appropriate response. Follow-up questions and countermeasures are automatically generated according to changes in emotions. The generated response is then sent to the device.

[0431] Input: Consultation details, emotion analysis results

[0432] Operation: The server generates appropriate answers and follow-up questions and sends them to the device.

[0433] Output: Generated answers, follow-up questions

[0434] Step 12:

[0435] Device:

[0436] The received response is displayed to the user.

[0437] Input: Generated answers, follow-up questions

[0438] Action: The device displays the answer.

[0439] Output: Display the answer to the user

[0440] Step 13:

[0441] User:

[0442] Based on the answers provided by the AI, users can ask further questions or add new information, and if necessary, they can move to another virtual office for another consultation.

[0443] Input: AI response

[0444] Action: The user enters additional questions or information, or navigates to another virtual office.

[0445] Output: Additional questions, information, and next steps

[0446] Step 14:

[0447] server:

[0448] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of AI responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0449] Input: consultation history, emotion log

[0450] What happens: The server saves the data to the database.

[0451] Output: Saved data log

[0452] (Application example 2)

[0453] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0454] The purpose of this invention is to provide an environment where customers can easily and quickly ask questions or make complaints in a physical store. In particular, it aims to improve customer satisfaction by analyzing customer emotional information in real time and providing personalized and appropriate responses based on that information.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices in the virtual space, means for managing user account information and granting access rights to authenticated users, and means for generating a user avatar in the virtual space and operating the avatar to move to the virtual office. This allows the user to access different consultation booths in the store through the avatar, and when entering the details of their consultation, they can receive personalized responses accompanied by facial expression and voice analysis using an emotion analysis engine.

[0456] The "metaverse space" is a three-dimensional digital space created using virtual reality technology, in which users can interact through avatars.

[0457] A "virtual office" is a virtual area with a specific purpose or function that is located within the metaverse space, and is a virtual space where users can carry out activities and consultations within their respective areas.

[0458] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and extracts emotional information in real time, and includes an emotion recognition and analysis system.

[0459] An "avatar" is a digital alter ego that represents and controls a user in a virtual space, and is an agent that represents the user's actions and intentions.

[0460] "AI means" refers to a means that uses artificial intelligence technology to analyze the content of inquiries received from users and generate appropriate answers and countermeasures.

[0461] "Emotion information" is data relating to the user's mental and emotional state obtained as a result of analysis by the emotion engine.

[0462] A "personalized response" is an appropriate answer or response tailored to an individual user, generated based on the user's specific inquiry content and emotional information.

[0463] "Consultation history" is a record of the consultation content and responses made by the user, and is data stored in a database for future learning and improvement.

[0464] MODE FOR CARRYING OUT THE INVENTION

[0465] The present invention aims to provide an AI system that uses the metaverse space and includes emotion analysis for the purpose of customer support in brick-and-mortar stores. Specific embodiments of the system are described below.

[0466] (server)

[0467] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as customer support booths and information booths. The server also manages user account information and grants appropriate access privileges to authenticated users.

[0468] (Terminal)

[0469] When a customer uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0470] (Log in to the virtual space and start using it)

[0471] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0472] (AI consultation service and use of emotion engine)

[0473] The user controls an avatar to access a specific virtual office (e.g., a customer support booth). After selecting a consultation booth, a form for entering the details of the consultation is displayed. A video camera and microphone are installed on the terminal to recognize the user's facial expressions and voice. The terminal simultaneously transmits the user's facial expressions and voice to an emotion analysis engine along with the consultation details entered by the user. The emotion engine analyzes the user's emotions in real time. The AI ​​system on the server generates an appropriate answer based on the consultation details and emotional information received. This generated answer is sent from the server to the terminal and displayed to the user. This allows the user to receive emotionally appropriate support.

[0474] (Emotion log and follow-up)

[0475] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0476] Specific examples

[0477] For example, if a user asks, "How do I get a refund for this product?" and their face recorded by the camera shows anxiety, the emotion engine will analyze the anxiety and the AI ​​will provide polite instructions such as, "Click here for the refund procedure. If you have any questions, our staff will be happy to assist you."

[0478] Prompt Sentence Examples

[0479] "We will develop an AI consultation assistant that customers can use in physical stores via their smartphones. We will use an emotion engine to analyze customer emotions and enable the AI ​​to provide appropriate responses."

[0480] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0481] Step 1:

[0482] The application is launched on the device and the user enters their login information.

[0483] Input: Username and Password.

[0484] Process: Sends login information to the server for authentication.

[0485] Output: The authentication result (success or failure).

[0486] The server checks the received login information against its database and, if successful, grants the user access.

[0487] Step 2:

[0488] For users who are successfully authenticated, the server generates an avatar in the metaverse space.

[0489] Input: User credentials.

[0490] Processing: Avatar generation processing.

[0491] Output: Avatar information in virtual space.

[0492] The server generates an avatar in the metaverse space for the authenticated user and sends that information to the terminal, making it operable.

[0493] Step 3:

[0494] The user operates an avatar to go to the virtual office and select a specific consultation booth.

[0495] Input: User operation information.

[0496] Processing: Movement processing within virtual space.

[0497] Output: Booth selection information.

[0498] Based on the user's operation, the terminal moves the avatar within the virtual space and transmits information about the selected booth to the server.

[0499] Step 4:

[0500] The user inputs the details of the consultation and transmits facial expression and voice data from the terminal.

[0501] Input: Consultation content, facial expression data, and voice data.

[0502] Processing: Sending input data.

[0503] Output: Sending data to the server.

[0504] The device sends the user's consultation details, facial expression data captured by a video camera, and audio data captured by a microphone to an emotion analysis engine.

[0505] Step 5:

[0506] The emotion engine analyzes facial expressions and voice to generate emotional information.

[0507] Input: Facial expression data, audio data.

[0508] Processing: Sentiment analysis processing.

[0509] Output: Emotional information.

[0510] The emotion engine analyzes the received data, extracts the user's emotional state in real time, and sends that information to the AI ​​system.

[0511] Step 6:

[0512] The AI ​​system generates an appropriate answer based on the content of the consultation and emotional information.

[0513] Input: Consultation content, emotional information.

[0514] Processing: Answer generation processing.

[0515] Output: Answer content.

[0516] The AI ​​means on the server analyzes the consultation content and emotional information, generates answers and follow-up questions, and sends the information to the terminal.

[0517] Step 7:

[0518] The device displays the generated answers to the user and prompts for additional questions or information as needed.

[0519] Input: The response from the AI ​​method.

[0520] Processing: Generate an interface for displaying and inputting answers.

[0521] Output: Display the answer to the user.

[0522] The terminal displays the received answers to the user and generates an interface to await further input from the user.

[0523] Step 8:

[0524] The server stores the consultation details and emotion logs in a database.

[0525] Input: Consultation content, emotion log.

[0526] Processing: Data storage processing.

[0527] Output: Information saved to the database.

[0528] The server stores the consultation details and emotional information in a database and uses it for future learning.

[0529] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0531] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0532] [Second embodiment]

[0533] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0534] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0536] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0540] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0541] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0542] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0543] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0544] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0545] The present invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail below.

[0546] Creating and managing metaverse spaces

[0547] server:

[0548] The server first generates a metaverse space and places multiple virtual offices within the virtual space. This allows users to access locations for various tasks and consultations. Each virtual office corresponds to a department (e.g., human resources, compliance), and each has its own unique interactive functions. The server also manages user account information and controls access privileges by authenticating users. This ensures that only appropriate users can access each virtual office.

[0549] User Account Settings and Login Authentication

[0550] Device:

[0551] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0552] Log in to the virtual space and start using it

[0553] server:

[0554] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0555] Device:

[0556] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0557] Consultation at AI help desk

[0558] User:

[0559] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation.

[0560] Device:

[0561] The device sends the consultation details entered by the user to the AI. The input method is primarily text input, but other methods such as voice input are also supported.

[0562] server:

[0563] The AI ​​on the server analyzes the received inquiry and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0564] User:

[0565] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[0566] Follow-up on consultation results and history management

[0567] server:

[0568] The server stores all consultation history in a database and uses it for future learning, which is an important step to improve the AI's response accuracy and customer satisfaction.

[0569] Specific examples

[0570] Scenario 1: Harassment consultation

[0571] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[0572] 2. The device displays a consultation form, and the user enters the specific details of the harassment they wish to discuss.

[0573] 3. The server receives the consultation content, and the AI ​​analyzes it and provides solutions.

[0574] 4. The device displays the AI's answer, and the user decides on the next action.

[0575] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, enabling early detection and response to problems.

[0576] The processing flow will be explained below.

[0577] Step 1:

[0578] The server creates a metaverse space and places multiple virtual offices within it, each containing multiple areas used for different purposes, such as the human resources department and the compliance department.

[0579] Step 2:

[0580] The server manages user account information and stores it in a database, including user IDs, passwords, and access privileges.

[0581] Step 3:

[0582] The terminal displays a screen for the user to log in and prompts the user to enter their ID and password.

[0583] Step 4:

[0584] The user enters login information (ID and password) and clicks the login button.

[0585] Step 5:

[0586] The terminal transmits the entered login information to the server.

[0587] Step 6:

[0588] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[0589] Step 7:

[0590] The server generates an avatar for the authenticated user and places it in the virtual space, as well as setting the user's initial position and viewpoint.

[0591] Step 8:

[0592] The device renders and displays the user's avatar and the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0593] Step 9:

[0594] The user controls the avatar and heads to the desired virtual office (e.g., the compliance department).

[0595] Step 10:

[0596] When the user arrives at the virtual office, the terminal displays a form for inputting the details of the consultation.

[0597] Step 11:

[0598] Users enter their complaint in text and click the send button, which includes details of the specific harassment and other compliance issues.

[0599] Step 12:

[0600] The terminal transmits the input consultation content to the server.

[0601] Step 13:

[0602] The AI ​​on the server analyzes the received consultation content and generates appropriate answers and measures, using natural language processing technology.

[0603] Step 14:

[0604] The server sends the generated response to the terminal.

[0605] Step 15:

[0606] The device displays the AI's answer to the user, who can then ask further questions or add new information based on the answer.

[0607] Step 16:

[0608] The server stores all consultation history in a database, which is used for future learning and analysis.

[0609] Example 1

[0610] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] Conventional consultation systems do not adequately provide an environment for quickly and accurately resolving users' problems. In particular, they lack a flexible system that can handle a wide range of consultation content while maintaining anonymity. Furthermore, because consultation history is not accumulated, it is difficult to make continuous improvements based on past consultation content. This makes it difficult to improve user satisfaction, and efficient problem-solving is required.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0613] In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices within the virtual space, means for managing user account information and granting access rights to authenticated users, means for generating a user avatar within the virtual space and manipulating the avatar to move to the virtual office, means for receiving consultation content from the user's input, analyzing it using a generative AI model, and generating an appropriate answer, and means for saving consultation history in a database and using it for future learning. This allows users to freely consult while maintaining anonymity, and makes it possible to accumulate history and improve the accuracy of the AI.

[0614] A "metaverse space" is a digital space created using virtual reality technology, where users can move freely beyond physical constraints.

[0615] A "virtual office" is a virtual work area located within the metaverse space that corresponds to a specific business department. Users can consult and process business matters within this area.

[0616] "User account information" is a series of data (e.g., username, password, job title, etc.) used for personal authentication of users and access control.

[0617] "Authentication" is the process of verifying that a user has valid authority based on the entered user information.

[0618] "Access Privileges" means the authorization for a User to access certain resources or functions within the System.

[0619] An "avatar" is a character that represents a user in a virtual space, and is a digital entity that reflects the user's actions and behavior.

[0620] A "generative AI model" is an artificial intelligence system that uses natural language processing and machine learning techniques to analyze input data from users and generate appropriate responses.

[0621] "Consultation content" is detailed information about a question or problem that a user inputs to a help desk in the virtual space.

[0622] The "consultation history" is a record of all consultations that a user has made with the system, including the contents of those consultations and the responses to those consultations, and is saved for future reference and analysis.

[0623] A "database" is a system for storing data in a structured digital format, allowing for efficient retrieval and management.

[0624] This invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail.

[0625] Creating and managing metaverse spaces

[0626] server:

[0627] The server generates a metaverse space using game engines such as Unity or Unreal Engine. Multiple virtual offices are placed within the metaverse space. Each virtual office corresponds to a specific business department, such as the human resources department or the compliance department, and each has its own interactive functions. The server also manages user account information and performs authentication to enforce access control.

[0628] User Account Settings and Login Authentication

[0629] Device:

[0630] When a user uses the system, they first launch a dedicated application on their device. The user enters their ID and password on the login screen, and the authentication information is sent to the server. The server verifies the authentication information using a database, and if authentication is successful, the device is given access to the metaverse space.

[0631] Log in to the virtual space and start using it

[0632] server:

[0633] Once the user has been authenticated, the server generates an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar and head to the desired virtual office.

[0634] Device:

[0635] The device renders and displays the virtual space in real time based on the data received from the server. The virtual space is created using game engines such as Unity and Unreal Engine, allowing users to explore seamlessly.

[0636] Consultation at AI help desk

[0637] User:

[0638] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and enters the details of the consultation.

[0639] Device:

[0640] The device sends the consultation details entered by the user to the generation AI model. Input methods supported include text and voice input.

[0641] server:

[0642] A generative AI model (e.g., OpenAI's GPT-3) on the server analyzes the prompt entered by the user and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0643] Follow-up on consultation results and history management

[0644] server:

[0645] The server stores all consultation history in a database, which is used for future training of the AI ​​model to improve response accuracy.

[0646] Specific examples

[0647] Scenario 1: Harassment consultation

[0648] 1. The user launches the dedicated app and enters their login information into the device.

[0649] 2. The server authenticates the login information and provides the user with access to the metaverse space.

[0650] 3. After logging in, the terminal displays the virtual space, and the user operates the avatar to move to the "Compliance Department."

[0651] 4. The user enters the details of their consultation at the consultation booth, and the device sends the data to the generative AI model.

[0652] 5. The server's generated AI model analyzes the consultation content and generates appropriate countermeasures.

[0653] 6. The device displays the generated answer, and the user decides on the next action based on it.

[0654] Example prompt sentence:

[0655] "I would like to consult about measures to deal with harassment. To be more specific, my boss has been verbally abusing me. Could you please give me some advice on how I should respond?"

[0656] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and supports early detection and response to problems.

[0657] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0658] Step 1:

[0659] Metaverse space generation and virtual office layout

[0660] server:

[0661] The server generates the metaverse space using Unity or Unreal Engine. First, a base environment for the virtual space is set up, and then multiple virtual offices are placed within it. The virtual offices are placed based on predefined coordinate data. Different interactive elements are also placed in each virtual office. For example, the HR department can provide an employee consultation service, and the compliance department can provide legal advice.

[0662] Input: Virtual space base setting data, virtual office metadata

[0663] Output: Location information within the metaverse space

[0664] Specific behavior:

[0665] The server runs the virtual space generation script and loads the layout information for each virtual office, completing the basic structure of the virtual space.

[0666] Step 2:

[0667] User Account Settings and Login Authentication

[0668] Device:

[0669] The user starts the dedicated application and enters their ID and password on the login screen. This authentication information is sent to the server as an HTTP request.

[0670] server:

[0671] The server uses a database to verify the submitted credentials, and if successful, stores the user's account information in the session and grants them access to the metaverse space.

[0672] Input: User ID and password

[0673] Output: Authentication result, session ID

[0674] Specific behavior:

[0675] The server checks the authentication information against the database, and if authentication is successful, generates a session ID and sends it to the terminal.

[0676] Step 3:

[0677] Log in to the virtual space and generate an avatar

[0678] server:

[0679] Once a user has been authenticated, the server generates an avatar in the virtual space. This avatar is a character that reflects the user's actions, and its appearance and functions are set based on the user's profile data.

[0680] Device:

[0681] The device renders and displays the virtual space in real time based on the avatar and virtual space information received from the server, allowing the user to begin activities within the virtual space.

[0682] Input: User credentials, profile data

[0683] Output: Avatar information in virtual space, virtual space rendering information

[0684] Specific behavior:

[0685] The server runs an avatar generation script based on the user's profile information and sends rendering data to the device, which then receives the data and renders the virtual space.

[0686] Step 4:

[0687] Consultation at AI help desk

[0688] User:

[0689] The user controls an avatar to move to a specific virtual office, select a consultation booth, and enter the details of their consultation. The input method is primarily text, but voice input is also supported.

[0690] Device:

[0691] The device converts the consultation content entered by the user into a text prompt and sends it to the generative AI model.

[0692] server:

[0693] A generative AI model (e.g., GPT-3) on the server analyzes the received prompt and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[0694] Input: User's inquiry (prompt text)

[0695] Output: Answer from the generative AI model

[0696] Specific behavior:

[0697] The user inputs the content of their inquiry using a keyboard or voice input, and the device sends this as an HTTP request to the server. The server then analyzes the content of the inquiry using a generative AI model, generates a response, and sends it to the device.

[0698] Step 5:

[0699] Follow-up on consultation results and history management

[0700] server:

[0701] The server stores all consultation history in a database, including the user's consultation content and the answers provided by the generative AI model. This data will be used as training data for future projects.

[0702] Input: User's inquiry, response from the generated AI model

[0703] Output: Consultation history data

[0704] Specific behavior:

[0705] The server stores the received inquiries and generated responses in a log file or database for future reference and analysis.

[0706] This system provides an environment where users can easily consult with the system, and also allows historical data to be used to improve the performance of the AI.

[0707] (Application example 1)

[0708] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] Maintenance and inspection work in factories takes a lot of time and effort, and responding to machine abnormalities and breakdowns in particular requires advanced expertise. Conventional systems make it difficult to provide immediate on-site responses or appropriate solutions. As a result, efficient production is often hindered. Furthermore, differences in the knowledge and experience of technicians can lead to variations in response quality, reducing overall operational efficiency. To solve these issues, a system that supports on-site maintenance and inspection work is needed.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0711] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; generation AI means for receiving consultation content from the user, analyzing it, and generating an appropriate answer; means for storing the consultation history in a database and using it for future learning; AI means for guiding procedures to support maintenance and inspection work in factories and providing solutions when problems occur; and means for factory employees and engineers to perform maintenance and inspection work in the virtual space using smartphones, smart glasses, or head-mounted displays. This enables immediate response and provision of appropriate solutions on-site, improving work efficiency throughout the factory.

[0712] The "metaverse space" is a virtual space that exists on the Internet, where users can interact with each other through avatars.

[0713] A "virtual office" is a virtual work location located within the metaverse space that provides the same functions as a real office.

[0714] "User account information" is information for uniquely identifying a user, and includes authentication information such as a login ID and password.

[0715] "Access privileges" refer to the rights of an authenticated user to access specific resources or functions within a system.

[0716] An "avatar" is a character that represents a user, moves within a virtual space, and interacts with other users and the system.

[0717] "Generative AI means" refers to artificial intelligence technology that analyzes user input and automatically generates appropriate answers and countermeasures.

[0718] "Consultation history" refers to a record of the consultations users make through the system, as well as the responses and countermeasures taken.

[0719] "Procedures to support maintenance and inspection work" are guidelines to instruct and support procedures for efficiently carrying out maintenance and inspection of machinery and equipment within a factory.

[0720] A "smartphone" is a portable information terminal that can connect to the Internet and use a variety of applications.

[0721] "Smart glasses" are a wearable eyeglass-type device equipped with information display and interactive functions.

[0722] A "head-mounted display" is a display device worn on the head, used to display a virtual space to the user.

[0723] "Factory employees" are workers who operate machines and perform maintenance and inspection work within a factory.

[0724] An "engineer" is a technician who designs, develops, maintains, and inspects machinery and equipment.

[0725] This invention is an AI-enabled metaverse system to support maintenance and inspection work in factories. This system has the function of placing multiple virtual offices in the metaverse space, managing user account information, and granting appropriate access rights through authentication. The details are explained below.

[0726] Creating and managing metaverse spaces

[0727] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices correspond to areas where maintenance and inspection work is carried out in the factory, and users can access them to carry out their work efficiently.

[0728] User Account Settings and Login Authentication

[0729] When a user uses the system, they first launch the application on their device (smartphone, smart glasses, head-mounted display) and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0730] Log in to the virtual space and start using it

[0731] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The user controls this avatar to move to the area where maintenance and inspection work will be performed. The device renders and displays the virtual space in real time based on the data received from the server.

[0732] Supporting maintenance and inspection work with AI tools

[0733] Users can control an avatar to move near specific equipment or machinery and begin maintenance or inspection tasks. The server-based AI generator analyzes the user's questions and problem reports and presents appropriate solutions and procedures. The following are examples of specific prompts:

[0734] Engineer: "I'm hearing an abnormal noise coming from the machine. What measures should I take?"

[0735] AI: "Please first perform a frequency analysis of the machine to identify the cause of the abnormal noise. If that doesn't solve the problem, please try the steps below."

[0736] Consultation history storage and follow-up

[0737] The server stores all inquiries and answers in a database for future learning, and the generative AI method provides continuous follow-up questions and solutions based on past history, allowing users to solve their problems effectively.

[0738] Program processing explanation

[0739] The server is operated as a FastAPI-based API server on a cloud service (e.g., AWS, Azure). Internet-enabled devices (smartphones, smart glasses, head-mounted displays) communicate with the server and render data in the virtual space. The generative AI means on the server analyzes the user's input, generates appropriate answers, and sends the answers to the user's device in real time. Using a specific generative AI model (e.g., GPT-4) enables more accurate responses.

[0740] In this way, the system of the present invention effectively supports maintenance and inspection work within a factory and contributes to improving work efficiency.

[0741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] The server generates a metaverse space and places multiple virtual offices within the virtual space. This sets up the business areas and consultation booths that users can access. The input is data on how to arrange the virtual offices and business areas as server configuration information, and the output is the metaverse space that users can access.

[0744] Step 2:

[0745] A user turns on a device (smartphone, smart glasses, or head-mounted display) and enters login information. The device sends this information to a server, which then authenticates the user's account information. The input is the user's authentication information (login ID and password), and the output is the success or failure of the authentication.

[0746] Step 3:

[0747] If authentication is successful, the server generates an avatar for the user in the virtual space. The device renders the virtual space based on the data received from the server and displays it to the user. The input is the user's account information and access privileges, and the output is the avatar display in the virtual space.

[0748] Step 4:

[0749] The user controls the avatar to move to a specific virtual office (maintenance and inspection area). The terminal reflects the user's operations in real time and updates the avatar's position. The input is the user's operation command (movement instruction), and the output is the avatar's new position information.

[0750] Step 5:

[0751] When a user moves an avatar near a specific machine or equipment to begin maintenance or inspection work, the device sends the user's question or problem details to the server. The input is the user's question or problem report, and the output is a receipt confirmation from the server.

[0752] Step 6:

[0753] The generative AI means on the server analyzes the received user question and generates an appropriate answer and response procedure. This uses a generative AI model such as GPT-4. The input is the user question, and the output is the generated answer and response procedure.

[0754] Step 7:

[0755] The server sends the generated answers and response procedures to the user's device. The device displays the received answers and response procedures to the user. The input is the answer data from the generation AI means, and the output is the content displayed on the user's device.

[0756] Step 8:

[0757] The user's consultation history and response procedures are stored in a database on the server. This allows them to be used for future consultations and problem solving. The input is the user's consultation content and the response results of the generated AI, and the output is the data stored in the database.

[0758] Step 9:

[0759] The server proposes continuous follow-up questions and solutions based on past history and provides them to the user. The input is past consultation history data, and the output is follow-up questions and solution suggestions.

[0760] The purpose of these processing steps is to ensure that maintenance and inspection work within the factory is carried out efficiently and to provide prompt support for users in resolving problems.

[0761] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0762] The present invention combines an AI consultation system that uses the metaverse space with an emotion engine, and the embodiments thereof will be described in detail below.

[0763] Creating and managing metaverse spaces

[0764] server:

[0765] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access each virtual office.

[0766] User Account Settings and Login Authentication

[0767] Device:

[0768] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0769] Log in to the virtual space and start using it

[0770] server:

[0771] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0772] Device:

[0773] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[0774] Consultation at AI help desk and use of emotion engine

[0775] User:

[0776] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation. The device is equipped with a video camera and microphone to recognize the user's facial expressions and voice.

[0777] Device:

[0778] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[0779] server:

[0780] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. It automatically generates follow-up questions and countermeasures according to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[0781] User:

[0782] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[0783] Emotion Log and Follow-up

[0784] server:

[0785] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0786] Specific examples

[0787] Scenario 1: Harassment consultation and sentiment analysis

[0788] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[0789] 2. The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[0790] 3. The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[0791] 4. The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional follow-up questions.

[0792] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and by using an emotion engine, more personalized responses are possible.

[0793] The processing flow will be explained below.

[0794] Step 1:

[0795] The server generates a metaverse space and places multiple virtual offices within it, each of which is set up as an area for accepting specific tasks or consultations.

[0796] Step 2:

[0797] The server registers user account information in a database and manages authentication information, including user ID, password, and access privileges.

[0798] Step 3:

[0799] The terminal displays a login screen for the user to use the system.

[0800] Step 4:

[0801] The user enters their ID and password on the login screen and clicks the login button.

[0802] Step 5:

[0803] The terminal transmits the entered login information to the server.

[0804] Step 6:

[0805] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[0806] Step 7:

[0807] The server generates an avatar for the authenticated user and places it in the virtual space, setting the avatar's initial position and viewpoint.

[0808] Step 8:

[0809] The device renders and displays the virtual space based on the data received from the server, and the user can control the avatar in real time and explore the virtual space.

[0810] Step 9:

[0811] The user controls the avatar to access the desired virtual office (e.g., the compliance department).

[0812] Step 10:

[0813] When the user arrives at the virtual office, the terminal activates the emotion engine, activates the video camera and microphone to transmit the user's facial expressions and voice to the emotion engine, and displays a form for inputting the details of the consultation.

[0814] Step 11:

[0815] The user enters the content of the consultation in text and clicks the send button.

[0816] Step 12:

[0817] The device sends the input consultation content and facial expression and voice data obtained from the emotion engine to the server.

[0818] Step 13:

[0819] The AI ​​on the server analyzes the received consultation content and emotional information, generates an appropriate response, and adjusts follow-up questions and countermeasures based on the emotional information according to the user's emotional state.

[0820] Step 14:

[0821] The server sends the generated answers and follow-up questions to the terminal.

[0822] Step 15:

[0823] The device displays the AI's answers and follow-up questions to the user.

[0824] Step 16:

[0825] The user reviews the AI's answers and enters additional questions or information as needed.

[0826] Step 17:

[0827] The server stores all consultation history and emotion logs in a database, which allows future learning and tracking of users' long-term emotional changes.

[0828] Through the above process, users can feel safe consulting within the metaverse space, and by utilizing the emotion engine, more personalized responses can be provided.

[0829] Example 2

[0830] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0831] In modern companies and organizations, it is important to provide an environment where employees can seek advice anonymously and with peace of mind. However, current systems make it difficult to accurately grasp the user's emotions based on the text of the consultation alone, making it difficult to respond appropriately. In addition, there is a lack of follow-up and personalized responses in response to changes in employees' emotions, so improvements are needed to increase employee satisfaction.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0833] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; AI means for receiving consultation content from the user, analyzing it, and generating an appropriate response; emotion engine means for recognizing and analyzing the user's facial expressions and voice; means for the AI ​​means to generate follow-up questions and countermeasures according to the emotion information obtained from the emotion engine; and means for storing consultation history and emotion logs in a database and using them for future learning. This allows for real-time analysis of user emotions and enables appropriate responses and follow-ups based on the analysis. This provides an environment where employees can consult with confidence and improves employee satisfaction.

[0834] The "metaverse space" is a digital virtual space generated using virtual reality technology, in which users can control avatars and carry out various activities.

[0835] A "virtual office" is a virtual office located within the metaverse space, where activities for specific purposes, such as the human resources department or compliance department, are carried out.

[0836] "User account information" refers to information for identifying a user, including a user name, password, email address, and the like.

[0837] The "means for granting access rights" is a mechanism that grants the right to access a specific virtual office or a specific area within the metaverse space when a user is authenticated.

[0838] An "avatar" is a virtual character that represents a user within the metaverse space, and moves and acts within the virtual space according to the user's instructions.

[0839] "AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of analyzing the content of inquiries from users and generating appropriate answers.

[0840] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to evaluate their current emotional state in real time.

[0841] The "means for generating follow-up questions and countermeasures" is a mechanism that continuously asks questions to the user and provides appropriate countermeasures based on information obtained from the emotion engine.

[0842] "Consultation history" is a record of the content of the consultation a user has had and the responses to those consultations, and is data that is saved for use in future learning and analysis.

[0843] An "emotion log" is data that records the user's emotional state during a consultation, and serves as the basis for providing a more personalized response the next time the user consults.

[0844] The present invention combines an AI consultation system that utilizes the metaverse space with an emotion engine, and the following describes its implementation in detail. The entire system is composed of the main elements of a server, a terminal, and a user.

[0845] Creating and managing metaverse spaces

[0846] server:

[0847] The server first generates a virtual metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access specific virtual offices.

[0848] User Account Settings and Login Authentication

[0849] Device:

[0850] When a user uses the system, they first launch a dedicated application on their device and enter their login information. The entered information is sent from the device to the server, and once authenticated by the server, the user is able to access the metaverse space.

[0851] Log in to the virtual space and start using it

[0852] server:

[0853] Once a user has been authenticated, an avatar is generated and displayed in the virtual space. The avatar is a character that represents the user and can move freely within the virtual space.

[0854] Device:

[0855] The device renders the virtual space based on the data received from the server and displays it to the user, allowing the user to explore the virtual space in real time.

[0856] Consultation at AI help desk and use of emotion engine

[0857] User:

[0858] The user controls an avatar to access a specific virtual office (e.g., the compliance department) and selects a consultation booth. A form for entering the details of the consultation is displayed. The terminal's video camera and microphone recognize the user's facial expressions and voice.

[0859] Device:

[0860] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[0861] server:

[0862] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. Follow-up questions and countermeasures are also automatically generated in response to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[0863] User:

[0864] Based on the answers provided by the AI, users can continue to ask questions or add new information, and can also move to another virtual office for another consultation if necessary.

[0865] Emotion Log and Follow-up

[0866] server:

[0867] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0868] Specific examples

[0869] Scenario 1: Harassment consultation and sentiment analysis

[0870] Users log in and access the "Compliance Department" within the virtual space.

[0871] The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[0872] The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[0873] The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional questions or follow-ups.

[0874] Prompt Sentence Examples

[0875] "Please tell me more about the harassment you recently experienced. Can you describe the specific situation and how it made you feel?"

[0876] In this way, according to the embodiment of the invention, it is possible to analyze the user's emotions in real time and provide appropriate responses and follow-ups based on the analysis. This system provides an environment where users can feel at ease when seeking advice, and realizes more personalized responses.

[0877] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0878] Step 1:

[0879] Device:

[0880] The user starts the application and enters account information (user name, password) on the login screen.

[0881] Input: Username, Password

[0882] Action: A user enters login information at a terminal.

[0883] Output: The entered login information

[0884] Step 2:

[0885] Device:

[0886] The entered login information is sent to the server.

[0887] Input: The login information entered

[0888] How it works: The device sends the login information to the server.

[0889] Output: Sending login information to the server

[0890] Step 3:

[0891] server:

[0892] The received account information is compared with information in the database to verify that the account exists.

[0893] Input: Login information sent from the device

[0894] Action: The server checks against the database.

[0895] Output: Authentication result (success / failure)

[0896] Step 4:

[0897] server:

[0898] If authentication is successful, the user's session information is generated and sent back to the terminal. If authentication fails, an error message is sent back to the terminal.

[0899] Input: Authentication result (success / failure), user account information

[0900] Action: The server generates session information or an error message.

[0901] Output: Session information (if successful), error message (if unsuccessful)

[0902] Step 5:

[0903] Device:

[0904] The terminal receives a message indicating successful authentication and provides the user with access to the metaverse space.

[0905] Input: Session information (if successful)

[0906] Action: The device displays the access interface to the metaverse space.

[0907] Output: Access rights to the metaverse space

[0908] Step 6:

[0909] server:

[0910] Once authenticated, an avatar is generated and displayed in the virtual space for the user. Multiple virtual offices are placed within the virtual space.

[0911] Input: User credentials, Virtual Office configuration data

[0912] How it works: The server generates an avatar and a virtual office.

[0913] Output: Avatar information, virtual office layout data

[0914] Step 7:

[0915] Device:

[0916] The virtual space is rendered based on the data received from the server and displayed to the user.

[0917] Input: Avatar information, virtual office layout data

[0918] How it works: The device renders the virtual space in real time.

[0919] Output: Visual information in virtual space

[0920] Step 8:

[0921] User:

[0922] The user operates an avatar, accesses a specific virtual office, selects a consultation booth, and inputs the details of the consultation.

[0923] Input: User operation (moving the avatar and selecting a consultation booth), consultation content

[0924] Operation: The user enters the details of the consultation at the consultation booth.

[0925] Output: Input consultation details

[0926] Step 9:

[0927] Device:

[0928] The system uses a video camera and microphone to capture the user's facial expressions and voice, and then sends the consultation details and the captured facial expression and voice data to the server.

[0929] Input: consultation details, facial expressions, and voice data

[0930] How it works: The device captures facial expressions and voice and sends them to the server.

[0931] Output: Send data to the server

[0932] Step 10:

[0933] server:

[0934] The received consultation content and user emotional information are sent to the emotion engine for analysis. The emotion engine analyzes the user's emotions in real time and obtains the results.

[0935] Input: consultation details, emotional information

[0936] Operation: The server sends data to the emotion engine and obtains the analysis results.

[0937] Output: Emotion analysis results

[0938] Step 11:

[0939] server:

[0940] Based on the emotional information and the content of the consultation, a generative AI model is used to generate an appropriate response. Follow-up questions and countermeasures are automatically generated according to changes in emotions. The generated response is then sent to the device.

[0941] Input: Consultation details, emotion analysis results

[0942] Operation: The server generates appropriate answers and follow-up questions and sends them to the device.

[0943] Output: Generated answers, follow-up questions

[0944] Step 12:

[0945] Device:

[0946] The received response is displayed to the user.

[0947] Input: Generated answers, follow-up questions

[0948] Action: The device displays the answer.

[0949] Output: Display the answer to the user

[0950] Step 13:

[0951] User:

[0952] Based on the answers provided by the AI, users can ask further questions or add new information, and if necessary, they can move to another virtual office for another consultation.

[0953] Input: AI response

[0954] Action: The user enters additional questions or information, or navigates to another virtual office.

[0955] Output: Additional questions, information, and next steps

[0956] Step 14:

[0957] server:

[0958] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of AI responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0959] Input: consultation history, emotion log

[0960] What happens: The server saves the data to the database.

[0961] Output: Saved data log

[0962] (Application example 2)

[0963] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] The purpose of this invention is to provide an environment where customers can easily and quickly ask questions or make complaints in a physical store. In particular, it aims to improve customer satisfaction by analyzing customer emotional information in real time and providing personalized and appropriate responses based on that information.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices in the virtual space, means for managing user account information and granting access rights to authenticated users, and means for generating a user avatar in the virtual space and operating the avatar to move to the virtual office. This allows the user to access different consultation booths in the store through the avatar, and when entering the details of their consultation, they can receive personalized responses accompanied by facial expression and voice analysis using an emotion analysis engine.

[0966] The "metaverse space" is a three-dimensional digital space created using virtual reality technology, in which users can interact through avatars.

[0967] A "virtual office" is a virtual area with a specific purpose or function that is located within the metaverse space, and is a virtual space where users can carry out activities and consultations within their respective areas.

[0968] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and extracts emotional information in real time, and includes an emotion recognition and analysis system.

[0969] An "avatar" is a digital alter ego that represents and controls a user in a virtual space, and is an agent that represents the user's actions and intentions.

[0970] "AI means" refers to a means that uses artificial intelligence technology to analyze the content of inquiries received from users and generate appropriate answers and countermeasures.

[0971] "Emotion information" is data relating to the user's mental and emotional state obtained as a result of analysis by the emotion engine.

[0972] A "personalized response" is an appropriate answer or response tailored to an individual user, generated based on the user's specific inquiry content and emotional information.

[0973] "Consultation history" is a record of the consultation content and responses made by the user, and is data stored in a database for future learning and improvement.

[0974] MODE FOR CARRYING OUT THE INVENTION

[0975] The present invention aims to provide an AI system that uses the metaverse space and includes emotion analysis for the purpose of customer support in brick-and-mortar stores. Specific embodiments of the system are described below.

[0976] (server)

[0977] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as customer support booths and information booths. The server also manages user account information and grants appropriate access privileges to authenticated users.

[0978] (Terminal)

[0979] When a customer uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[0980] (Log in to the virtual space and start using it)

[0981] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[0982] (AI consultation service and use of emotion engine)

[0983] The user controls an avatar to access a specific virtual office (e.g., a customer support booth). After selecting a consultation booth, a form for entering the details of the consultation is displayed. A video camera and microphone are installed on the terminal to recognize the user's facial expressions and voice. The terminal simultaneously transmits the user's facial expressions and voice to an emotion analysis engine along with the consultation details entered by the user. The emotion engine analyzes the user's emotions in real time. The AI ​​system on the server generates an appropriate answer based on the consultation details and emotional information received. This generated answer is sent from the server to the terminal and displayed to the user. This allows the user to receive emotionally appropriate support.

[0984] (Emotion log and follow-up)

[0985] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[0986] Specific examples

[0987] For example, if a user asks, "How do I get a refund for this product?" and their face recorded by the camera shows anxiety, the emotion engine will analyze the anxiety and the AI ​​will provide polite instructions such as, "Click here for the refund procedure. If you have any questions, our staff will be happy to assist you."

[0988] Prompt Sentence Examples

[0989] "We will develop an AI consultation assistant that customers can use in physical stores via their smartphones. We will use an emotion engine to analyze customer emotions and enable the AI ​​to provide appropriate responses."

[0990] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0991] Step 1:

[0992] The application is launched on the device and the user enters their login information.

[0993] Input: Username and Password.

[0994] Process: Sends login information to the server for authentication.

[0995] Output: The authentication result (success or failure).

[0996] The server checks the received login information against its database and, if successful, grants the user access.

[0997] Step 2:

[0998] For users who are successfully authenticated, the server generates an avatar in the metaverse space.

[0999] Input: User credentials.

[1000] Processing: Avatar generation processing.

[1001] Output: Avatar information in virtual space.

[1002] The server generates an avatar in the metaverse space for the authenticated user and sends that information to the terminal, making it operable.

[1003] Step 3:

[1004] The user operates an avatar to go to the virtual office and select a specific consultation booth.

[1005] Input: User operation information.

[1006] Processing: Movement processing within virtual space.

[1007] Output: Booth selection information.

[1008] Based on the user's operation, the terminal moves the avatar within the virtual space and transmits information about the selected booth to the server.

[1009] Step 4:

[1010] The user inputs the details of the consultation and transmits facial expression and voice data from the terminal.

[1011] Input: Consultation content, facial expression data, and voice data.

[1012] Processing: Sending input data.

[1013] Output: Sending data to the server.

[1014] The device sends the user's consultation details, facial expression data captured by a video camera, and audio data captured by a microphone to an emotion analysis engine.

[1015] Step 5:

[1016] The emotion engine analyzes facial expressions and voice to generate emotional information.

[1017] Input: Facial expression data, audio data.

[1018] Processing: Sentiment analysis processing.

[1019] Output: Emotional information.

[1020] The emotion engine analyzes the received data, extracts the user's emotional state in real time, and sends that information to the AI ​​system.

[1021] Step 6:

[1022] The AI ​​system generates an appropriate answer based on the content of the consultation and emotional information.

[1023] Input: Consultation content, emotional information.

[1024] Processing: Answer generation processing.

[1025] Output: Answer content.

[1026] The AI ​​means on the server analyzes the consultation content and emotional information, generates answers and follow-up questions, and sends the information to the terminal.

[1027] Step 7:

[1028] The device displays the generated answers to the user and prompts for additional questions or information as needed.

[1029] Input: The response from the AI ​​method.

[1030] Processing: Generate an interface for displaying and inputting answers.

[1031] Output: Display the answer to the user.

[1032] The terminal displays the received answers to the user and generates an interface to await further input from the user.

[1033] Step 8:

[1034] The server stores the consultation details and emotion logs in a database.

[1035] Input: Consultation content, emotion log.

[1036] Processing: Data storage processing.

[1037] Output: Information saved to the database.

[1038] The server stores the consultation details and emotional information in a database and uses it for future learning.

[1039] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1040] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1041] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1042] [Third embodiment]

[1043] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1044] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1045] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1046] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1047] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1048] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1049] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1050] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1051] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1052] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1053] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1054] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1055] The present invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail below.

[1056] Creating and managing metaverse spaces

[1057] server:

[1058] The server first generates a metaverse space and places multiple virtual offices within the virtual space. This allows users to access locations for various tasks and consultations. Each virtual office corresponds to a department (e.g., human resources, compliance), and each has its own unique interactive functions. The server also manages user account information and controls access privileges by authenticating users. This ensures that only appropriate users can access each virtual office.

[1059] User Account Settings and Login Authentication

[1060] Device:

[1061] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1062] Log in to the virtual space and start using it

[1063] server:

[1064] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[1065] Device:

[1066] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1067] Consultation at AI help desk

[1068] User:

[1069] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation.

[1070] Device:

[1071] The device sends the consultation details entered by the user to the AI. The input method is primarily text input, but other methods such as voice input are also supported.

[1072] server:

[1073] The AI ​​on the server analyzes the received inquiry and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1074] User:

[1075] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[1076] Follow-up on consultation results and history management

[1077] server:

[1078] The server stores all consultation history in a database and uses it for future learning, which is an important step to improve the AI's response accuracy and customer satisfaction.

[1079] Specific examples

[1080] Scenario 1: Harassment consultation

[1081] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[1082] 2. The device displays a consultation form, and the user enters the specific details of the harassment they wish to discuss.

[1083] 3. The server receives the consultation content, and the AI ​​analyzes it and provides solutions.

[1084] 4. The device displays the AI's answer, and the user decides on the next action.

[1085] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, enabling early detection and response to problems.

[1086] The processing flow will be explained below.

[1087] Step 1:

[1088] The server creates a metaverse space and places multiple virtual offices within it, each containing multiple areas used for different purposes, such as the human resources department and the compliance department.

[1089] Step 2:

[1090] The server manages user account information and stores it in a database, including user IDs, passwords, and access privileges.

[1091] Step 3:

[1092] The terminal displays a screen for the user to log in and prompts the user to enter their ID and password.

[1093] Step 4:

[1094] The user enters login information (ID and password) and clicks the login button.

[1095] Step 5:

[1096] The terminal transmits the entered login information to the server.

[1097] Step 6:

[1098] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[1099] Step 7:

[1100] The server generates an avatar for the authenticated user and places it in the virtual space, as well as setting the user's initial position and viewpoint.

[1101] Step 8:

[1102] The device renders and displays the user's avatar and the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1103] Step 9:

[1104] The user controls the avatar and heads to the desired virtual office (e.g., the compliance department).

[1105] Step 10:

[1106] When the user arrives at the virtual office, the terminal displays a form for inputting the details of the consultation.

[1107] Step 11:

[1108] Users enter their complaint in text and click the send button, which includes details of the specific harassment and other compliance issues.

[1109] Step 12:

[1110] The terminal transmits the input consultation content to the server.

[1111] Step 13:

[1112] The AI ​​on the server analyzes the received consultation content and generates appropriate answers and measures, using natural language processing technology.

[1113] Step 14:

[1114] The server sends the generated response to the terminal.

[1115] Step 15:

[1116] The device displays the AI's answer to the user, who can then ask further questions or add new information based on the answer.

[1117] Step 16:

[1118] The server stores all consultation history in a database, which is used for future learning and analysis.

[1119] Example 1

[1120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] Conventional consultation systems do not adequately provide an environment for quickly and accurately resolving users' problems. In particular, they lack a flexible system that can handle a wide range of consultation content while maintaining anonymity. Furthermore, because consultation history is not accumulated, it is difficult to make continuous improvements based on past consultation content. This makes it difficult to improve user satisfaction, and efficient problem-solving is required.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1123] In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices within the virtual space, means for managing user account information and granting access rights to authenticated users, means for generating a user avatar within the virtual space and manipulating the avatar to move to the virtual office, means for receiving consultation content from the user's input, analyzing it using a generative AI model, and generating an appropriate answer, and means for saving consultation history in a database and using it for future learning. This allows users to freely consult while maintaining anonymity, and makes it possible to accumulate history and improve the accuracy of the AI.

[1124] A "metaverse space" is a digital space created using virtual reality technology, where users can move freely beyond physical constraints.

[1125] A "virtual office" is a virtual work area located within the metaverse space that corresponds to a specific business department. Users can consult and process business matters within this area.

[1126] "User account information" is a series of data (e.g., username, password, job title, etc.) used for personal authentication of users and access control.

[1127] "Authentication" is the process of verifying that a user has valid authority based on the entered user information.

[1128] "Access Privileges" means the authorization for a User to access certain resources or functions within the System.

[1129] An "avatar" is a character that represents a user in a virtual space, and is a digital entity that reflects the user's actions and behavior.

[1130] A "generative AI model" is an artificial intelligence system that uses natural language processing and machine learning techniques to analyze input data from users and generate appropriate responses.

[1131] "Consultation content" is detailed information about a question or problem that a user inputs to a help desk in the virtual space.

[1132] The "consultation history" is a record of all consultations that a user has made with the system, including the contents of those consultations and the responses to those consultations, and is saved for future reference and analysis.

[1133] A "database" is a system for storing data in a structured digital format, allowing for efficient retrieval and management.

[1134] This invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail.

[1135] Creating and managing metaverse spaces

[1136] server:

[1137] The server generates a metaverse space using game engines such as Unity or Unreal Engine. Multiple virtual offices are placed within the metaverse space. Each virtual office corresponds to a specific business department, such as the human resources department or the compliance department, and each has its own interactive functions. The server also manages user account information and performs authentication to enforce access control.

[1138] User Account Settings and Login Authentication

[1139] Device:

[1140] When a user uses the system, they first launch a dedicated application on their device. The user enters their ID and password on the login screen, and the authentication information is sent to the server. The server verifies the authentication information using a database, and if authentication is successful, the device is given access to the metaverse space.

[1141] Log in to the virtual space and start using it

[1142] server:

[1143] Once the user has been authenticated, the server generates an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar and head to the desired virtual office.

[1144] Device:

[1145] The device renders and displays the virtual space in real time based on the data received from the server. The virtual space is created using game engines such as Unity and Unreal Engine, allowing users to explore seamlessly.

[1146] Consultation at AI help desk

[1147] User:

[1148] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and enters the details of the consultation.

[1149] Device:

[1150] The device sends the consultation details entered by the user to the generation AI model. Input methods supported include text and voice input.

[1151] server:

[1152] A generative AI model (e.g., OpenAI's GPT-3) on the server analyzes the prompt entered by the user and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1153] Follow-up on consultation results and history management

[1154] server:

[1155] The server stores all consultation history in a database, which is used for future training of the AI ​​model to improve response accuracy.

[1156] Specific examples

[1157] Scenario 1: Harassment consultation

[1158] 1. The user launches the dedicated app and enters their login information into the device.

[1159] 2. The server authenticates the login information and provides the user with access to the metaverse space.

[1160] 3. After logging in, the terminal displays the virtual space, and the user operates the avatar to move to the "Compliance Department."

[1161] 4. The user enters the details of their consultation at the consultation booth, and the device sends the data to the generative AI model.

[1162] 5. The server's generated AI model analyzes the consultation content and generates appropriate countermeasures.

[1163] 6. The device displays the generated answer, and the user decides on the next action based on it.

[1164] Example prompt sentence:

[1165] "I would like to consult about measures to deal with harassment. To be more specific, my boss has been verbally abusing me. Could you please give me some advice on how I should respond?"

[1166] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and supports early detection and response to problems.

[1167] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1168] Step 1:

[1169] Metaverse space generation and virtual office layout

[1170] server:

[1171] The server generates the metaverse space using Unity or Unreal Engine. First, a base environment for the virtual space is set up, and then multiple virtual offices are placed within it. The virtual offices are placed based on predefined coordinate data. Different interactive elements are also placed in each virtual office. For example, the HR department can provide an employee consultation service, and the compliance department can provide legal advice.

[1172] Input: Virtual space base setting data, virtual office metadata

[1173] Output: Location information within the metaverse space

[1174] Specific behavior:

[1175] The server runs the virtual space generation script and loads the layout information for each virtual office, completing the basic structure of the virtual space.

[1176] Step 2:

[1177] User Account Settings and Login Authentication

[1178] Device:

[1179] The user starts the dedicated application and enters their ID and password on the login screen. This authentication information is sent to the server as an HTTP request.

[1180] server:

[1181] The server uses a database to verify the submitted credentials, and if successful, stores the user's account information in the session and grants them access to the metaverse space.

[1182] Input: User ID and password

[1183] Output: Authentication result, session ID

[1184] Specific behavior:

[1185] The server checks the authentication information against the database, and if authentication is successful, generates a session ID and sends it to the terminal.

[1186] Step 3:

[1187] Log in to the virtual space and generate an avatar

[1188] server:

[1189] Once a user has been authenticated, the server generates an avatar in the virtual space. This avatar is a character that reflects the user's actions, and its appearance and functions are set based on the user's profile data.

[1190] Device:

[1191] The device renders and displays the virtual space in real time based on the avatar and virtual space information received from the server, allowing the user to begin activities within the virtual space.

[1192] Input: User credentials, profile data

[1193] Output: Avatar information in virtual space, virtual space rendering information

[1194] Specific behavior:

[1195] The server runs an avatar generation script based on the user's profile information and sends rendering data to the device, which then receives the data and renders the virtual space.

[1196] Step 4:

[1197] Consultation at AI help desk

[1198] User:

[1199] The user controls an avatar to move to a specific virtual office, select a consultation booth, and enter the details of their consultation. The input method is primarily text, but voice input is also supported.

[1200] Device:

[1201] The device converts the consultation content entered by the user into a text prompt and sends it to the generative AI model.

[1202] server:

[1203] A generative AI model (e.g., GPT-3) on the server analyzes the received prompt and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1204] Input: User's inquiry (prompt text)

[1205] Output: Answer from the generative AI model

[1206] Specific behavior:

[1207] The user inputs the content of their inquiry using a keyboard or voice input, and the device sends this as an HTTP request to the server. The server then analyzes the content of the inquiry using a generative AI model, generates a response, and sends it to the device.

[1208] Step 5:

[1209] Follow-up on consultation results and history management

[1210] server:

[1211] The server stores all consultation history in a database, including the user's consultation content and the answers provided by the generative AI model. This data will be used as training data for future projects.

[1212] Input: User's inquiry, response from the generated AI model

[1213] Output: Consultation history data

[1214] Specific behavior:

[1215] The server stores the received inquiries and generated responses in a log file or database for future reference and analysis.

[1216] This system provides an environment where users can easily consult with the system, and also allows historical data to be used to improve the performance of the AI.

[1217] (Application example 1)

[1218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1219] Maintenance and inspection work in factories takes a lot of time and effort, and responding to machine abnormalities and breakdowns in particular requires advanced expertise. Conventional systems make it difficult to provide immediate on-site responses or appropriate solutions. As a result, efficient production is often hindered. Furthermore, differences in the knowledge and experience of technicians can lead to variations in response quality, reducing overall operational efficiency. To solve these issues, a system that supports on-site maintenance and inspection work is needed.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1221] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; generation AI means for receiving consultation content from the user, analyzing it, and generating an appropriate answer; means for storing the consultation history in a database and using it for future learning; AI means for guiding procedures to support maintenance and inspection work in factories and providing solutions when problems occur; and means for factory employees and engineers to perform maintenance and inspection work in the virtual space using smartphones, smart glasses, or head-mounted displays. This enables immediate response and provision of appropriate solutions on-site, improving work efficiency throughout the factory.

[1222] The "metaverse space" is a virtual space that exists on the Internet, where users can interact with each other through avatars.

[1223] A "virtual office" is a virtual work location located within the metaverse space that provides the same functions as a real office.

[1224] "User account information" is information for uniquely identifying a user, and includes authentication information such as a login ID and password.

[1225] "Access privileges" refer to the rights of an authenticated user to access specific resources or functions within a system.

[1226] An "avatar" is a character that represents a user, moves within a virtual space, and interacts with other users and the system.

[1227] "Generative AI means" refers to artificial intelligence technology that analyzes user input and automatically generates appropriate answers and countermeasures.

[1228] "Consultation history" refers to a record of the consultations users make through the system, as well as the responses and countermeasures taken.

[1229] "Procedures to support maintenance and inspection work" are guidelines to instruct and support procedures for efficiently carrying out maintenance and inspection of machinery and equipment within a factory.

[1230] A "smartphone" is a portable information terminal that can connect to the Internet and use a variety of applications.

[1231] "Smart glasses" are a wearable eyeglass-type device equipped with information display and interactive functions.

[1232] A "head-mounted display" is a display device worn on the head, used to display a virtual space to the user.

[1233] "Factory employees" are workers who operate machines and perform maintenance and inspection work within a factory.

[1234] An "engineer" is a technician who designs, develops, maintains, and inspects machinery and equipment.

[1235] This invention is an AI-enabled metaverse system to support maintenance and inspection work in factories. This system has the function of placing multiple virtual offices in the metaverse space, managing user account information, and granting appropriate access rights through authentication. The details are explained below.

[1236] Creating and managing metaverse spaces

[1237] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices correspond to areas where maintenance and inspection work is carried out in the factory, and users can access them to carry out their work efficiently.

[1238] User Account Settings and Login Authentication

[1239] When a user uses the system, they first launch the application on their device (smartphone, smart glasses, head-mounted display) and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1240] Log in to the virtual space and start using it

[1241] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The user controls this avatar to move to the area where maintenance and inspection work will be performed. The device renders and displays the virtual space in real time based on the data received from the server.

[1242] Supporting maintenance and inspection work with AI tools

[1243] Users can control an avatar to move near specific equipment or machinery and begin maintenance or inspection tasks. The server-based AI generator analyzes the user's questions and problem reports and presents appropriate solutions and procedures. The following are examples of specific prompts:

[1244] Engineer: "I'm hearing an abnormal noise coming from the machine. What measures should I take?"

[1245] AI: "Please first perform a frequency analysis of the machine to identify the cause of the abnormal noise. If that doesn't solve the problem, please try the steps below."

[1246] Consultation history storage and follow-up

[1247] The server stores all inquiries and answers in a database for future learning, and the generative AI method provides continuous follow-up questions and solutions based on past history, allowing users to solve their problems effectively.

[1248] Program processing explanation

[1249] The server is operated as a FastAPI-based API server on a cloud service (e.g., AWS, Azure). Internet-enabled devices (smartphones, smart glasses, head-mounted displays) communicate with the server and render data in the virtual space. The generative AI means on the server analyzes the user's input, generates appropriate answers, and sends the answers to the user's device in real time. Using a specific generative AI model (e.g., GPT-4) enables more accurate responses.

[1250] In this way, the system of the present invention effectively supports maintenance and inspection work within a factory and contributes to improving work efficiency.

[1251] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1252] Step 1:

[1253] The server generates a metaverse space and places multiple virtual offices within the virtual space. This sets up the business areas and consultation booths that users can access. The input is data on how to arrange the virtual offices and business areas as server configuration information, and the output is the metaverse space that users can access.

[1254] Step 2:

[1255] A user turns on a device (smartphone, smart glasses, or head-mounted display) and enters login information. The device sends this information to a server, which then authenticates the user's account information. The input is the user's authentication information (login ID and password), and the output is the success or failure of the authentication.

[1256] Step 3:

[1257] If authentication is successful, the server generates an avatar for the user in the virtual space. The device renders the virtual space based on the data received from the server and displays it to the user. The input is the user's account information and access privileges, and the output is the avatar display in the virtual space.

[1258] Step 4:

[1259] The user controls the avatar to move to a specific virtual office (maintenance and inspection area). The terminal reflects the user's operations in real time and updates the avatar's position. The input is the user's operation command (movement instruction), and the output is the avatar's new position information.

[1260] Step 5:

[1261] When a user moves an avatar near a specific machine or equipment to begin maintenance or inspection work, the device sends the user's question or problem details to the server. The input is the user's question or problem report, and the output is a receipt confirmation from the server.

[1262] Step 6:

[1263] The generative AI means on the server analyzes the received user question and generates an appropriate answer and response procedure. This uses a generative AI model such as GPT-4. The input is the user question, and the output is the generated answer and response procedure.

[1264] Step 7:

[1265] The server sends the generated answers and response procedures to the user's device. The device displays the received answers and response procedures to the user. The input is the answer data from the generation AI means, and the output is the content displayed on the user's device.

[1266] Step 8:

[1267] The user's consultation history and response procedures are stored in a database on the server. This allows them to be used for future consultations and problem solving. The input is the user's consultation content and the response results of the generated AI, and the output is the data stored in the database.

[1268] Step 9:

[1269] The server proposes continuous follow-up questions and solutions based on past history and provides them to the user. The input is past consultation history data, and the output is follow-up questions and solution suggestions.

[1270] The purpose of these processing steps is to ensure that maintenance and inspection work within the factory is carried out efficiently and to provide prompt support for users in resolving problems.

[1271] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1272] The present invention combines an AI consultation system that uses the metaverse space with an emotion engine, and the embodiments thereof will be described in detail below.

[1273] Creating and managing metaverse spaces

[1274] server:

[1275] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access each virtual office.

[1276] User Account Settings and Login Authentication

[1277] Device:

[1278] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1279] Log in to the virtual space and start using it

[1280] server:

[1281] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[1282] Device:

[1283] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1284] Consultation at AI help desk and use of emotion engine

[1285] User:

[1286] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation. The device is equipped with a video camera and microphone to recognize the user's facial expressions and voice.

[1287] Device:

[1288] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[1289] server:

[1290] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. It automatically generates follow-up questions and countermeasures according to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[1291] User:

[1292] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[1293] Emotion Log and Follow-up

[1294] server:

[1295] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1296] Specific examples

[1297] Scenario 1: Harassment consultation and sentiment analysis

[1298] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[1299] 2. The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[1300] 3. The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[1301] 4. The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional follow-up questions.

[1302] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and by using an emotion engine, more personalized responses are possible.

[1303] The processing flow will be explained below.

[1304] Step 1:

[1305] The server generates a metaverse space and places multiple virtual offices within it, each of which is set up as an area for accepting specific tasks or consultations.

[1306] Step 2:

[1307] The server registers user account information in a database and manages authentication information, including user ID, password, and access privileges.

[1308] Step 3:

[1309] The terminal displays a login screen for the user to use the system.

[1310] Step 4:

[1311] The user enters their ID and password on the login screen and clicks the login button.

[1312] Step 5:

[1313] The terminal transmits the entered login information to the server.

[1314] Step 6:

[1315] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[1316] Step 7:

[1317] The server generates an avatar for the authenticated user and places it in the virtual space, setting the avatar's initial position and viewpoint.

[1318] Step 8:

[1319] The device renders and displays the virtual space based on the data received from the server, and the user can control the avatar in real time and explore the virtual space.

[1320] Step 9:

[1321] The user controls the avatar to access the desired virtual office (e.g., the compliance department).

[1322] Step 10:

[1323] When the user arrives at the virtual office, the terminal activates the emotion engine, activates the video camera and microphone to transmit the user's facial expressions and voice to the emotion engine, and displays a form for inputting the details of the consultation.

[1324] Step 11:

[1325] The user enters the content of the consultation in text and clicks the send button.

[1326] Step 12:

[1327] The device sends the input consultation content and facial expression and voice data obtained from the emotion engine to the server.

[1328] Step 13:

[1329] The AI ​​on the server analyzes the received consultation content and emotional information, generates an appropriate response, and adjusts follow-up questions and countermeasures based on the emotional information according to the user's emotional state.

[1330] Step 14:

[1331] The server sends the generated answers and follow-up questions to the terminal.

[1332] Step 15:

[1333] The device displays the AI's answers and follow-up questions to the user.

[1334] Step 16:

[1335] The user reviews the AI's answers and enters additional questions or information as needed.

[1336] Step 17:

[1337] The server stores all consultation history and emotion logs in a database, which allows future learning and tracking of users' long-term emotional changes.

[1338] Through the above process, users can feel safe consulting within the metaverse space, and by utilizing the emotion engine, more personalized responses can be provided.

[1339] Example 2

[1340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1341] In modern companies and organizations, it is important to provide an environment where employees can seek advice anonymously and with peace of mind. However, current systems make it difficult to accurately grasp the user's emotions based on the text of the consultation alone, making it difficult to respond appropriately. In addition, there is a lack of follow-up and personalized responses in response to changes in employees' emotions, so improvements are needed to increase employee satisfaction.

[1342] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1343] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; AI means for receiving consultation content from the user, analyzing it, and generating an appropriate response; emotion engine means for recognizing and analyzing the user's facial expressions and voice; means for the AI ​​means to generate follow-up questions and countermeasures according to the emotion information obtained from the emotion engine; and means for storing consultation history and emotion logs in a database and using them for future learning. This allows for real-time analysis of user emotions and enables appropriate responses and follow-ups based on the analysis. This provides an environment where employees can consult with confidence and improves employee satisfaction.

[1344] The "metaverse space" is a digital virtual space generated using virtual reality technology, in which users can control avatars and carry out various activities.

[1345] A "virtual office" is a virtual office located within the metaverse space, where activities for specific purposes, such as the human resources department or compliance department, are carried out.

[1346] "User account information" refers to information for identifying a user, including a user name, password, email address, and the like.

[1347] The "means for granting access rights" is a mechanism that grants the right to access a specific virtual office or a specific area within the metaverse space when a user is authenticated.

[1348] An "avatar" is a virtual character that represents a user within the metaverse space, and moves and acts within the virtual space according to the user's instructions.

[1349] "AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of analyzing the content of inquiries from users and generating appropriate answers.

[1350] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to evaluate their current emotional state in real time.

[1351] The "means for generating follow-up questions and countermeasures" is a mechanism that continuously asks questions to the user and provides appropriate countermeasures based on information obtained from the emotion engine.

[1352] "Consultation history" is a record of the content of the consultation a user has had and the responses to those consultations, and is data that is saved for use in future learning and analysis.

[1353] An "emotion log" is data that records the user's emotional state during a consultation, and serves as the basis for providing a more personalized response the next time the user consults.

[1354] The present invention combines an AI consultation system that utilizes the metaverse space with an emotion engine, and the following describes its implementation in detail. The entire system is composed of the main elements of a server, a terminal, and a user.

[1355] Creating and managing metaverse spaces

[1356] server:

[1357] The server first generates a virtual metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access specific virtual offices.

[1358] User Account Settings and Login Authentication

[1359] Device:

[1360] When a user uses the system, they first launch a dedicated application on their device and enter their login information. The entered information is sent from the device to the server, and once authenticated by the server, the user is able to access the metaverse space.

[1361] Log in to the virtual space and start using it

[1362] server:

[1363] Once a user has been authenticated, an avatar is generated and displayed in the virtual space. The avatar is a character that represents the user and can move freely within the virtual space.

[1364] Device:

[1365] The device renders the virtual space based on the data received from the server and displays it to the user, allowing the user to explore the virtual space in real time.

[1366] Consultation at AI help desk and use of emotion engine

[1367] User:

[1368] The user controls an avatar to access a specific virtual office (e.g., the compliance department) and selects a consultation booth. A form for entering the details of the consultation is displayed. The terminal's video camera and microphone recognize the user's facial expressions and voice.

[1369] Device:

[1370] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[1371] server:

[1372] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. Follow-up questions and countermeasures are also automatically generated in response to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[1373] User:

[1374] Based on the answers provided by the AI, users can continue to ask questions or add new information, and can also move to another virtual office for another consultation if necessary.

[1375] Emotion Log and Follow-up

[1376] server:

[1377] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1378] Specific examples

[1379] Scenario 1: Harassment consultation and sentiment analysis

[1380] Users log in and access the "Compliance Department" within the virtual space.

[1381] The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[1382] The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[1383] The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional questions or follow-ups.

[1384] Prompt Sentence Examples

[1385] "Please tell me more about the harassment you recently experienced. Can you describe the specific situation and how it made you feel?"

[1386] In this way, according to the embodiment of the invention, it is possible to analyze the user's emotions in real time and provide appropriate responses and follow-ups based on the analysis. This system provides an environment where users can feel at ease when seeking advice, and realizes more personalized responses.

[1387] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1388] Step 1:

[1389] Device:

[1390] The user starts the application and enters account information (user name, password) on the login screen.

[1391] Input: Username, Password

[1392] Action: A user enters login information at a terminal.

[1393] Output: The entered login information

[1394] Step 2:

[1395] Device:

[1396] The entered login information is sent to the server.

[1397] Input: The login information entered

[1398] How it works: The device sends the login information to the server.

[1399] Output: Sending login information to the server

[1400] Step 3:

[1401] server:

[1402] The received account information is compared with information in the database to verify that the account exists.

[1403] Input: Login information sent from the device

[1404] Action: The server checks against the database.

[1405] Output: Authentication result (success / failure)

[1406] Step 4:

[1407] server:

[1408] If authentication is successful, the user's session information is generated and sent back to the terminal. If authentication fails, an error message is sent back to the terminal.

[1409] Input: Authentication result (success / failure), user account information

[1410] Action: The server generates session information or an error message.

[1411] Output: Session information (if successful), error message (if unsuccessful)

[1412] Step 5:

[1413] Device:

[1414] The terminal receives a message indicating successful authentication and provides the user with access to the metaverse space.

[1415] Input: Session information (if successful)

[1416] Action: The device displays the access interface to the metaverse space.

[1417] Output: Access rights to the metaverse space

[1418] Step 6:

[1419] server:

[1420] Once authenticated, an avatar is generated and displayed in the virtual space for the user. Multiple virtual offices are placed within the virtual space.

[1421] Input: User credentials, Virtual Office configuration data

[1422] How it works: The server generates an avatar and a virtual office.

[1423] Output: Avatar information, virtual office layout data

[1424] Step 7:

[1425] Device:

[1426] The virtual space is rendered based on the data received from the server and displayed to the user.

[1427] Input: Avatar information, virtual office layout data

[1428] How it works: The device renders the virtual space in real time.

[1429] Output: Visual information in virtual space

[1430] Step 8:

[1431] User:

[1432] The user operates an avatar, accesses a specific virtual office, selects a consultation booth, and inputs the details of the consultation.

[1433] Input: User operation (moving the avatar and selecting a consultation booth), consultation content

[1434] Operation: The user enters the details of the consultation at the consultation booth.

[1435] Output: Input consultation details

[1436] Step 9:

[1437] Device:

[1438] The system uses a video camera and microphone to capture the user's facial expressions and voice, and then sends the consultation details and the captured facial expression and voice data to the server.

[1439] Input: consultation details, facial expressions, and voice data

[1440] How it works: The device captures facial expressions and voice and sends them to the server.

[1441] Output: Send data to the server

[1442] Step 10:

[1443] server:

[1444] The received consultation content and user emotional information are sent to the emotion engine for analysis. The emotion engine analyzes the user's emotions in real time and obtains the results.

[1445] Input: consultation details, emotional information

[1446] Operation: The server sends data to the emotion engine and obtains the analysis results.

[1447] Output: Emotion analysis results

[1448] Step 11:

[1449] server:

[1450] Based on the emotional information and the content of the consultation, a generative AI model is used to generate an appropriate response. Follow-up questions and countermeasures are automatically generated according to changes in emotions. The generated response is then sent to the device.

[1451] Input: Consultation details, emotion analysis results

[1452] Operation: The server generates appropriate answers and follow-up questions and sends them to the device.

[1453] Output: Generated answers, follow-up questions

[1454] Step 12:

[1455] Device:

[1456] The received response is displayed to the user.

[1457] Input: Generated answers, follow-up questions

[1458] Action: The device displays the answer.

[1459] Output: Display the answer to the user

[1460] Step 13:

[1461] User:

[1462] Based on the answers provided by the AI, users can ask further questions or add new information, and if necessary, they can move to another virtual office for another consultation.

[1463] Input: AI response

[1464] Action: The user enters additional questions or information, or navigates to another virtual office.

[1465] Output: Additional questions, information, and next steps

[1466] Step 14:

[1467] server:

[1468] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of AI responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1469] Input: consultation history, emotion log

[1470] What happens: The server saves the data to the database.

[1471] Output: Saved data log

[1472] (Application example 2)

[1473] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1474] The purpose of this invention is to provide an environment where customers can easily and quickly ask questions or make complaints in a physical store. In particular, it aims to improve customer satisfaction by analyzing customer emotional information in real time and providing personalized and appropriate responses based on that information.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices in the virtual space, means for managing user account information and granting access rights to authenticated users, and means for generating a user avatar in the virtual space and operating the avatar to move to the virtual office. This allows the user to access different consultation booths in the store through the avatar, and when entering the details of their consultation, they can receive personalized responses accompanied by facial expression and voice analysis using an emotion analysis engine.

[1476] The "metaverse space" is a three-dimensional digital space created using virtual reality technology, in which users can interact through avatars.

[1477] A "virtual office" is a virtual area with a specific purpose or function that is located within the metaverse space, and is a virtual space where users can carry out activities and consultations within their respective areas.

[1478] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and extracts emotional information in real time, and includes an emotion recognition and analysis system.

[1479] An "avatar" is a digital alter ego that represents and controls a user in a virtual space, and is an agent that represents the user's actions and intentions.

[1480] "AI means" refers to a means that uses artificial intelligence technology to analyze the content of inquiries received from users and generate appropriate answers and countermeasures.

[1481] "Emotion information" is data relating to the user's mental and emotional state obtained as a result of analysis by the emotion engine.

[1482] A "personalized response" is an appropriate answer or response tailored to an individual user, generated based on the user's specific inquiry content and emotional information.

[1483] "Consultation history" is a record of the consultation content and responses made by the user, and is data stored in a database for future learning and improvement.

[1484] MODE FOR CARRYING OUT THE INVENTION

[1485] The present invention aims to provide an AI system that uses the metaverse space and includes emotion analysis for the purpose of customer support in brick-and-mortar stores. Specific embodiments of the system are described below.

[1486] (server)

[1487] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as customer support booths and information booths. The server also manages user account information and grants appropriate access privileges to authenticated users.

[1488] (Terminal)

[1489] When a customer uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1490] (Log in to the virtual space and start using it)

[1491] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[1492] (AI consultation service and use of emotion engine)

[1493] The user controls an avatar to access a specific virtual office (e.g., a customer support booth). After selecting a consultation booth, a form for entering the details of the consultation is displayed. A video camera and microphone are installed on the terminal to recognize the user's facial expressions and voice. The terminal simultaneously transmits the user's facial expressions and voice to an emotion analysis engine along with the consultation details entered by the user. The emotion engine analyzes the user's emotions in real time. The AI ​​system on the server generates an appropriate answer based on the consultation details and emotional information received. This generated answer is sent from the server to the terminal and displayed to the user. This allows the user to receive emotionally appropriate support.

[1494] (Emotion log and follow-up)

[1495] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1496] Specific examples

[1497] For example, if a user asks, "How do I get a refund for this product?" and their face recorded by the camera shows anxiety, the emotion engine will analyze the anxiety and the AI ​​will provide polite instructions such as, "Click here for the refund procedure. If you have any questions, our staff will be happy to assist you."

[1498] Prompt Sentence Examples

[1499] "We will develop an AI consultation assistant that customers can use in physical stores via their smartphones. We will use an emotion engine to analyze customer emotions and enable the AI ​​to provide appropriate responses."

[1500] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1501] Step 1:

[1502] The application is launched on the device and the user enters their login information.

[1503] Input: Username and Password.

[1504] Process: Sends login information to the server for authentication.

[1505] Output: The authentication result (success or failure).

[1506] The server checks the received login information against its database and, if successful, grants the user access.

[1507] Step 2:

[1508] For users who are successfully authenticated, the server generates an avatar in the metaverse space.

[1509] Input: User credentials.

[1510] Processing: Avatar generation processing.

[1511] Output: Avatar information in virtual space.

[1512] The server generates an avatar in the metaverse space for the authenticated user and sends that information to the terminal, making it operable.

[1513] Step 3:

[1514] The user operates an avatar to go to the virtual office and select a specific consultation booth.

[1515] Input: User operation information.

[1516] Processing: Movement processing within virtual space.

[1517] Output: Booth selection information.

[1518] Based on the user's operation, the terminal moves the avatar within the virtual space and transmits information about the selected booth to the server.

[1519] Step 4:

[1520] The user inputs the details of the consultation and transmits facial expression and voice data from the terminal.

[1521] Input: Consultation content, facial expression data, and voice data.

[1522] Processing: Sending input data.

[1523] Output: Sending data to the server.

[1524] The device sends the user's consultation details, facial expression data captured by a video camera, and audio data captured by a microphone to an emotion analysis engine.

[1525] Step 5:

[1526] The emotion engine analyzes facial expressions and voice to generate emotional information.

[1527] Input: Facial expression data, audio data.

[1528] Processing: Sentiment analysis processing.

[1529] Output: Emotional information.

[1530] The emotion engine analyzes the received data, extracts the user's emotional state in real time, and sends that information to the AI ​​system.

[1531] Step 6:

[1532] The AI ​​system generates an appropriate answer based on the content of the consultation and emotional information.

[1533] Input: Consultation content, emotional information.

[1534] Processing: Answer generation processing.

[1535] Output: Answer content.

[1536] The AI ​​means on the server analyzes the consultation content and emotional information, generates answers and follow-up questions, and sends the information to the terminal.

[1537] Step 7:

[1538] The device displays the generated answers to the user and prompts for additional questions or information as needed.

[1539] Input: The response from the AI ​​method.

[1540] Processing: Generate an interface for displaying and inputting answers.

[1541] Output: Display the answer to the user.

[1542] The terminal displays the received answers to the user and generates an interface to await further input from the user.

[1543] Step 8:

[1544] The server stores the consultation details and emotion logs in a database.

[1545] Input: Consultation content, emotion log.

[1546] Processing: Data storage processing.

[1547] Output: Information saved to the database.

[1548] The server stores the consultation details and emotional information in a database and uses it for future learning.

[1549] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1550] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1551] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1552] [Fourth embodiment]

[1553] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1554] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1555] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1556] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1557] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1558] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1559] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1560] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1561] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1562] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1563] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1564] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1565] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1566] The present invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail below.

[1567] Creating and managing metaverse spaces

[1568] server:

[1569] The server first generates a metaverse space and places multiple virtual offices within the virtual space. This allows users to access locations for various tasks and consultations. Each virtual office corresponds to a department (e.g., human resources, compliance), and each has its own unique interactive functions. The server also manages user account information and controls access privileges by authenticating users. This ensures that only appropriate users can access each virtual office.

[1570] User Account Settings and Login Authentication

[1571] Device:

[1572] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1573] Log in to the virtual space and start using it

[1574] server:

[1575] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[1576] Device:

[1577] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1578] Consultation at AI help desk

[1579] User:

[1580] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation.

[1581] Device:

[1582] The device sends the consultation details entered by the user to the AI. The input method is primarily text input, but other methods such as voice input are also supported.

[1583] server:

[1584] The AI ​​on the server analyzes the received inquiry and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1585] User:

[1586] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[1587] Follow-up on consultation results and history management

[1588] server:

[1589] The server stores all consultation history in a database and uses it for future learning, which is an important step to improve the AI's response accuracy and customer satisfaction.

[1590] Specific examples

[1591] Scenario 1: Harassment consultation

[1592] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[1593] 2. The device displays a consultation form, and the user enters the specific details of the harassment they wish to discuss.

[1594] 3. The server receives the consultation content, and the AI ​​analyzes it and provides solutions.

[1595] 4. The device displays the AI's answer, and the user decides on the next action.

[1596] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, enabling early detection and response to problems.

[1597] The processing flow will be explained below.

[1598] Step 1:

[1599] The server creates a metaverse space and places multiple virtual offices within it, each containing multiple areas used for different purposes, such as the human resources department and the compliance department.

[1600] Step 2:

[1601] The server manages user account information and stores it in a database, including user IDs, passwords, and access privileges.

[1602] Step 3:

[1603] The terminal displays a screen for the user to log in and prompts the user to enter their ID and password.

[1604] Step 4:

[1605] The user enters login information (ID and password) and clicks the login button.

[1606] Step 5:

[1607] The terminal transmits the entered login information to the server.

[1608] Step 6:

[1609] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[1610] Step 7:

[1611] The server generates an avatar for the authenticated user and places it in the virtual space, as well as setting the user's initial position and viewpoint.

[1612] Step 8:

[1613] The device renders and displays the user's avatar and the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1614] Step 9:

[1615] The user controls the avatar and heads to the desired virtual office (e.g., the compliance department).

[1616] Step 10:

[1617] When the user arrives at the virtual office, the terminal displays a form for inputting the details of the consultation.

[1618] Step 11:

[1619] Users enter their complaint in text and click the send button, which includes details of the specific harassment and other compliance issues.

[1620] Step 12:

[1621] The terminal transmits the input consultation content to the server.

[1622] Step 13:

[1623] The AI ​​on the server analyzes the received consultation content and generates appropriate answers and measures, using natural language processing technology.

[1624] Step 14:

[1625] The server sends the generated response to the terminal.

[1626] Step 15:

[1627] The device displays the AI's answer to the user, who can then ask further questions or add new information based on the answer.

[1628] Step 16:

[1629] The server stores all consultation history in a database, which is used for future learning and analysis.

[1630] Example 1

[1631] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] Conventional consultation systems do not adequately provide an environment for quickly and accurately resolving users' problems. In particular, they lack a flexible system that can handle a wide range of consultation content while maintaining anonymity. Furthermore, because consultation history is not accumulated, it is difficult to make continuous improvements based on past consultation content. This makes it difficult to improve user satisfaction, and efficient problem-solving is required.

[1633] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1634] In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices within the virtual space, means for managing user account information and granting access rights to authenticated users, means for generating a user avatar within the virtual space and manipulating the avatar to move to the virtual office, means for receiving consultation content from the user's input, analyzing it using a generative AI model, and generating an appropriate answer, and means for saving consultation history in a database and using it for future learning. This allows users to freely consult while maintaining anonymity, and makes it possible to accumulate history and improve the accuracy of the AI.

[1635] A "metaverse space" is a digital space created using virtual reality technology, where users can move freely beyond physical constraints.

[1636] A "virtual office" is a virtual work area located within the metaverse space that corresponds to a specific business department. Users can consult and process business matters within this area.

[1637] "User account information" is a series of data (e.g., username, password, job title, etc.) used for personal authentication of users and access control.

[1638] "Authentication" is the process of verifying that a user has valid authority based on the entered user information.

[1639] "Access Privileges" means the authorization for a User to access certain resources or functions within the System.

[1640] An "avatar" is a character that represents a user in a virtual space, and is a digital entity that reflects the user's actions and behavior.

[1641] A "generative AI model" is an artificial intelligence system that uses natural language processing and machine learning techniques to analyze input data from users and generate appropriate responses.

[1642] "Consultation content" is detailed information about a question or problem that a user inputs to a help desk in the virtual space.

[1643] The "consultation history" is a record of all consultations that a user has made with the system, including the contents of those consultations and the responses to those consultations, and is saved for future reference and analysis.

[1644] A "database" is a system for storing data in a structured digital format, allowing for efficient retrieval and management.

[1645] This invention is an AI consultation desk system that uses the metaverse space, and an embodiment thereof will be described in detail.

[1646] Creating and managing metaverse spaces

[1647] server:

[1648] The server generates a metaverse space using game engines such as Unity or Unreal Engine. Multiple virtual offices are placed within the metaverse space. Each virtual office corresponds to a specific business department, such as the human resources department or the compliance department, and each has its own interactive functions. The server also manages user account information and performs authentication to enforce access control.

[1649] User Account Settings and Login Authentication

[1650] Device:

[1651] When a user uses the system, they first launch a dedicated application on their device. The user enters their ID and password on the login screen, and the authentication information is sent to the server. The server verifies the authentication information using a database, and if authentication is successful, the device is given access to the metaverse space.

[1652] Log in to the virtual space and start using it

[1653] server:

[1654] Once the user has been authenticated, the server generates an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar and head to the desired virtual office.

[1655] Device:

[1656] The device renders and displays the virtual space in real time based on the data received from the server. The virtual space is created using game engines such as Unity and Unreal Engine, allowing users to explore seamlessly.

[1657] Consultation at AI help desk

[1658] User:

[1659] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and enters the details of the consultation.

[1660] Device:

[1661] The device sends the consultation details entered by the user to the generation AI model. Input methods supported include text and voice input.

[1662] server:

[1663] A generative AI model (e.g., OpenAI's GPT-3) on the server analyzes the prompt entered by the user and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1664] Follow-up on consultation results and history management

[1665] server:

[1666] The server stores all consultation history in a database, which is used for future training of the AI ​​model to improve response accuracy.

[1667] Specific examples

[1668] Scenario 1: Harassment consultation

[1669] 1. The user launches the dedicated app and enters their login information into the device.

[1670] 2. The server authenticates the login information and provides the user with access to the metaverse space.

[1671] 3. After logging in, the terminal displays the virtual space, and the user operates the avatar to move to the "Compliance Department."

[1672] 4. The user enters the details of their consultation at the consultation booth, and the device sends the data to the generative AI model.

[1673] 5. The server's generated AI model analyzes the consultation content and generates appropriate countermeasures.

[1674] 6. The device displays the generated answer, and the user decides on the next action based on it.

[1675] Example prompt sentence:

[1676] "I would like to consult about measures to deal with harassment. To be more specific, my boss has been verbally abusing me. Could you please give me some advice on how I should respond?"

[1677] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and supports early detection and response to problems.

[1678] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1679] Step 1:

[1680] Metaverse space generation and virtual office layout

[1681] server:

[1682] The server generates the metaverse space using Unity or Unreal Engine. First, a base environment for the virtual space is set up, and then multiple virtual offices are placed within it. The virtual offices are placed based on predefined coordinate data. Different interactive elements are also placed in each virtual office. For example, the HR department can provide an employee consultation service, and the compliance department can provide legal advice.

[1683] Input: Virtual space base setting data, virtual office metadata

[1684] Output: Location information within the metaverse space

[1685] Specific behavior:

[1686] The server runs the virtual space generation script and loads the layout information for each virtual office, completing the basic structure of the virtual space.

[1687] Step 2:

[1688] User Account Settings and Login Authentication

[1689] Device:

[1690] The user starts the dedicated application and enters their ID and password on the login screen. This authentication information is sent to the server as an HTTP request.

[1691] server:

[1692] The server uses a database to verify the submitted credentials, and if successful, stores the user's account information in the session and grants them access to the metaverse space.

[1693] Input: User ID and password

[1694] Output: Authentication result, session ID

[1695] Specific behavior:

[1696] The server checks the authentication information against the database, and if authentication is successful, generates a session ID and sends it to the terminal.

[1697] Step 3:

[1698] Log in to the virtual space and generate an avatar

[1699] server:

[1700] Once a user has been authenticated, the server generates an avatar in the virtual space. This avatar is a character that reflects the user's actions, and its appearance and functions are set based on the user's profile data.

[1701] Device:

[1702] The device renders and displays the virtual space in real time based on the avatar and virtual space information received from the server, allowing the user to begin activities within the virtual space.

[1703] Input: User credentials, profile data

[1704] Output: Avatar information in virtual space, virtual space rendering information

[1705] Specific behavior:

[1706] The server runs an avatar generation script based on the user's profile information and sends rendering data to the device, which then receives the data and renders the virtual space.

[1707] Step 4:

[1708] Consultation at AI help desk

[1709] User:

[1710] The user controls an avatar to move to a specific virtual office, select a consultation booth, and enter the details of their consultation. The input method is primarily text, but voice input is also supported.

[1711] Device:

[1712] The device converts the consultation content entered by the user into a text prompt and sends it to the generative AI model.

[1713] server:

[1714] A generative AI model (e.g., GPT-3) on the server analyzes the received prompt and generates an appropriate answer, which is then sent from the server to the device and displayed to the user.

[1715] Input: User's inquiry (prompt text)

[1716] Output: Answer from the generative AI model

[1717] Specific behavior:

[1718] The user inputs the content of their inquiry using a keyboard or voice input, and the device sends this as an HTTP request to the server. The server then analyzes the content of the inquiry using a generative AI model, generates a response, and sends it to the device.

[1719] Step 5:

[1720] Follow-up on consultation results and history management

[1721] server:

[1722] The server stores all consultation history in a database, including the user's consultation content and the answers provided by the generative AI model. This data will be used as training data for future projects.

[1723] Input: User's inquiry, response from the generated AI model

[1724] Output: Consultation history data

[1725] Specific behavior:

[1726] The server stores the received inquiries and generated responses in a log file or database for future reference and analysis.

[1727] This system provides an environment where users can easily consult with the system, and also allows historical data to be used to improve the performance of the AI.

[1728] (Application example 1)

[1729] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] Maintenance and inspection work in factories takes a lot of time and effort, and responding to machine abnormalities and breakdowns in particular requires advanced expertise. Conventional systems make it difficult to provide immediate on-site responses or appropriate solutions. As a result, efficient production is often hindered. Furthermore, differences in the knowledge and experience of technicians can lead to variations in response quality, reducing overall operational efficiency. To solve these issues, a system that supports on-site maintenance and inspection work is needed.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1732] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; generation AI means for receiving consultation content from the user, analyzing it, and generating an appropriate answer; means for storing the consultation history in a database and using it for future learning; AI means for guiding procedures to support maintenance and inspection work in factories and providing solutions when problems occur; and means for factory employees and engineers to perform maintenance and inspection work in the virtual space using smartphones, smart glasses, or head-mounted displays. This enables immediate response and provision of appropriate solutions on-site, improving work efficiency throughout the factory.

[1733] The "metaverse space" is a virtual space that exists on the Internet, where users can interact with each other through avatars.

[1734] A "virtual office" is a virtual work location located within the metaverse space that provides the same functions as a real office.

[1735] "User account information" is information for uniquely identifying a user, and includes authentication information such as a login ID and password.

[1736] "Access privileges" refer to the rights of an authenticated user to access specific resources or functions within a system.

[1737] An "avatar" is a character that represents a user, moves within a virtual space, and interacts with other users and the system.

[1738] "Generative AI means" refers to artificial intelligence technology that analyzes user input and automatically generates appropriate answers and countermeasures.

[1739] "Consultation history" refers to a record of the consultations users make through the system, as well as the responses and countermeasures taken.

[1740] "Procedures to support maintenance and inspection work" are guidelines to instruct and support procedures for efficiently carrying out maintenance and inspection of machinery and equipment within a factory.

[1741] A "smartphone" is a portable information terminal that can connect to the Internet and use a variety of applications.

[1742] "Smart glasses" are a wearable eyeglass-type device equipped with information display and interactive functions.

[1743] A "head-mounted display" is a display device worn on the head, used to display a virtual space to the user.

[1744] "Factory employees" are workers who operate machines and perform maintenance and inspection work within a factory.

[1745] An "engineer" is a technician who designs, develops, maintains, and inspects machinery and equipment.

[1746] This invention is an AI-enabled metaverse system to support maintenance and inspection work in factories. This system has the function of placing multiple virtual offices in the metaverse space, managing user account information, and granting appropriate access rights through authentication. The details are explained below.

[1747] Creating and managing metaverse spaces

[1748] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices correspond to areas where maintenance and inspection work is carried out in the factory, and users can access them to carry out their work efficiently.

[1749] User Account Settings and Login Authentication

[1750] When a user uses the system, they first launch the application on their device (smartphone, smart glasses, head-mounted display) and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1751] Log in to the virtual space and start using it

[1752] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The user controls this avatar to move to the area where maintenance and inspection work will be performed. The device renders and displays the virtual space in real time based on the data received from the server.

[1753] Supporting maintenance and inspection work with AI tools

[1754] Users can control an avatar to move near specific equipment or machinery and begin maintenance or inspection tasks. The server-based AI generator analyzes the user's questions and problem reports and presents appropriate solutions and procedures. The following are examples of specific prompts:

[1755] Engineer: "I'm hearing an abnormal noise coming from the machine. What measures should I take?"

[1756] AI: "Please first perform a frequency analysis of the machine to identify the cause of the abnormal noise. If that doesn't solve the problem, please try the steps below."

[1757] Consultation history storage and follow-up

[1758] The server stores all inquiries and answers in a database for future learning, and the generative AI method provides continuous follow-up questions and solutions based on past history, allowing users to solve their problems effectively.

[1759] Program processing explanation

[1760] The server is operated as a FastAPI-based API server on a cloud service (e.g., AWS, Azure). Internet-enabled devices (smartphones, smart glasses, head-mounted displays) communicate with the server and render data in the virtual space. The generative AI means on the server analyzes the user's input, generates appropriate answers, and sends the answers to the user's device in real time. Using a specific generative AI model (e.g., GPT-4) enables more accurate responses.

[1761] In this way, the system of the present invention effectively supports maintenance and inspection work within a factory and contributes to improving work efficiency.

[1762] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1763] Step 1:

[1764] The server generates a metaverse space and places multiple virtual offices within the virtual space. This sets up the business areas and consultation booths that users can access. The input is data on how to arrange the virtual offices and business areas as server configuration information, and the output is the metaverse space that users can access.

[1765] Step 2:

[1766] A user turns on a device (smartphone, smart glasses, or head-mounted display) and enters login information. The device sends this information to a server, which then authenticates the user's account information. The input is the user's authentication information (login ID and password), and the output is the success or failure of the authentication.

[1767] Step 3:

[1768] If authentication is successful, the server generates an avatar for the user in the virtual space. The device renders the virtual space based on the data received from the server and displays it to the user. The input is the user's account information and access privileges, and the output is the avatar display in the virtual space.

[1769] Step 4:

[1770] The user controls the avatar to move to a specific virtual office (maintenance and inspection area). The terminal reflects the user's operations in real time and updates the avatar's position. The input is the user's operation command (movement instruction), and the output is the avatar's new position information.

[1771] Step 5:

[1772] When a user moves an avatar near a specific machine or equipment to begin maintenance or inspection work, the device sends the user's question or problem details to the server. The input is the user's question or problem report, and the output is a receipt confirmation from the server.

[1773] Step 6:

[1774] The generative AI means on the server analyzes the received user question and generates an appropriate answer and response procedure. This uses a generative AI model such as GPT-4. The input is the user question, and the output is the generated answer and response procedure.

[1775] Step 7:

[1776] The server sends the generated answers and response procedures to the user's device. The device displays the received answers and response procedures to the user. The input is the answer data from the generation AI means, and the output is the content displayed on the user's device.

[1777] Step 8:

[1778] The user's consultation history and response procedures are stored in a database on the server. This allows them to be used for future consultations and problem solving. The input is the user's consultation content and the response results of the generated AI, and the output is the data stored in the database.

[1779] Step 9:

[1780] The server proposes continuous follow-up questions and solutions based on past history and provides them to the user. The input is past consultation history data, and the output is follow-up questions and solution suggestions.

[1781] The purpose of these processing steps is to ensure that maintenance and inspection work within the factory is carried out efficiently and to provide prompt support for users in resolving problems.

[1782] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1783] The present invention combines an AI consultation system that uses the metaverse space with an emotion engine, and the embodiments thereof will be described in detail below.

[1784] Creating and managing metaverse spaces

[1785] server:

[1786] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access each virtual office.

[1787] User Account Settings and Login Authentication

[1788] Device:

[1789] When a user uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[1790] Log in to the virtual space and start using it

[1791] server:

[1792] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[1793] Device:

[1794] The device renders and displays the virtual space based on the data received from the server, allowing the user to explore the virtual space in real time.

[1795] Consultation at AI help desk and use of emotion engine

[1796] User:

[1797] The user controls an avatar to access a specific virtual office (e.g., the compliance department), selects a consultation booth, and is presented with a form to enter the details of the consultation. The device is equipped with a video camera and microphone to recognize the user's facial expressions and voice.

[1798] Device:

[1799] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[1800] server:

[1801] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. It automatically generates follow-up questions and countermeasures according to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[1802] User:

[1803] Based on the answers provided by the AI, users can continue to ask questions or add new information, and if necessary, they can move to another virtual office and conduct another consultation.

[1804] Emotion Log and Follow-up

[1805] server:

[1806] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1807] Specific examples

[1808] Scenario 1: Harassment consultation and sentiment analysis

[1809] 1. The user logs in and accesses the "Compliance Department" within the virtual space.

[1810] 2. The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[1811] 3. The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[1812] 4. The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional follow-up questions.

[1813] In this way, the present invention provides an environment in which users can seek advice easily and while maintaining anonymity, and by using an emotion engine, more personalized responses are possible.

[1814] The processing flow will be explained below.

[1815] Step 1:

[1816] The server generates a metaverse space and places multiple virtual offices within it, each of which is set up as an area for accepting specific tasks or consultations.

[1817] Step 2:

[1818] The server registers user account information in a database and manages authentication information, including user ID, password, and access privileges.

[1819] Step 3:

[1820] The terminal displays a login screen for the user to use the system.

[1821] Step 4:

[1822] The user enters their ID and password on the login screen and clicks the login button.

[1823] Step 5:

[1824] The terminal transmits the entered login information to the server.

[1825] Step 6:

[1826] The server checks the login information against a database and performs authentication. If authentication is successful, the user is granted access to the metaverse space.

[1827] Step 7:

[1828] The server generates an avatar for the authenticated user and places it in the virtual space, setting the avatar's initial position and viewpoint.

[1829] Step 8:

[1830] The device renders and displays the virtual space based on the data received from the server, and the user can control the avatar in real time and explore the virtual space.

[1831] Step 9:

[1832] The user controls the avatar to access the desired virtual office (e.g., the compliance department).

[1833] Step 10:

[1834] When the user arrives at the virtual office, the terminal activates the emotion engine, activates the video camera and microphone to transmit the user's facial expressions and voice to the emotion engine, and displays a form for inputting the details of the consultation.

[1835] Step 11:

[1836] The user enters the content of the consultation in text and clicks the send button.

[1837] Step 12:

[1838] The device sends the input consultation content and facial expression and voice data obtained from the emotion engine to the server.

[1839] Step 13:

[1840] The AI ​​on the server analyzes the received consultation content and emotional information, generates an appropriate response, and adjusts follow-up questions and countermeasures based on the emotional information according to the user's emotional state.

[1841] Step 14:

[1842] The server sends the generated answers and follow-up questions to the terminal.

[1843] Step 15:

[1844] The device displays the AI's answers and follow-up questions to the user.

[1845] Step 16:

[1846] The user reviews the AI's answers and enters additional questions or information as needed.

[1847] Step 17:

[1848] The server stores all consultation history and emotion logs in a database, which allows future learning and tracking of users' long-term emotional changes.

[1849] Through the above process, users can feel safe consulting within the metaverse space, and by utilizing the emotion engine, more personalized responses can be provided.

[1850] Example 2

[1851] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1852] In modern companies and organizations, it is important to provide an environment where employees can seek advice anonymously and with peace of mind. However, current systems make it difficult to accurately grasp the user's emotions based on the text of the consultation alone, making it difficult to respond appropriately. In addition, there is a lack of follow-up and personalized responses in response to changes in employees' emotions, so improvements are needed to increase employee satisfaction.

[1853] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1854] In this invention, the server includes: means for generating a metaverse space and arranging multiple virtual offices within the virtual space; means for managing user account information and granting access privileges to authenticated users; means for generating a user avatar within the virtual space and operating the avatar to move to the virtual office; AI means for receiving consultation content from the user, analyzing it, and generating an appropriate response; emotion engine means for recognizing and analyzing the user's facial expressions and voice; means for the AI ​​means to generate follow-up questions and countermeasures according to the emotion information obtained from the emotion engine; and means for storing consultation history and emotion logs in a database and using them for future learning. This allows for real-time analysis of user emotions and enables appropriate responses and follow-ups based on the analysis. This provides an environment where employees can consult with confidence and improves employee satisfaction.

[1855] The "metaverse space" is a digital virtual space generated using virtual reality technology, in which users can control avatars and carry out various activities.

[1856] A "virtual office" is a virtual office located within the metaverse space, where activities for specific purposes, such as the human resources department or compliance department, are carried out.

[1857] "User account information" refers to information for identifying a user, including a user name, password, email address, and the like.

[1858] The "means for granting access rights" is a mechanism that grants the right to access a specific virtual office or a specific area within the metaverse space when a user is authenticated.

[1859] An "avatar" is a virtual character that represents a user within the metaverse space, and moves and acts within the virtual space according to the user's instructions.

[1860] "AI means" refers to technology that uses artificial intelligence to perform specific tasks, and in this case refers to the function of analyzing the content of inquiries from users and generating appropriate answers.

[1861] The "emotion engine" is a technology that analyzes the user's facial expressions and voice to evaluate their current emotional state in real time.

[1862] The "means for generating follow-up questions and countermeasures" is a mechanism that continuously asks questions to the user and provides appropriate countermeasures based on information obtained from the emotion engine.

[1863] "Consultation history" is a record of the content of the consultation a user has had and the responses to those consultations, and is data that is saved for use in future learning and analysis.

[1864] An "emotion log" is data that records the user's emotional state during a consultation, and serves as the basis for providing a more personalized response the next time the user consults.

[1865] The present invention combines an AI consultation system that utilizes the metaverse space with an emotion engine, and the following describes its implementation in detail. The entire system is composed of the main elements of a server, a terminal, and a user.

[1866] Creating and managing metaverse spaces

[1867] server:

[1868] The server first generates a virtual metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as the human resources department and compliance department. The server manages user account information and controls access privileges through authentication, ensuring that only appropriate users can access specific virtual offices.

[1869] User Account Settings and Login Authentication

[1870] Device:

[1871] When a user uses the system, they first launch a dedicated application on their device and enter their login information. The entered information is sent from the device to the server, and once authenticated by the server, the user is able to access the metaverse space.

[1872] Log in to the virtual space and start using it

[1873] server:

[1874] Once a user has been authenticated, an avatar is generated and displayed in the virtual space. The avatar is a character that represents the user and can move freely within the virtual space.

[1875] Device:

[1876] The device renders the virtual space based on the data received from the server and displays it to the user, allowing the user to explore the virtual space in real time.

[1877] Consultation at AI help desk and use of emotion engine

[1878] User:

[1879] The user controls an avatar to access a specific virtual office (e.g., the compliance department) and selects a consultation booth. A form for entering the details of the consultation is displayed. The terminal's video camera and microphone recognize the user's facial expressions and voice.

[1880] Device:

[1881] The device sends the user's facial expressions and voice to the emotion engine, which then analyzes the user's emotions in real time.

[1882] server:

[1883] The AI ​​on the server analyzes the received consultation content and generates an appropriate answer based on the emotional information obtained from the emotion engine. Follow-up questions and countermeasures are also automatically generated in response to changes in emotions. The generated answer is sent from the server to the device and displayed to the user.

[1884] User:

[1885] Based on the answers provided by the AI, users can continue to ask questions or add new information, and can also move to another virtual office for another consultation if necessary.

[1886] Emotion Log and Follow-up

[1887] server:

[1888] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1889] Specific examples

[1890] Scenario 1: Harassment consultation and sentiment analysis

[1891] Users log in and access the "Compliance Department" within the virtual space.

[1892] The device displays a consultation form, and the user inputs the specific details of the harassment they wish to discuss. At the same time, the video camera and microphone transmit the user's facial expressions and voice to the emotion engine.

[1893] The server receives the consultation content and emotional information, and the AI ​​analyzes it and provides solutions.

[1894] The device displays the AI's answer, and the user decides on the next action. If the user's emotions are unstable, the server generates additional questions or follow-ups.

[1895] Prompt Sentence Examples

[1896] "Please tell me more about the harassment you recently experienced. Can you describe the specific situation and how it made you feel?"

[1897] In this way, according to the embodiment of the invention, it is possible to analyze the user's emotions in real time and provide appropriate responses and follow-ups based on the analysis. This system provides an environment where users can feel at ease when seeking advice, and realizes more personalized responses.

[1898] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1899] Step 1:

[1900] Device:

[1901] The user starts the application and enters account information (user name, password) on the login screen.

[1902] Input: Username, Password

[1903] Action: A user enters login information at a terminal.

[1904] Output: The entered login information

[1905] Step 2:

[1906] Device:

[1907] The entered login information is sent to the server.

[1908] Input: The login information entered

[1909] How it works: The device sends the login information to the server.

[1910] Output: Sending login information to the server

[1911] Step 3:

[1912] server:

[1913] The received account information is compared with information in the database to verify that the account exists.

[1914] Input: Login information sent from the device

[1915] Action: The server checks against the database.

[1916] Output: Authentication result (success / failure)

[1917] Step 4:

[1918] server:

[1919] If authentication is successful, the user's session information is generated and sent back to the terminal. If authentication fails, an error message is sent back to the terminal.

[1920] Input: Authentication result (success / failure), user account information

[1921] Action: The server generates session information or an error message.

[1922] Output: Session information (if successful), error message (if unsuccessful)

[1923] Step 5:

[1924] Device:

[1925] The terminal receives a message indicating successful authentication and provides the user with access to the metaverse space.

[1926] Input: Session information (if successful)

[1927] Action: The device displays the access interface to the metaverse space.

[1928] Output: Access rights to the metaverse space

[1929] Step 6:

[1930] server:

[1931] Once authenticated, an avatar is generated and displayed in the virtual space for the user. Multiple virtual offices are placed within the virtual space.

[1932] Input: User credentials, Virtual Office configuration data

[1933] How it works: The server generates an avatar and a virtual office.

[1934] Output: Avatar information, virtual office layout data

[1935] Step 7:

[1936] Device:

[1937] The virtual space is rendered based on the data received from the server and displayed to the user.

[1938] Input: Avatar information, virtual office layout data

[1939] How it works: The device renders the virtual space in real time.

[1940] Output: Visual information in virtual space

[1941] Step 8:

[1942] User:

[1943] The user operates an avatar, accesses a specific virtual office, selects a consultation booth, and inputs the details of the consultation.

[1944] Input: User operation (moving the avatar and selecting a consultation booth), consultation content

[1945] Operation: The user enters the details of the consultation at the consultation booth.

[1946] Output: Input consultation details

[1947] Step 9:

[1948] Device:

[1949] The system uses a video camera and microphone to capture the user's facial expressions and voice, and then sends the consultation details and the captured facial expression and voice data to the server.

[1950] Input: consultation details, facial expressions, and voice data

[1951] How it works: The device captures facial expressions and voice and sends them to the server.

[1952] Output: Send data to the server

[1953] Step 10:

[1954] server:

[1955] The received consultation content and user emotional information are sent to the emotion engine for analysis. The emotion engine analyzes the user's emotions in real time and obtains the results.

[1956] Input: consultation details, emotional information

[1957] Operation: The server sends data to the emotion engine and obtains the analysis results.

[1958] Output: Emotion analysis results

[1959] Step 11:

[1960] server:

[1961] Based on the emotional information and the content of the consultation, a generative AI model is used to generate an appropriate response. Follow-up questions and countermeasures are automatically generated according to changes in emotions. The generated response is then sent to the device.

[1962] Input: Consultation details, emotion analysis results

[1963] Operation: The server generates appropriate answers and follow-up questions and sends them to the device.

[1964] Output: Generated answers, follow-up questions

[1965] Step 12:

[1966] Device:

[1967] The received response is displayed to the user.

[1968] Input: Generated answers, follow-up questions

[1969] Action: The device displays the answer.

[1970] Output: Display the answer to the user

[1971] Step 13:

[1972] User:

[1973] Based on the answers provided by the AI, users can ask further questions or add new information, and if necessary, they can move to another virtual office for another consultation.

[1974] Input: AI response

[1975] Action: The user enters additional questions or information, or navigates to another virtual office.

[1976] Output: Additional questions, information, and next steps

[1977] Step 14:

[1978] server:

[1979] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of AI responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[1980] Input: consultation history, emotion log

[1981] What happens: The server saves the data to the database.

[1982] Output: Saved data log

[1983] (Application example 2)

[1984] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1985] The purpose of this invention is to provide an environment where customers can easily and quickly ask questions or make complaints in a physical store. In particular, it aims to improve customer satisfaction by analyzing customer emotional information in real time and providing personalized and appropriate responses based on that information.

[1986] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating a metaverse space and arranging multiple virtual offices in the virtual space, means for managing user account information and granting access rights to authenticated users, and means for generating a user avatar in the virtual space and operating the avatar to move to the virtual office. This allows the user to access different consultation booths in the store through the avatar, and when entering the details of their consultation, they can receive personalized responses accompanied by facial expression and voice analysis using an emotion analysis engine.

[1987] The "metaverse space" is a three-dimensional digital space created using virtual reality technology, in which users can interact through avatars.

[1988] A "virtual office" is a virtual area with a specific purpose or function that is located within the metaverse space, and is a virtual space where users can carry out activities and consultations within their respective areas.

[1989] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and extracts emotional information in real time, and includes an emotion recognition and analysis system.

[1990] An "avatar" is a digital alter ego that represents and controls a user in a virtual space, and is an agent that represents the user's actions and intentions.

[1991] "AI means" refers to a means that uses artificial intelligence technology to analyze the content of inquiries received from users and generate appropriate answers and countermeasures.

[1992] "Emotion information" is data relating to the user's mental and emotional state obtained as a result of analysis by the emotion engine.

[1993] A "personalized response" is an appropriate answer or response tailored to an individual user, generated based on the user's specific inquiry content and emotional information.

[1994] "Consultation history" is a record of the consultation content and responses made by the user, and is data stored in a database for future learning and improvement.

[1995] MODE FOR CARRYING OUT THE INVENTION

[1996] The present invention aims to provide an AI system that uses the metaverse space and includes emotion analysis for the purpose of customer support in brick-and-mortar stores. Specific embodiments of the system are described below.

[1997] (server)

[1998] The server first creates a metaverse space and places multiple virtual offices within it. These virtual offices include multiple areas used for different purposes, such as customer support booths and information booths. The server also manages user account information and grants appropriate access privileges to authenticated users.

[1999] (Terminal)

[2000] When a customer uses the system, they first launch the application on their device and enter their login information. The device then sends this information to the server, and if authentication is successful, they are given access to the metaverse space.

[2001] (Log in to the virtual space and start using it)

[2002] Once the user has been authenticated, the server generates and displays an avatar in the virtual space. The avatar represents the user and can move freely within the virtual space. The user can then control the avatar to head to the desired virtual office.

[2003] (AI consultation service and use of emotion engine)

[2004] The user controls an avatar to access a specific virtual office (e.g., a customer support booth). After selecting a consultation booth, a form for entering the details of the consultation is displayed. A video camera and microphone are installed on the terminal to recognize the user's facial expressions and voice. The terminal simultaneously transmits the user's facial expressions and voice to an emotion analysis engine along with the consultation details entered by the user. The emotion engine analyzes the user's emotions in real time. The AI ​​system on the server generates an appropriate answer based on the consultation details and emotional information received. This generated answer is sent from the server to the terminal and displayed to the user. This allows the user to receive emotionally appropriate support.

[2005] (Emotion log and follow-up)

[2006] The server stores all consultation history and emotion logs in a database. This information is used for future learning, improving the accuracy of the AI's responses and improving customer satisfaction. Long-term changes in the user's emotions are also recorded, allowing for more personalized support the next time the service is used.

[2007] Specific examples

[2008] For example, if a user asks, "How do I get a refund for this product?" and their face recorded by the camera shows anxiety, the emotion engine will analyze the anxiety and the AI ​​will provide polite instructions such as, "Click here for the refund procedure. If you have any questions, our staff will be happy to assist you."

[2009] Prompt Sentence Examples

[2010] "We will develop an AI consultation assistant that customers can use in physical stores via their smartphones. We will use an emotion engine to analyze customer emotions and enable the AI ​​to provide appropriate responses."

[2011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2012] Step 1:

[2013] The application is launched on the device and the user enters their login information.

[2014] Input: Username and Password.

[2015] Process: Sends login information to the server for authentication.

[2016] Output: The authentication result (success or failure).

[2017] The server checks the received login information against its database and, if successful, grants the user access.

[2018] Step 2:

[2019] For users who are successfully authenticated, the server generates an avatar in the metaverse space.

[2020] Input: User credentials.

[2021] Processing: Avatar generation processing.

[2022] Output: Avatar information in virtual space.

[2023] The server generates an avatar in the metaverse space for the authenticated user and sends that information to the terminal, making it operable.

[2024] Step 3:

[2025] The user operates an avatar to go to the virtual office and select a specific consultation booth.

[2026] Input: User operation information.

[2027] Processing: Movement processing within virtual space.

[2028] Output: Booth selection information.

[2029] Based on the user's operation, the terminal moves the avatar within the virtual space and transmits information about the selected booth to the server.

[2030] Step 4:

[2031] The user inputs the details of the consultation and transmits facial expression and voice data from the terminal.

[2032] Input: Consultation content, facial expression data, and voice data.

[2033] Processing: Sending input data.

[2034] Output: Sending data to the server.

[2035] The device sends the user's consultation details, facial expression data captured by a video camera, and audio data captured by a microphone to an emotion analysis engine.

[2036] Step 5:

[2037] The emotion engine analyzes facial expressions and voice to generate emotional information.

[2038] Input: Facial expression data, audio data.

[2039] Processing: Sentiment analysis processing.

[2040] Output: Emotional information.

[2041] The emotion engine analyzes the received data, extracts the user's emotional state in real time, and sends that information to the AI ​​system.

[2042] Step 6:

[2043] The AI ​​system generates an appropriate answer based on the content of the consultation and emotional information.

[2044] Input: Consultation content, emotional information.

[2045] Processing: Answer generation processing.

[2046] Output: Answer content.

[2047] The AI ​​means on the server analyzes the consultation content and emotional information, generates answers and follow-up questions, and sends the information to the terminal.

[2048] Step 7:

[2049] The device displays the generated answers to the user and prompts for additional questions or information as needed.

[2050] Input: The response from the AI ​​method.

[2051] Processing: Generate an interface for displaying and inputting answers.

[2052] Output: Display the answer to the user.

[2053] The terminal displays the received answers to the user and generates an interface to await further input from the user.

[2054] Step 8:

[2055] The server stores the consultation details and emotion logs in a database.

[2056] Input: Consultation content, emotion log.

[2057] Processing: Data storage processing.

[2058] Output: Information saved to the database.

[2059] The server stores the consultation details and emotional information in a database and uses it for future learning.

[2060] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2061] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2062] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2063] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2064] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2065] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2066] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2067] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2068] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2069] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2070] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2071] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2072] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2073] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2074] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2075] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2076] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2077] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2078] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2079] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2080] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2081] The following is further disclosed regarding the above embodiment.

[2082] (Claim 1)

[2083] A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space;

[2084] a means for managing user account information and granting access rights to authenticated users;

[2085] A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office;

[2086] An AI method that receives the consultation content from the user's input, analyzes it, and generates an appropriate answer;

[2087] The system includes a means to store consultation history in a database and use it for future learning.

[2088] (Claim 2)

[2089] 2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation topic when accessing a virtual office in the metaverse space.

[2090] (Claim 3)

[2091] 10. The system of claim 1, further comprising: means for generating ongoing follow-up questions and responses based on the consultation content analyzed by the AI ​​means.

[2092] "Example 1"

[2093] (Claim 1)

[2094] A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space;

[2095] a means for managing user account information and granting access rights to authenticated users;

[2096] A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office;

[2097] A means to receive the consultation content from the user's input, analyze it using a generative AI model, and generate an appropriate answer;

[2098] The system includes a means to store consultation history in a database and use it for future learning.

[2099] (Claim 2)

[2100] 2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation topic when accessing a virtual office in the metaverse space.

[2101] (Claim 3)

[2102] The system of claim 1, further comprising means for generating ongoing follow-up questions and responses based on the consultation content analyzed by the generative AI model.

[2103] "Application Example 1"

[2104] (Claim 1)

[2105] A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space;

[2106] a means for managing user account information and granting access rights to authenticated users;

[2107] A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office;

[2108] A generation AI means that receives the consultation content from the user's input, analyzes it, and generates an appropriate answer;

[2109] A means to store consultation history in a database and use it for future learning;

[2110] AI tools to guide procedures to support maintenance and inspection work in factories and provide solutions when problems occur,

[2111] A means for factory workers and engineers to carry out maintenance and inspection work in a virtual space using smartphones, smart glasses, or head-mounted displays,

[2112] A system including:

[2113] (Claim 2)

[2114] 2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation topic when accessing a virtual office in the metaverse space.

[2115] (Claim 3)

[2116] The system of claim 1, further comprising means for generating continuous follow-up questions and countermeasures based on the consultation content analyzed by the generating AI means.

[2117] "Example 2: Combining Emotion Engines"

[2118] (Claim 1)

[2119] A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space;

[2120] a means for managing user account information and granting access rights to authenticated users;

[2121] A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office;

[2122] An AI method that receives the consultation content from the user's input, analyzes it, and generates an appropriate answer;

[2123] emotion engine means for recognizing and analyzing the facial expressions and voice of a user;

[2124] A means for generating follow-up questions and countermeasures according to the emotion based on the emotion information obtained from the emotion engine;

[2125] The system includes a means to store consultation history and emotion logs in a database and use them for future learning.

[2126] (Claim 2)

[2127] 2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation topic when accessing a virtual office in the metaverse space.

[2128] (Claim 3)

[2129] 10. The system of claim 1, further comprising: means for generating continuous follow-up questions and countermeasures based on the consultation content and emotional information analyzed by the AI ​​means.

[2130] "Application example 2 when combining emotion engines"

[2131] (Claim 1)

[2132] A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space;

[2133] a means for managing user account information and granting access rights to authenticated users;

[2134] A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office;

[2135] An AI method that receives the consultation content from the user's input, analyzes it, and generates an appropriate answer;

[2136] A means for using an emotion engine that analyzes the user's facial expressions and voice to determine emotions;

[2137] a means for providing a personalized response based on the emotional information;

[2138] The system includes a means to store consultation history in a database and use it for future learning.

[2139] (Claim 2)

[2140] 2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation topic when accessing a virtual office in the metaverse space.

[2141] (Claim 3)

[2142] 10. The system of claim 1, further comprising: means for generating ongoing follow-up questions and responses based on the consultation content and emotional information analyzed by the AI ​​means. [Explanation of symbols]

[2143] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for generating a metaverse space and arranging a plurality of virtual offices within the virtual space; a means for managing user account information and granting access rights to authenticated users; A means for generating an avatar of a user in a virtual space and operating the avatar to move to the virtual office; An AI method that receives the consultation content from the user's input, analyzes it, and generates an appropriate answer; The system includes a means to store consultation history in a database and use it for future learning.

2. The system according to claim 1, further comprising means for enabling a user to select a booth related to a specific consultation subject when accessing the virtual office in the metaverse space.

3. The system of claim 1 , further comprising means for generating continuous follow-up questions and countermeasures based on the consultation content analyzed by the AI ​​means.

Images