system

Abstract

Provide a system. [Solution] an account authentication means for users to access the online platform and perform login authentication; A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user; means for generating a chat interface and corresponding resources for exchanging messages in real time; A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; A means of keeping a log of all conversations and interactions; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional online customer service systems resulted in long waiting times and no guarantee that users would receive optimal service. Furthermore, misunderstandings and misinformation frequently occurred between users and customer service staff, resulting in an increase in complaints and inquiries. Furthermore, compared to brick-and-mortar stores, online systems lacked the level of trust and familiarity, making it difficult to improve customer satisfaction. [Means for solving the problem]

[0005] The present invention provides a system that allows users to receive face-to-face customer service online with the same peace of mind as visiting a brick-and-mortar store, comprising the following means: First, the user accesses an online platform and logs in using account authentication means. Next, a service selection means is provided that allows the user to select chat support or avatar video and audio support. In the case of chat support, the system includes means for generating a chat interface and response resources for exchanging messages in real time. In the case of avatar support, the system includes means for generating an avatar and executing a scenario that responds to the user via audio and video. Furthermore, the system provides data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests, and includes means for saving logs of all conversations and interactions. In this way, a system is realized that allows users to receive optimal service with peace of mind, while reducing the occurrence of misinformation and complaints.

[0006] An "online platform" refers to a virtual environment in which users can perform various operations using services and functions provided over the Internet.

[0007] "Account authentication method" refers to a mechanism that uses information such as user ID and password to authenticate a user to confirm that they are a legitimate user.

[0008] "Service selection means" refers to an interface or function that allows a user to select an appropriate service from multiple options.

[0009] "Chat support" refers to a form of communication between users and staff by exchanging text messages in real time.

[0010] "Avatar video and audio support" refers to a method in which a computer-generated virtual character responds to the user through audio and video.

[0011] "Chat interface" refers to the screen and functionality that allows users and staff to exchange text messages.

[0012] "Support resources" refers to staff and virtual characters required to respond to inquiries and requests from users.

[0013] "Data analysis and plan generation means" refers to a mechanism that analyzes users' needs and requests and automatically generates the optimal proposed plan.

[0014] "Log storage means" refers to a mechanism that records all conversations and interactions that take place within the system and stores them in a form that can be referenced at a later date. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] As an embodiment of the present invention, a system is provided that allows users to receive face-to-face customer service online with the same feeling as visiting a real store. Specific processing and operation of the system are described in detail below.

[0037] 1. User Access and Login

[0038] A user accesses the system's URL using a web browser. The server sends the HTML, CSS, and JavaScript (registered trademark) files for the top page to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the login button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[0039] 2. Service Selection

[0040] On the dashboard, the user selects either "chat support" or "avatar video and audio support." The device sends this selection information to the server, which then assigns available resources (staff or avatars) based on the user's selection.

[0041] 3. Customer service

[0042] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. These messages are sent to the server and exchanged in real time, and all messages are stored on the server as a log.

[0043] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0044] 4. Confirmation of requests and provision of plans

[0045] When a user communicates their needs and desires through chat or interaction with an avatar, the server records this and stores it in a database. The server then analyzes the recorded information to determine the optimal plan and generates plan information. The generated plan information is then sent from the server to the user's device and displayed on the device.

[0046] 5. Logging and terminating the session

[0047] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0048] As a concrete example, a user accesses a service site, logs in, selects chat options, exchanges messages with a staff member in real time, receives a service plan proposal, and finally ends the session. This allows the user to receive the most suitable service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] A user accesses the system's URL using a web browser.

[0052] Step 2:

[0053] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[0054] Step 3:

[0055] The device displays the top page in the browser.

[0056] Step 4:

[0057] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[0058] Step 5:

[0059] The terminal sends the entered account information to the server as a POST request.

[0060] Step 6:

[0061] The server checks the received account information against user information in its database.

[0062] Step 7:

[0063] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[0064] Step 8:

[0065] The device displays the dashboard page in a browser.

[0066] Step 9:

[0067] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[0068] Step 10:

[0069] The terminal sends the selected information to the server as a POST request.

[0070] Step 11:

[0071] The server allocates available resources (staff or avatars) based on the user's selection.

[0072] Step 12:

[0073] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[0074] Step 13:

[0075] The terminal will display the chat interface.

[0076] Step 14:

[0077] The user types a message in the chat interface and clicks the send button.

[0078] Step 15:

[0079] The device sends a chat message to the server.

[0080] Step 16:

[0081] The server delivers the received message to the staff member in charge, and similarly delivers the staff member's reply to the user.

[0082] Step 17:

[0083] The server stores all message logs.

[0084] Step 18:

[0085] If the server supports avatars, it generates avatars and prepares the scenario.

[0086] Step 19:

[0087] The server transmits the interface of the virtual correspondence room to the user's terminal.

[0088] Step 20:

[0089] The device displays the virtual response room interface.

[0090] Step 21:

[0091] The server executes the scenario in which the avatar responds to the user with voice and video.

[0092] Step 22:

[0093] The server generates appropriate responses to the user's questions and requests and provides them through an avatar.

[0094] Step 23:

[0095] Users communicate their needs and desires through chat or interaction with an avatar.

[0096] Step 24:

[0097] The server stores the requests received from the user in a database.

[0098] Step 25:

[0099] The server analyzes and generates the information for the best plan based on the user's needs.

[0100] Step 26:

[0101] The server transmits the generated plan information to the user's terminal.

[0102] Step 27:

[0103] The device will display the plan information on the screen.

[0104] Step 28:

[0105] The server stores a log of all conversations and interactions in a database.

[0106] Step 29:

[0107] The user clicks the "Exit" button to end the session.

[0108] Step 30:

[0109] The terminal sends a termination request to the server.

[0110] Step 31:

[0111] The server ends the session and logs out the user.

[0112] Step 32:

[0113] The terminal displays the logout completion screen.

[0114] Example 1

[0115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] Conventional online customer service systems make it difficult for users to experience the same sense of security as face-to-face customer service in a brick-and-mortar store, and they face challenges in quickly and appropriately responding to user needs and requests. Furthermore, there is a lack of effective ways to store and analyze conversation history and logs for future reference or to improve services. Furthermore, when using avatars, the technological means to provide accurate responses to user questions and requests are not yet fully developed.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0118] In this invention, the server includes an account authentication means for allowing a user to access the online platform and perform login authentication, a service selection means for selecting between chat support or avatar video and audio support from the user, a means for generating a chat interface and response resources for exchanging messages in real time, a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests, a means for saving a log of all dialogues and exchanges, and, in the case of avatar video and audio support, a means for using a generative AI model to analyze questions and requests from the user and generate appropriate responses. This allows users to receive reliable face-to-face customer service online just like in a physical store, provides optimal services tailored to the user's needs, and effectively saves and analyzes the history of all dialogues.

[0119] "Account authentication means" refers to the means by which a user accesses the online platform and performs login authentication.

[0120] The "service selection means" is a means for a user to select a service that supports chat or video and audio using an avatar.

[0121] The "chat interface" is an interface that allows users and staff to exchange messages in real time.

[0122] The "means for generating response resources" refers to a means for generating resources (staff or avatars) required for chat response or avatar response.

[0123] The "data analysis and plan generation means" is a means for analyzing data to propose an optimal plan based on the user's needs and requests, and generating a plan.

[0124] "Means for storing logs" refers to a means for storing logs of all conversations and interactions.

[0125] "Means for using a generative AI model" refers to means for using a generative AI model to analyze questions or requests from users and generate appropriate responses in the case of video and audio support using avatars.

[0126] MODE FOR CARRYING OUT THE INVENTION

[0127] As an embodiment of this invention, we provide a system that allows users to receive face-to-face customer service online with the same sense of security as in a physical store. The system is described in detail below. Specific names of the hardware and software used are also specified.

[0128] This system is based on the premise that users will use a web browser to access the service. The server runs on web server software such as Apache (registered trademark) or Nginx. A relational database such as MySQL (registered trademark) or PostgreSQL is used as the database. The front end that users access is built using web technologies such as HTML, CSS, and JavaScript.

[0129] 1. User Access and Login

[0130] When a user accesses the system's URL using a web browser (e.g., Chrome or Firefox), the server sends the top page HTML, CSS, and JavaScript files to the user's device. The user's device receives this and displays the top page in the browser. The user enters their account information (user ID and password) and clicks the login button. The device sends this information to the server, which checks it against a database for authentication. If authentication is successful, the server generates a session ID and sends the dashboard page to the user's device, which displays it.

[0131] 2. Service Selection

[0132] Users select the option "chat support" or "avatar video and audio support" on the dashboard. The user's device sends the selection to the server, which then allocates the necessary resources (staff or avatars) based on the selection.

[0133] 3. Customer service

[0134] If chat is selected, the server creates a dedicated chat room and sends the chat interface HTML and JavaScript to the user's and staff's devices. The devices display this, allowing for the exchange of text messages in real time. Text messages are exchanged via the server and are all saved as logs.

[0135] If avatar-based video and audio support is selected, the server generates a 3D avatar and prepares the scenario. The virtual support room interface is sent to the user's device, which displays it. The user's questions and requests are sent from the device to the server, which uses a generative AI model (e.g., ChatGPT (registered trademark)) to generate appropriate responses and provide them through the avatar. All of these interactions are also logged.

[0136] 4. Confirmation of requests and provision of plans

[0137] The needs and requests communicated by users through chat or avatars are recorded on the server. The server analyzes the recorded data and generates an optimal plan. This plan information is sent from the server to the user's device and displayed on the device.

[0138] 5. Logging and terminating the session

[0139] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[0140] Examples of prompt statements

[0141] As a concrete example, let's assume that the system implements a series of steps in which a user accesses a service site, logs in, selects chat support, and exchanges messages with a staff member in real time. By inputting the following prompt sentences into the generative AI model, we can specifically explain how the system works.

[0142] Example prompt sentence:

[0143] "You have devised a system that allows users to safely receive face-to-face customer service online. Based on the specific scenario below, please write down how a user would use this service. The scenario includes accessing the service from a web browser, logging in, chatting, exchanging messages with a staff member, and receiving a service plan proposal."

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1: User accesses the system's URL in a web browser

[0146] A user accesses the system's URL using a web browser (e.g., Chrome or Firefox). The server generates the HTML, CSS, and JavaScript files for the top page and sends them to the user's device. The device receives these files and displays the top page in the browser.

[0147] Input: User enters a URL

[0148] Output: The HTML, CSS, and JavaScript files for the top page are sent to the terminal, and the top page is displayed in the browser.

[0149] Step 2: User enters account information and logs in

[0150] The user enters their account information (user ID and password) into the input fields on the top page and clicks the login button. The device uses JavaScript to perform basic format checks on the input data and then sends it to the server.

[0151] Input: Enter your account information (user ID, password)

[0152] Output: The input data is sent to the server

[0153] Step 3: The server authenticates the account

[0154] The server checks the received account information against its database. If authentication is successful, the server generates a session ID, dynamically generates a dashboard page with this information, and sends it to the user's device.

[0155] Input: Entered account information

[0156] Data processing and data calculation: Matching with database and generating session ID

[0157] Output: Dashboard page and session ID sent to device

[0158] Step 4: User selects service

[0159] The user selects "chat support" or "avatar video and audio support" from the options on the dashboard, and the device sends the selection information to the server.

[0160] Input: Select service (chat support, avatar support)

[0161] Output: Selection information is sent to the server

[0162] Step 5: Server allocates resources

[0163] Based on the received selection information, the server allocates the necessary resources (staff or avatars) and generates the necessary interfaces, such as chat rooms or virtual interaction rooms.

[0164] Input: Service selection information

[0165] Data processing and data calculation: Allocation of corresponding resources and generation of interfaces

[0166] Output: The appropriate interface is sent to the user's terminal.

[0167] Step 6: Start serving customers via chat

[0168] If the user selects chat support, a chat room is created and a chat interface is sent to the user's and staff's terminals.

[0169] Input: Chat response selection

[0170] Output: The chat room and chat interface are sent to the device.

[0171] Step 7: Exchange chat messages and save logs

[0172] Users and staff can type and send text messages, and the terminals send the messages to the server, which exchanges messages in real time and stores all messages as a log.

[0173] Input: Messages entered by users and staff

[0174] Data processing and data calculation: Message sending and receiving and log storage

[0175] Output: Real-time exchange of messages and storage of logs

[0176] Step 8: Start serving customers with avatar support

[0177] If the user selects avatar-based video and audio support, the server generates an avatar and prepares the scenario. It also generates a virtual support room interface and sends it to the user's device.

[0178] Input: Avatar-enabled selection

[0179] Output: The virtual room and avatar are sent to the user's device.

[0180] Step 9: Avatar response and logging

[0181] When a user types a question or request, the device sends it to the server, which uses a generative AI model to generate a response and delivers it through an avatar. All interactions are logged.

[0182] Input: User questions or requests

[0183] Data processing and data calculation: response generation and logging using generative AI models

[0184] Output: Avatar response and log storage

[0185] Step 10: Confirm your needs and provide a plan

[0186] The needs and requests communicated by the user are recorded on the server. The server analyzes the recorded data and generates an optimal plan. The generated plan information is sent to the user's device and displayed on the device.

[0187] Input: User needs and desires

[0188] Data processing and data calculations: Data analysis and plan generation

[0189] Output: Plan information is sent to the user's device

[0190] Step 11: Save the log and end the session

[0191] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[0192] Input: User request to terminate

[0193] Data processing and data calculation: Session termination processing and logout processing

[0194] Output: The logout completion screen is displayed on the terminal.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] With the development of online shopping, users are increasingly seeking the same sense of security and experience as face-to-face customer service in a physical store. However, current online platforms make it difficult for users to receive immediate, personalized assistance, especially when they need detailed information about a product or are unsure about a purchase, resulting in a decline in satisfaction and a decrease in purchasing motivation. Additionally, there is a lack of mechanisms for appropriately logging and storing conversation content and customer needs, and utilizing them to improve services in the future.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes a means for users to interact with avatars and specialized staff in real time through a shopping experience in a virtual store, an account authentication means, and a service selection means for users to choose between chat support or avatar video and audio support. This allows users to enjoy the same sense of security as face-to-face customer service in a physical store while online, and to receive optimal suggestions based on their individual needs. In addition, by storing a log of all conversations and interactions, this information can be used to improve services in the future.

[0200] "User" means any individual or group of people who accesses and uses the functionality of the system or platform.

[0201] "Online Platform" refers to software or websites provided over the Internet that offer specific services or functions to users.

[0202] "Account authentication means" refers to a mechanism for verifying that a user is a legitimate user when accessing a system or platform.

[0203] "Service selection means" refers to an interface or mechanism that allows a user to select one of multiple service options provided.

[0204] "Chat interface" refers to a screen or function that allows users and support staff to exchange text messages in real time.

[0205] "Supporting resources" refers to the technical elements such as servers, network equipment, and software required for chat support and avatar video and audio support.

[0206] "Data analysis and plan generation means" refers to a mechanism for collecting and analyzing data to propose an optimal service plan based on the user's needs and requests, and for generating a plan based on the results of the analysis.

[0207] "Log storage means" refers to a mechanism for storing records of all conversations and interactions so that they can be referenced at a later date.

[0208] A "virtual store" refers to a virtual store space provided on the Internet, where users can browse and purchase products just as if they were in a physical store.

[0209] An "avatar" is a virtual person or character used to interact with users and has the ability to respond through voice and video.

[0210] "Specialist staff" refers to personnel who have specialized knowledge about products and services and who provide assistance to users.

[0211] This invention is an online system that allows users to shop in a virtual store. This system provides users with a sense of security similar to face-to-face customer service in a real store, and includes a function that allows users to interact with avatars and professional staff in real time.

[0212] System Overview

[0213] The system mainly consists of the following components:

[0214] 1. Account Authentication Methods

[0215] 2. Service Selection Method

[0216] 3. Chat interface and corresponding resource generation means

[0217] 4. Data Analysis and Plan Generation Methods

[0218] 5. Log storage method

[0219] 6. Real-time interaction with avatars and expert staff

[0220] Account authentication method

[0221] It is a means for users to access online platforms and perform login authentication. It uses software such as Flask and SQLAlchemy to perform the user ID and password authentication process. This is the gateway for users to access the system and is a function to maintain security.

[0222] Service Selection Method

[0223] After logging in, users can choose between chat support or avatar video and audio support. This selection information is sent to the server, which allocates appropriate resources based on the user's selection. The user interface is built using HTML, CSS, and JavaScript.

[0224] Chat interface and corresponding resource generation means

[0225] If chat support is selected, the server generates a dedicated chat room and provides an interface for exchanging text messages in real time, using real-time communication technologies such as Flask-SocketIO. This allows users and expert staff to communicate in real time.

[0226] Data Analysis and Plan Generation Tools

[0227] It is a means of proposing optimal service plans based on the user's needs and requests. Here, machine learning models and generative AI models are used to collect and analyze data obtained from users, thereby generating and providing individually optimized plans for the user.

[0228] Log storage method

[0229] It is a way to store all conversations and interactions as logs, which can be referenced later and used to improve the service. The database is SQLite or another RDBMS.

[0230] Real-time interaction with avatars and expert staff

[0231] If the user selects avatar video and audio support, the server generates an avatar and executes a scenario that responds to the user with audio and video. This process utilizes text-to-speech (TTS) and video rendering technologies. The avatar's movements and responses are based on a generative AI model.

[0232] Examples of concrete examples and prompts

[0233] Specific examples

[0234] The user launches the virtual store app and logs in by entering their user ID and password on the login screen. They then enter a chat room and can ask questions about products in real time with a specialist staff member. An avatar will explain the product using video and audio based on the scenario, and if the user is satisfied, they can make a purchase.

[0235] Prompt Sentence Examples

[0236] User launches app and enters login details:

[0237] "Please enter your login information"

[0238] Join the chat room:

[0239] "Joined the room"

[0240] Chat message exchange:

[0241] User: "Please describe this product."

[0242] Staff: "Please wait. This product has the following features..."

[0243] Avatar response:

[0244] "Here are some products we recommend to you..."

[0245] This system allows users to enjoy the same high-quality service online as they would in a physical store, and also enables the storage of logs that will lead to improved service quality.

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] A user accesses the online platform and enters their user ID and password on the login screen for authentication. The server receives the entered user ID and password information and verifies them against the database. If authentication is successful, the server starts the user's session, generates a dashboard page, and sends it to the user's device.

[0249] Input: User ID, Password

[0250] Data processing: Matching with database, creating sessions

[0251] Output: Dashboard page displayed

[0252] Step 2:

[0253] The user selects the "chat support" or "avatar video and audio support" service on the dashboard. The device sends the user's selection information to the server, which then allocates the appropriate resource (specialist staff or avatar).

[0254] Input: User's service selection information

[0255] Data processing: receiving selection information, allocating resources

[0256] Output: Prepare corresponding resources

[0257] Step 3:

[0258] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The user and the staff exchange text messages in real time, and all messages are sent to the server and saved as a log.

[0259] Input: (Chat selection) User message, staff message

[0260] Data processing: Chat room creation, message exchange, message log storage

[0261] Output: Real-time chat

[0262] Step 4:

[0263] If the user selects video and audio support via an avatar, the server generates an avatar and executes a scenario in which the user responds via video and audio. When the user makes a question or request, the server analyzes it using a generative AI model, generates an appropriate response, and provides it through the avatar.

[0264] Input: (Avatar selection) User question or request

[0265] Data processing: Avatar generation, analysis using generative AI models, response generation

[0266] Output: Avatar video and audio response

[0267] Step 5:

[0268] The server analyzes the data based on the user's needs and requests and generates an optimal service plan. The generated service plan information is sent from the server to the user's terminal and displayed on the user's terminal.

[0269] Input: User needs and desires

[0270] Data processing: Data collection and analysis, service plan generation

[0271] Output: Display service plan information

[0272] Step 6:

[0273] By saving a log of all conversations and interactions, they can be managed in a form that can be referenced later. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs out the user.

[0274] Input: Content of conversation or exchange, end request

[0275] Data processing: saving logs, ending sessions

[0276] Output: Display of logout completion screen

[0277] In this way, a system is realized that allows users to receive real-time support throughout their shopping experience in a virtual store.

[0278] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0279] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a physical store, and that provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[0280] 1. User Access and Login

[0281] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the submit button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[0282] 2. Service Selection

[0283] On the dashboard, the user selects either "chat support" or "avatar support." The device sends the selection information to the server, which then assigns available resources (staff or avatars) according to the user's selection.

[0284] 3. Customer service

[0285] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0286] If the user selects avatar-based video and audio support, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0287] 4. Confirmation of requests and provision of plans

[0288] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The server then analyzes the recorded information to find the optimal plan and generates plan information. The generated plan information is sent from the server to the user's device and displayed on the device.

[0289] 5. Introducing the Emotion Engine

[0290] In this embodiment, an emotion engine is added. The emotion engine analyzes the user's text message, voice, and facial expression to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes emotions from the user's text message and feeds the information back to the server. Based on the feedback information, the server provides a response method to the chat support resource in real time.

[0291] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0292] 6. Logging and terminating sessions

[0293] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0294] As a specific example, a user accesses a service site, logs in, selects chat, and exchanges messages with a staff member in real time. The emotion engine analyzes the user's emotions and provides appropriate feedback to the staff member, enabling them to provide the most appropriate service to the user. There is also a series of steps that provide plan information generated based on the user's needs and requests. This allows the user to receive the most appropriate service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems from occurring.

[0295] The processing flow will be explained below.

[0296] Step 1:

[0297] A user accesses the system's URL using a web browser.

[0298] Step 2:

[0299] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[0300] Step 3:

[0301] The device displays the top page in the browser.

[0302] Step 4:

[0303] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[0304] Step 5:

[0305] The terminal sends the entered account information to the server as a POST request.

[0306] Step 6:

[0307] The server checks the received account information against user information in its database.

[0308] Step 7:

[0309] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[0310] Step 8:

[0311] The device displays the dashboard page in a browser.

[0312] Step 9:

[0313] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[0314] Step 10:

[0315] The terminal sends the selected information to the server as a POST request.

[0316] Step 11:

[0317] The server allocates available resources (staff or avatars) based on the user's selection.

[0318] Step 12:

[0319] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[0320] Step 13:

[0321] The terminal will display the chat interface.

[0322] Step 14:

[0323] The user types a message in the chat interface and clicks the send button.

[0324] Step 15:

[0325] The device sends a chat message to the server.

[0326] Step 16:

[0327] The server delivers the received message to the staff member in charge, and the staff member's reply is also delivered to the user. In addition, an emotion engine within the chat interface analyzes the user's message in real time and feeds the emotional information back to the server.

[0328] Step 17:

[0329] Based on the emotional feedback information, the server provides the appropriate response method to the staff member in charge, which is reflected on the chat interface.

[0330] Step 18:

[0331] The server stores all messages and sentiment analysis logs.

[0332] Step 19:

[0333] If the server supports avatars, it generates avatars and prepares the scenario.

[0334] Step 20:

[0335] The server transmits the interface of the virtual correspondence room to the user's terminal.

[0336] Step 21:

[0337] The device displays the virtual response room interface.

[0338] Step 22:

[0339] The server executes a scenario in which the avatar responds to the user with voice and video. The emotion engine within the avatar interface analyzes the user's voice and facial expressions in real time and feeds the emotion information back to the server.

[0340] Step 23:

[0341] The server adjusts the avatar's response method based on the emotional feedback information and provides it to the user.

[0342] Step 24:

[0343] Users communicate their needs and desires through chat or interaction with an avatar.

[0344] Step 25:

[0345] The server records and stores the user's requests and needs in a database.

[0346] Step 26:

[0347] The server analyzes the optimum plan based on the recorded information and generates plan information.

[0348] Step 27:

[0349] The server transmits the generated plan information to the user's terminal.

[0350] Step 28:

[0351] The device will display the plan information in a browser.

[0352] Step 29:

[0353] The server stores a log of all conversations and interactions in a database.

[0354] Step 30:

[0355] The user clicks the "Exit" button to end the session.

[0356] Step 31:

[0357] The terminal sends a termination request to the server.

[0358] Step 32:

[0359] The server ends the session and logs out the user.

[0360] Step 33:

[0361] The terminal displays the logout completion screen.

[0362] Example 2

[0363] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Modern online services face the challenge of making it difficult for users to obtain an experience equivalent to that of face-to-face customer service in a brick-and-mortar store. Furthermore, there is a need for systems that can accurately understand the user's emotional state and provide services based on that understanding, but conventional systems are inadequate in this regard. Furthermore, there is a need for systems that can log all conversations and interactions to prevent problems before they occur.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0366] In this invention, the server includes an account authentication means for users to access the online platform and perform login authentication; a service selection means for users to select between chat support or avatar video and audio support; a means for generating a chat interface and response resources for exchanging messages in real time; a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; a means for saving logs of all conversations and interactions; a means including an emotion engine that analyzes the user's text messages, voice, and facial expressions to recognize their emotional state; and a means for providing a response method based on feedback information from the emotion engine. This allows for an online experience equivalent to face-to-face customer service in a physical store, enabling advanced responses based on the user's emotional state. Furthermore, by saving all conversations and interactions as logs, problems can be prevented before they occur.

[0367] "Account authentication means" refers to the means by which a user accesses an online platform, collates the authentication information (such as a user ID and password) required to log in, and verifies its legitimacy.

[0368] The "service selection means" is a means for transmitting information about whether a user selects chat support or video and audio support using an avatar to the server.

[0369] A "chat interface" is a user interface that allows users and staff to exchange text messages in real time.

[0370] The "means for generating corresponding resources" is a means for allocating and generating resources such as appropriate staff members and avatars based on the user's selection.

[0371] The "data analysis and plan generation means" is a means for analyzing data based on the user's needs and requests, and for designing and proposing the most suitable plan.

[0372] A "means for storing logs" is a means for recording all interactions and exchanges between a user and a system and storing them in a form that can be referenced at a later date.

[0373] An "emotion engine" is a technology that analyzes a user's text messages, voice, and facial expressions to recognize the user's emotional state in real time.

[0374] The "means for providing a response method based on feedback information from the emotion engine" is a means for appropriately adjusting the response method of the staff member or avatar based on the emotional state of the user analyzed by the emotion engine.

[0375] The present invention relates to a system that enables users to receive face-to-face customer service online in the same way as visiting a physical store, and provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[0376] This system provides an account authentication method for users to access the online platform and perform login authentication. When a user accesses the system using a web browser, the server sends the HTML, CSS, and JavaScript files for the top page to the user's device, which then displays the top page. When the user enters their user ID and password and clicks the submit button, the device sends this information to the server, which then checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, which the device displays.

[0377] Next, the user selects "chat support" or "avatar support" on the dashboard. The terminal sends this selection information to the server, and the server assigns available resources (staff or avatar) according to the user's selection. For example, if the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0378] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0379] Furthermore, this system has an added emotion engine. The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. For example, in the case of chat support, the emotion engine analyzes the user's emotions from the text messages and feeds the information back to the server. The server provides a response method to the chat support resource in real time based on the feedback information. In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds the information back to the server. The server adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0380] For example, if a user sends a chat message saying, "Please tell me how to use this product," the system will analyze the message and provide appropriate information. If the emotion engine detects anxiety in a message such as, "I'm in a lot of trouble," the system will pass that information on to the appropriate staff member, who will then respond promptly and courteously to the user.

[0381] In this way, this system aims to provide an online experience similar to face-to-face customer service in a physical store. Users can receive optimal service suggestions without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[0382] Examples of prompts include:

[0383] "If the message entered by the user is 'Please tell me how to use the recommended product,' generate an appropriate product description."

[0384] "If the emotion engine interprets the user's facial expression as serious, be a little more careful with the avatar's response tone."

[0385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0386] Step 1:

[0387] The user opens a web browser and accesses the system's URL. The server sends the top page HTML, CSS, and JavaScript files to the user's device. The device receives these files and displays the top page. The user's input is the system's URL, and the output is the display of the top page. Specifically, the user enters the URL in the browser's address bar and presses the Enter key.

[0388] Step 2:

[0389] The user enters a user ID and password into the login form and clicks the submit button. The terminal sends this input information to the server. The server compares the received information with a database and performs authentication. If authentication is successful, the server starts a session for the user, generates a dashboard page, and sends it to the user's terminal. The terminal displays the dashboard page. The input is the user ID and password, and the output is the display of the dashboard page. The specific action is when the user clicks the "Login" button.

[0390] Step 3:

[0391] The user selects "Chat support" or "Avatar support" on the dashboard. The device sends the selection information to the server. The server allocates available resources (staff or avatars) based on the user's selection. The input is the selection information, and the output is the allocation of resources. The specific operation is when the user clicks the "Chat support" or "Avatar support" button.

[0392] Step 4:

[0393] (For chat support) The server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages and saves them as a log. The input is the chat message, and the output is the message saved as a log. The specific action is when the user clicks the "Send message" button.

[0394] Step 5:

[0395] (In the case of avatar support) The server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. When the user asks a question or makes a request, the server analyzes it and generates a response according to the avatar's scenario. The input is the user's question or request, and the output is the avatar's response. In concrete terms, the user speaks into the microphone.

[0396] Step 6:

[0397] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The optimal plan is analyzed and generated based on this data. The server then sends the generated plan information to the user's device, which displays it. The input is the user's needs and desires, and the output is a display of the optimal plan. A specific operation is when the user clicks the "Confirm Plan" button.

[0398] Step 7:

[0399] The system is equipped with an emotion engine that analyzes the user's text messages, voice, and facial expressions. The emotion engine recognizes the user's emotional state in real time and feeds that information back to the server. The server uses the feedback information to adjust the chat response and avatar's response method. The input is the user's message and facial expression, and the output is feedback on how to respond. Specifically, the emotion engine recognizes the user's facial expression.

[0400] Step 8:

[0401] When all dialogue and exchanges are complete, the user clicks the "Exit" button, and the terminal sends an end request to the server. The server ends the session and logs the user out. The terminal displays a logout completion screen. The input is the end request, and the output is a message indicating that the logout has been completed. The specific action is when the user clicks the "Exit" button.

[0402] (Application example 2)

[0403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Conventional online shopping systems struggle to provide the same sense of security and face-to-face customer service as brick-and-mortar stores, and lack the means to alleviate users' dissatisfaction and anxiety. They also lack the ability to understand users' emotional state and respond optimally accordingly. As a result, the user experience remains stagnant, leading to problems with declining customer satisfaction.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an account authentication means for a user to access the online platform and perform login authentication, a service selection means for the user to select from chat support or video and audio support using avatars, a means for generating a chat interface and corresponding resources for exchanging messages in real time, an emotion analysis means for analyzing emotions from the user's text messages and providing feedback, and a means for adjusting the chat support resources and avatar responses based on the emotion analysis results. This allows users to feel as secure online as if they were in a physical store, and receive optimal service in real time according to their emotions.

[0406] "Account authentication means" refers to the function by which a user accesses the online platform and enters an appropriate ID and password for authentication.

[0407] The "service selection means" is a function that provides an interface for the user to select either chat support or video and audio support using an avatar.

[0408] A "chat interface" is an interface that allows users and staff to exchange text messages in real time.

[0409] "Response resources" are staff or system resources assigned to provide services to users.

[0410] The "data analysis means" is a function that performs data analysis to propose the optimal plan based on the user's needs and requests.

[0411] The "plan generation means" is a function that generates an optimal proposed plan for the user based on the results of data analysis.

[0412] The "log storage means" is a function that stores the history of all conversations and interactions and makes them available for later reference as needed.

[0413] The "emotion analysis means" is a function that analyzes emotions from the user's text message and provides feedback on the results.

[0414] The "response adjustment means" is a function that adjusts the responses of chat resources and avatars based on the results of emotion analysis to provide an appropriate response.

[0415] A "chat room" is a dedicated virtual space where users and staff can exchange text messages in real time.

[0416] The "avatar generation means" is a function that generates an avatar and executes a scenario that responds to the user with audio and video.

[0417] The "scenario execution means" is a function that enables an avatar to respond to the user with voice and video based on a predefined response scenario.

[0418] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a real store, and also provides advanced services using an emotion engine. The system of the present invention operates as follows.

[0419] First, a user accesses the system's URL using a web browser and performs login authentication through the web browser interface. When the user enters their account information (user ID and password) and sends it to the server, the server checks it against a database, and if authentication is successful, the user's session begins. This authentication method prevents unauthorized access and ensures a secure connection.

[0420] After logging in, the user selects "chat support" or "avatar support" on the dashboard. The service selection means allows the user to select a chat interface for exchanging messages in real time as needed. Alternatively, the user can select voice and video responses from an avatar. The service selection means transmits the user's selection information to the server, and the server allocates available resources.

[0421] If the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the user's terminal and the terminal of the staff member in charge for display. Through this interface, the user and staff member exchange messages in real time, and the server saves all messages as a log. This log saving means can be used to prevent problems from occurring or as reference information later.

[0422] If the user selects video and audio support using an avatar, the server generates an avatar and executes a scenario that responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar. The avatar generation means and scenario execution means make it possible to provide a high level of interaction to the user.

[0423] Furthermore, in this embodiment, an emotion engine is added. The emotion analysis means analyzes emotions from the user's text message, voice, and facial expression. In the case of chat correspondence, the emotion engine analyzes the user's text message and feeds back the information to the server. The response adjustment means allows the server to provide an appropriate response method to the chat correspondence resource in real time based on the feedback information.

[0424] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions, sends the feedback information to the server, and the server adjusts the avatar's response method, thereby enabling the optimal response to the user's emotions.

[0425] As a specific example, if a user types into the chat interface, "I'm looking for a new smartphone. Which one do you recommend?", the emotion engine will analyze the emotion of the user's message, and based on that information, a staff member will respond by saying, "We will suggest the best options for you."

[0426] Examples of prompt sentences that may be used in the present invention include the following:

[0427] Prompt: Parse the user's message to recognize their sentiment and generate an appropriate response. Message: "I'm looking for a new smartphone. Which one would you recommend?"

[0428] The hardware and software used are as follows:

[0429] Web browser: displaying the user interface

[0430] Server: Provides API endpoints, manages sessions, operates the database, and runs the emotion engine

[0431] EmotionRecognition library: Analyzes user text messages and recognizes emotions in real time

[0432] Through these measures, users can feel as secure online as if they were in a physical store, and can receive optimal service tailored to their emotions in real time.

[0433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] A user accesses the system's URL using a web browser. The terminal receives the top page HTML, CSS, and JavaScript files from the server and displays the top page, allowing the user to access the login screen.

[0436] Input: System URL

[0437] Output: Login screen displayed in web browser

[0438] What happens: A user enters a URL into the browser's address bar and loads a page.

[0439] Step 2:

[0440] The user enters account information (user ID and password) and clicks the submit button. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, the server starts the user's session, generates a dashboard page, sends it to the terminal, and the terminal displays it.

[0441] Input: User ID, Password

[0442] Output: Dashboard page upon successful authentication

[0443] Specific operation: The user enters their ID and password in the login form and presses the "Login" button.

[0444] Step 3:

[0445] The user selects either "chat support" or "avatar support" on the dashboard. The terminal sends the selection information to the server, and the server assigns available resources (staff or avatars).

[0446] Input: User selection information

[0447] Output: Resource allocation according to the selection

[0448] What happens: A user clicks on an option on a dashboard.

[0449] Step 4:

[0450] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0451] Input: User's chat response selection, text message

[0452] Output: Chat interface, real-time message exchange, logs

[0453] Specific operation: The server creates a chat room, and users and staff members can input, send, and receive chat messages.

[0454] Step 5:

[0455] If the user selects avatar support, the server generates an avatar and sends the virtual support room interface to the user's device. The server analyzes the user's questions and requests, generates appropriate responses, and provides them through the avatar.

[0456] Input: User avatar-enabled selection, user question or request

[0457] Output: Virtual interaction room, avatar responses

[0458] Specific operation: The server generates an avatar and executes a scenario to respond to the user's questions.

[0459] Step 6:

[0460] The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes the user's text messages and feeds the information back to the server. Based on the feedback information, the server provides chat support resources with appropriate responses in real time.

[0461] Input: User's text message, voice, and facial expression information

[0462] Output: Sentiment analysis results, feedback information, and appropriate response methods

[0463] Specific behavior: The emotion engine analyzes messages and voice and provides appropriate feedback instantly.

[0464] Step 7:

[0465] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0466] Input: User's voice and facial expression

[0467] Output: Sentiment analysis results, feedback information, appropriate avatar responses

[0468] Specific behavior: The emotion engine analyzes the user's voice and facial expressions and adjusts the avatar's response.

[0469] Step 8:

[0470] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0471] Input: User's exit request

[0472] Output: Interaction log, session termination, logout completion screen

[0473] Specific operation: The user presses the end button, and the server performs the session termination process.

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] As an embodiment of the present invention, a system is provided that allows users to receive face-to-face customer service online with the same feeling as visiting a real store. Specific processing and operation of the system are described in detail below.

[0491] 1. User Access and Login

[0492] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the login button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[0493] 2. Service Selection

[0494] On the dashboard, the user selects either "chat support" or "avatar video and audio support." The device sends this selection information to the server, which then assigns available resources (staff or avatars) based on the user's selection.

[0495] 3. Customer service

[0496] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. These messages are sent to the server and exchanged in real time, and all messages are stored on the server as a log.

[0497] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0498] 4. Confirmation of requests and provision of plans

[0499] When a user communicates their needs and desires through chat or interaction with an avatar, the server records this and stores it in a database. The server then analyzes the recorded information to determine the optimal plan and generates plan information. The generated plan information is then sent from the server to the user's device and displayed on the device.

[0500] 5. Logging and terminating the session

[0501] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0502] As a concrete example, a user accesses a service site, logs in, selects chat options, exchanges messages with a staff member in real time, receives a service plan proposal, and finally ends the session. This allows the user to receive the most suitable service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[0503] The processing flow will be explained below.

[0504] Step 1:

[0505] A user accesses the system's URL using a web browser.

[0506] Step 2:

[0507] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[0508] Step 3:

[0509] The device displays the top page in the browser.

[0510] Step 4:

[0511] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[0512] Step 5:

[0513] The terminal sends the entered account information to the server as a POST request.

[0514] Step 6:

[0515] The server checks the received account information against user information in its database.

[0516] Step 7:

[0517] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[0518] Step 8:

[0519] The device displays the dashboard page in a browser.

[0520] Step 9:

[0521] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[0522] Step 10:

[0523] The terminal sends the selected information to the server as a POST request.

[0524] Step 11:

[0525] The server allocates available resources (staff or avatars) based on the user's selection.

[0526] Step 12:

[0527] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[0528] Step 13:

[0529] The terminal will display the chat interface.

[0530] Step 14:

[0531] The user types a message in the chat interface and clicks the send button.

[0532] Step 15:

[0533] The device sends a chat message to the server.

[0534] Step 16:

[0535] The server delivers the received message to the staff member in charge, and similarly delivers the staff member's reply to the user.

[0536] Step 17:

[0537] The server stores all message logs.

[0538] Step 18:

[0539] If the server supports avatars, it generates avatars and prepares the scenario.

[0540] Step 19:

[0541] The server transmits the interface of the virtual correspondence room to the user's terminal.

[0542] Step 20:

[0543] The device displays the virtual response room interface.

[0544] Step 21:

[0545] The server executes the scenario in which the avatar responds to the user with voice and video.

[0546] Step 22:

[0547] The server generates appropriate responses to the user's questions and requests and provides them through an avatar.

[0548] Step 23:

[0549] Users communicate their needs and desires through chat or interaction with an avatar.

[0550] Step 24:

[0551] The server stores the requests received from the user in a database.

[0552] Step 25:

[0553] The server analyzes and generates the information for the best plan based on the user's needs.

[0554] Step 26:

[0555] The server transmits the generated plan information to the user's terminal.

[0556] Step 27:

[0557] The device will display the plan information on the screen.

[0558] Step 28:

[0559] The server stores a log of all conversations and interactions in a database.

[0560] Step 29:

[0561] The user clicks the "Exit" button to end the session.

[0562] Step 30:

[0563] The terminal sends a termination request to the server.

[0564] Step 31:

[0565] The server ends the session and logs out the user.

[0566] Step 32:

[0567] The terminal displays the logout completion screen.

[0568] Example 1

[0569] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Conventional online customer service systems make it difficult for users to experience the same sense of security as face-to-face customer service in a brick-and-mortar store, and they face challenges in quickly and appropriately responding to user needs and requests. Furthermore, there is a lack of effective ways to store and analyze conversation history and logs for future reference or to improve services. Furthermore, when using avatars, the technological means to provide accurate responses to user questions and requests are not yet fully developed.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0572] In this invention, the server includes an account authentication means for allowing a user to access the online platform and perform login authentication, a service selection means for selecting between chat support or avatar video and audio support from the user, a means for generating a chat interface and response resources for exchanging messages in real time, a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests, a means for saving a log of all dialogues and exchanges, and, in the case of avatar video and audio support, a means for using a generative AI model to analyze questions and requests from the user and generate appropriate responses. This allows users to receive reliable face-to-face customer service online just like in a physical store, provides optimal services tailored to the user's needs, and effectively saves and analyzes the history of all dialogues.

[0573] "Account authentication means" refers to the means by which a user accesses the online platform and performs login authentication.

[0574] The "service selection means" is a means for a user to select a service that supports chat or video and audio using an avatar.

[0575] The "chat interface" is an interface that allows users and staff to exchange messages in real time.

[0576] The "means for generating response resources" refers to a means for generating resources (staff or avatars) required for chat response or avatar response.

[0577] The "data analysis and plan generation means" is a means for analyzing data to propose an optimal plan based on the user's needs and requests, and generating a plan.

[0578] "Means for storing logs" refers to a means for storing logs of all conversations and interactions.

[0579] "Means for using a generative AI model" refers to means for using a generative AI model to analyze questions or requests from users and generate appropriate responses in the case of video and audio support using avatars.

[0580] MODE FOR CARRYING OUT THE INVENTION

[0581] As an embodiment of this invention, we provide a system that allows users to receive face-to-face customer service online with the same sense of security as in a physical store. The system is described in detail below. Specific names of the hardware and software used are also specified.

[0582] This system is based on the premise that users will use a web browser to access the service. The server runs on web server software such as Apache or Nginx. A relational database such as MySQL or PostgreSQL is used as the database. The front end that users access is built using web technologies such as HTML, CSS, and JavaScript.

[0583] 1. User Access and Login

[0584] When a user accesses the system's URL using a web browser (e.g., Chrome or Firefox), the server sends the top page HTML, CSS, and JavaScript files to the user's device. The user's device receives this and displays the top page in the browser. The user enters their account information (user ID and password) and clicks the login button. The device sends this information to the server, which checks it against a database for authentication. If authentication is successful, the server generates a session ID and sends the dashboard page to the user's device, which displays it.

[0585] 2. Service Selection

[0586] Users select the option "chat support" or "avatar video and audio support" on the dashboard. The user's device sends the selection to the server, which then allocates the necessary resources (staff or avatars) based on the selection.

[0587] 3. Customer service

[0588] If chat is selected, the server creates a dedicated chat room and sends the chat interface HTML and JavaScript to the user's and staff's devices. The devices display this, allowing for the exchange of text messages in real time. Text messages are exchanged via the server and are all saved as logs.

[0589] If avatar-based video and audio support is selected, the server generates a 3D avatar and prepares the scenario. The virtual support room interface is sent to the user's device, which displays it. The user's questions and requests are sent from the device to the server, which uses a generative AI model (e.g., ChatGPT) to generate appropriate responses and provides them through the avatar. All of these interactions are also logged.

[0590] 4. Confirmation of requests and provision of plans

[0591] The needs and requests communicated by users through chat or avatars are recorded on the server. The server analyzes the recorded data and generates an optimal plan. This plan information is sent from the server to the user's device and displayed on the device.

[0592] 5. Logging and terminating the session

[0593] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[0594] Examples of prompt statements

[0595] As a concrete example, let's assume that the system implements a series of steps in which a user accesses a service site, logs in, selects chat support, and exchanges messages with a staff member in real time. By inputting the following prompt sentences into the generative AI model, we can specifically explain how the system works.

[0596] Example prompt sentence:

[0597] "You have devised a system that allows users to safely receive face-to-face customer service online. Based on the specific scenario below, please write down how a user would use this service. The scenario includes accessing the service from a web browser, logging in, chatting, exchanging messages with a staff member, and receiving a service plan proposal."

[0598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0599] Step 1: User accesses the system's URL in a web browser

[0600] A user accesses the system's URL using a web browser (e.g., Chrome or Firefox). The server generates the HTML, CSS, and JavaScript files for the top page and sends them to the user's device. The device receives these files and displays the top page in the browser.

[0601] Input: User enters a URL

[0602] Output: The HTML, CSS, and JavaScript files for the top page are sent to the terminal, and the top page is displayed in the browser.

[0603] Step 2: User enters account information and logs in

[0604] The user enters their account information (user ID and password) into the input fields on the top page and clicks the login button. The device uses JavaScript to perform basic format checks on the input data and then sends it to the server.

[0605] Input: Enter your account information (user ID, password)

[0606] Output: The input data is sent to the server

[0607] Step 3: The server authenticates the account

[0608] The server checks the received account information against its database. If authentication is successful, the server generates a session ID, dynamically generates a dashboard page with this information, and sends it to the user's device.

[0609] Input: Entered account information

[0610] Data processing and data calculation: Matching with database and generating session ID

[0611] Output: Dashboard page and session ID sent to device

[0612] Step 4: User selects service

[0613] The user selects "chat support" or "avatar video and audio support" from the options on the dashboard, and the device sends the selection information to the server.

[0614] Input: Select service (chat support, avatar support)

[0615] Output: Selection information is sent to the server

[0616] Step 5: Server allocates resources

[0617] Based on the received selection information, the server allocates the necessary resources (staff or avatars) and generates the necessary interfaces, such as chat rooms or virtual interaction rooms.

[0618] Input: Service selection information

[0619] Data processing and data calculation: Allocation of corresponding resources and generation of interfaces

[0620] Output: The appropriate interface is sent to the user's terminal.

[0621] Step 6: Start serving customers via chat

[0622] If the user selects chat support, a chat room is created and a chat interface is sent to the user's and staff's terminals.

[0623] Input: Chat response selection

[0624] Output: The chat room and chat interface are sent to the device.

[0625] Step 7: Exchange chat messages and save logs

[0626] Users and staff can type and send text messages, and the terminals send the messages to the server, which exchanges messages in real time and stores all messages as a log.

[0627] Input: Messages entered by users and staff

[0628] Data processing and data calculation: Message sending and receiving and log storage

[0629] Output: Real-time exchange of messages and storage of logs

[0630] Step 8: Start serving customers with avatar support

[0631] If the user selects avatar-based video and audio support, the server generates an avatar and prepares the scenario. It also generates a virtual support room interface and sends it to the user's device.

[0632] Input: Avatar-enabled selection

[0633] Output: The virtual room and avatar are sent to the user's device.

[0634] Step 9: Avatar response and logging

[0635] When a user types a question or request, the device sends it to the server, which uses a generative AI model to generate a response and delivers it through an avatar. All interactions are logged.

[0636] Input: User questions or requests

[0637] Data processing and data calculation: response generation and logging using generative AI models

[0638] Output: Avatar response and log storage

[0639] Step 10: Confirm your needs and provide a plan

[0640] The needs and requests communicated by the user are recorded on the server. The server analyzes the recorded data and generates an optimal plan. The generated plan information is sent to the user's device and displayed on the device.

[0641] Input: User needs and desires

[0642] Data processing and data calculations: Data analysis and plan generation

[0643] Output: Plan information is sent to the user's device

[0644] Step 11: Save the log and end the session

[0645] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[0646] Input: User request to terminate

[0647] Data processing and data calculation: Session termination processing and logout processing

[0648] Output: The logout completion screen is displayed on the terminal.

[0649] (Application example 1)

[0650] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] With the development of online shopping, users are increasingly seeking the same sense of security and experience as face-to-face customer service in a physical store. However, current online platforms make it difficult for users to receive immediate, personalized assistance, especially when they need detailed information about a product or are unsure about a purchase, resulting in a decline in satisfaction and a decrease in purchasing motivation. Additionally, there is a lack of mechanisms for appropriately logging and storing conversation content and customer needs, and utilizing them to improve services in the future.

[0652] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0653] In this invention, the server includes a means for users to interact with avatars and specialized staff in real time through a shopping experience in a virtual store, an account authentication means, and a service selection means for users to choose between chat support or avatar video and audio support. This allows users to enjoy the same sense of security as face-to-face customer service in a physical store while online, and to receive optimal suggestions based on their individual needs. In addition, by storing a log of all conversations and interactions, this information can be used to improve services in the future.

[0654] "User" means any individual or group of people who accesses and uses the functionality of the system or platform.

[0655] "Online Platform" refers to software or websites provided over the Internet that offer specific services or functions to users.

[0656] "Account authentication means" refers to a mechanism for verifying that a user is a legitimate user when accessing a system or platform.

[0657] "Service selection means" refers to an interface or mechanism that allows a user to select one of multiple service options provided.

[0658] "Chat interface" refers to a screen or function that allows users and support staff to exchange text messages in real time.

[0659] "Supporting resources" refers to the technical elements such as servers, network equipment, and software required for chat support and avatar video and audio support.

[0660] "Data analysis and plan generation means" refers to a mechanism for collecting and analyzing data to propose an optimal service plan based on the user's needs and requests, and for generating a plan based on the results of the analysis.

[0661] "Log storage means" refers to a mechanism for storing records of all conversations and interactions so that they can be referenced at a later date.

[0662] A "virtual store" refers to a virtual store space provided on the Internet, where users can browse and purchase products just as if they were in a physical store.

[0663] An "avatar" is a virtual person or character used to interact with users and has the ability to respond through voice and video.

[0664] "Specialist staff" refers to personnel who have specialized knowledge about products and services and who provide assistance to users.

[0665] This invention is an online system that allows users to shop in a virtual store. This system provides users with a sense of security similar to face-to-face customer service in a real store, and includes a function that allows users to interact with avatars and professional staff in real time.

[0666] System Overview

[0667] The system mainly consists of the following components:

[0668] 1. Account Authentication Methods

[0669] 2. Service Selection Method

[0670] 3. Chat interface and corresponding resource generation means

[0671] 4. Data Analysis and Plan Generation Methods

[0672] 5. Log storage method

[0673] 6. Real-time interaction with avatars and expert staff

[0674] Account authentication method

[0675] It is a means for users to access online platforms and perform login authentication. It uses software such as Flask and SQLAlchemy to perform the user ID and password authentication process. This is the gateway for users to access the system and is a function to maintain security.

[0676] Service Selection Method

[0677] After logging in, users can choose between chat support or avatar video and audio support. This selection information is sent to the server, which allocates appropriate resources based on the user's selection. The user interface is built using HTML, CSS, and JavaScript.

[0678] Chat interface and corresponding resource generation means

[0679] If chat support is selected, the server generates a dedicated chat room and provides an interface for exchanging text messages in real time, using real-time communication technologies such as Flask-SocketIO. This allows users and expert staff to communicate in real time.

[0680] Data Analysis and Plan Generation Tools

[0681] It is a means of proposing optimal service plans based on the user's needs and requests. Here, machine learning models and generative AI models are used to collect and analyze data obtained from users, thereby generating and providing individually optimized plans for the user.

[0682] Log storage method

[0683] It is a way to store all conversations and interactions as logs, which can be referenced later and used to improve the service. The database is SQLite or another RDBMS.

[0684] Real-time interaction with avatars and expert staff

[0685] If the user selects avatar video and audio support, the server generates an avatar and executes a scenario that responds to the user with audio and video. This process utilizes text-to-speech (TTS) and video rendering technologies. The avatar's movements and responses are based on a generative AI model.

[0686] Examples of concrete examples and prompts

[0687] Specific examples

[0688] The user launches the virtual store app and logs in by entering their user ID and password on the login screen. They then enter a chat room and can ask questions about products in real time with a specialist staff member. An avatar will explain the product using video and audio based on the scenario, and if the user is satisfied, they can make a purchase.

[0689] Prompt Sentence Examples

[0690] User launches app and enters login details:

[0691] "Please enter your login information"

[0692] Join the chat room:

[0693] "Joined the room"

[0694] Chat message exchange:

[0695] User: "Please describe this product."

[0696] Staff: "Please wait. This product has the following features..."

[0697] Avatar response:

[0698] "Here are some products we recommend to you..."

[0699] This system allows users to enjoy the same high-quality service online as they would in a physical store, and also enables the storage of logs that will lead to improved service quality.

[0700] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0701] Step 1:

[0702] A user accesses the online platform and enters their user ID and password on the login screen for authentication. The server receives the entered user ID and password information and verifies them against the database. If authentication is successful, the server starts the user's session, generates a dashboard page, and sends it to the user's device.

[0703] Input: User ID, Password

[0704] Data processing: Matching with database, creating sessions

[0705] Output: Dashboard page displayed

[0706] Step 2:

[0707] The user selects the "chat support" or "avatar video and audio support" service on the dashboard. The device sends the user's selection information to the server, which then allocates the appropriate resource (specialist staff or avatar).

[0708] Input: User's service selection information

[0709] Data processing: receiving selection information, allocating resources

[0710] Output: Prepare corresponding resources

[0711] Step 3:

[0712] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The user and the staff exchange text messages in real time, and all messages are sent to the server and saved as a log.

[0713] Input: (Chat selection) User message, staff message

[0714] Data processing: Chat room creation, message exchange, message log storage

[0715] Output: Real-time chat

[0716] Step 4:

[0717] If the user selects video and audio support via an avatar, the server generates an avatar and executes a scenario in which the user responds via video and audio. When the user makes a question or request, the server analyzes it using a generative AI model, generates an appropriate response, and provides it through the avatar.

[0718] Input: (Avatar selection) User question or request

[0719] Data processing: Avatar generation, analysis using generative AI models, response generation

[0720] Output: Avatar video and audio response

[0721] Step 5:

[0722] The server analyzes the data based on the user's needs and requests and generates an optimal service plan. The generated service plan information is sent from the server to the user's terminal and displayed on the user's terminal.

[0723] Input: User needs and desires

[0724] Data processing: Data collection and analysis, service plan generation

[0725] Output: Display service plan information

[0726] Step 6:

[0727] By saving a log of all conversations and interactions, they can be managed in a form that can be referenced later. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs out the user.

[0728] Input: Content of conversation or exchange, end request

[0729] Data processing: saving logs, ending sessions

[0730] Output: Display of logout completion screen

[0731] In this way, a system is realized that allows users to receive real-time support throughout their shopping experience in a virtual store.

[0732] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0733] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a physical store, and that provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[0734] 1. User Access and Login

[0735] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the submit button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[0736] 2. Service Selection

[0737] On the dashboard, the user selects either "chat support" or "avatar support." The device sends the selection information to the server, which then assigns available resources (staff or avatars) according to the user's selection.

[0738] 3. Customer service

[0739] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0740] If the user selects avatar-based video and audio support, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0741] 4. Confirmation of requests and provision of plans

[0742] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The server then analyzes the recorded information to find the optimal plan and generates plan information. The generated plan information is sent from the server to the user's device and displayed on the device.

[0743] 5. Introducing the Emotion Engine

[0744] In this embodiment, an emotion engine is added. The emotion engine analyzes the user's text message, voice, and facial expression to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes emotions from the user's text message and feeds the information back to the server. Based on the feedback information, the server provides a response method to the chat support resource in real time.

[0745] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0746] 6. Logging and terminating sessions

[0747] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0748] As a specific example, a user accesses a service site, logs in, selects chat, and exchanges messages with a staff member in real time. The emotion engine analyzes the user's emotions and provides appropriate feedback to the staff member, enabling them to provide the most appropriate service to the user. There is also a series of steps that provide plan information generated based on the user's needs and requests. This allows the user to receive the most appropriate service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems from occurring.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] A user accesses the system's URL using a web browser.

[0752] Step 2:

[0753] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[0754] Step 3:

[0755] The device displays the top page in the browser.

[0756] Step 4:

[0757] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[0758] Step 5:

[0759] The terminal sends the entered account information to the server as a POST request.

[0760] Step 6:

[0761] The server checks the received account information against user information in its database.

[0762] Step 7:

[0763] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[0764] Step 8:

[0765] The device displays the dashboard page in a browser.

[0766] Step 9:

[0767] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[0768] Step 10:

[0769] The terminal sends the selected information to the server as a POST request.

[0770] Step 11:

[0771] The server allocates available resources (staff or avatars) based on the user's selection.

[0772] Step 12:

[0773] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[0774] Step 13:

[0775] The terminal will display the chat interface.

[0776] Step 14:

[0777] The user types a message in the chat interface and clicks the send button.

[0778] Step 15:

[0779] The device sends a chat message to the server.

[0780] Step 16:

[0781] The server delivers the received message to the staff member in charge, and the staff member's reply is also delivered to the user. In addition, an emotion engine within the chat interface analyzes the user's message in real time and feeds the emotional information back to the server.

[0782] Step 17:

[0783] Based on the emotional feedback information, the server provides the appropriate response method to the staff member in charge, which is reflected on the chat interface.

[0784] Step 18:

[0785] The server stores all messages and sentiment analysis logs.

[0786] Step 19:

[0787] If the server supports avatars, it generates avatars and prepares the scenario.

[0788] Step 20:

[0789] The server transmits the interface of the virtual correspondence room to the user's terminal.

[0790] Step 21:

[0791] The device displays the virtual response room interface.

[0792] Step 22:

[0793] The server executes a scenario in which the avatar responds to the user with voice and video. The emotion engine within the avatar interface analyzes the user's voice and facial expressions in real time and feeds the emotion information back to the server.

[0794] Step 23:

[0795] The server adjusts the avatar's response method based on the emotional feedback information and provides it to the user.

[0796] Step 24:

[0797] Users communicate their needs and desires through chat or interaction with an avatar.

[0798] Step 25:

[0799] The server records and stores the user's requests and needs in a database.

[0800] Step 26:

[0801] The server analyzes the optimum plan based on the recorded information and generates plan information.

[0802] Step 27:

[0803] The server transmits the generated plan information to the user's terminal.

[0804] Step 28:

[0805] The device will display the plan information in a browser.

[0806] Step 29:

[0807] The server stores a log of all conversations and interactions in a database.

[0808] Step 30:

[0809] The user clicks the "Exit" button to end the session.

[0810] Step 31:

[0811] The terminal sends a termination request to the server.

[0812] Step 32:

[0813] The server ends the session and logs out the user.

[0814] Step 33:

[0815] The terminal displays the logout completion screen.

[0816] Example 2

[0817] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] Modern online services face the challenge of making it difficult for users to obtain an experience equivalent to that of face-to-face customer service in a brick-and-mortar store. Furthermore, there is a need for systems that can accurately understand the user's emotional state and provide services based on that understanding, but conventional systems are inadequate in this regard. Furthermore, there is a need for systems that can log all conversations and interactions to prevent problems before they occur.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0820] In this invention, the server includes an account authentication means for users to access the online platform and perform login authentication; a service selection means for users to select between chat support or avatar video and audio support; a means for generating a chat interface and response resources for exchanging messages in real time; a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; a means for saving logs of all conversations and interactions; a means including an emotion engine that analyzes the user's text messages, voice, and facial expressions to recognize their emotional state; and a means for providing a response method based on feedback information from the emotion engine. This allows for an online experience equivalent to face-to-face customer service in a physical store, enabling advanced responses based on the user's emotional state. Furthermore, by saving all conversations and interactions as logs, problems can be prevented before they occur.

[0821] "Account authentication means" refers to the means by which a user accesses an online platform, collates the authentication information (such as a user ID and password) required to log in, and verifies its legitimacy.

[0822] The "service selection means" is a means for transmitting information about whether a user selects chat support or video and audio support using an avatar to the server.

[0823] A "chat interface" is a user interface that allows users and staff to exchange text messages in real time.

[0824] The "means for generating corresponding resources" is a means for allocating and generating resources such as appropriate staff members and avatars based on the user's selection.

[0825] The "data analysis and plan generation means" is a means for analyzing data based on the user's needs and requests, and for designing and proposing the most suitable plan.

[0826] A "means for storing logs" is a means for recording all interactions and exchanges between a user and a system and storing them in a form that can be referenced at a later date.

[0827] An "emotion engine" is a technology that analyzes a user's text messages, voice, and facial expressions to recognize the user's emotional state in real time.

[0828] The "means for providing a response method based on feedback information from the emotion engine" is a means for appropriately adjusting the response method of the staff member or avatar based on the emotional state of the user analyzed by the emotion engine.

[0829] The present invention relates to a system that enables users to receive face-to-face customer service online in the same way as visiting a physical store, and provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[0830] This system provides an account authentication method for users to access the online platform and perform login authentication. When a user accesses the system using a web browser, the server sends the HTML, CSS, and JavaScript files for the top page to the user's device, which then displays the top page. When the user enters their user ID and password and clicks the submit button, the device sends this information to the server, which then checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, which the device displays.

[0831] Next, the user selects "chat support" or "avatar support" on the dashboard. The terminal sends this selection information to the server, and the server assigns available resources (staff or avatar) according to the user's selection. For example, if the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0832] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0833] Furthermore, this system has an added emotion engine. The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. For example, in the case of chat support, the emotion engine analyzes the user's emotions from the text messages and feeds the information back to the server. The server provides a response method to the chat support resource in real time based on the feedback information. In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds the information back to the server. The server adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0834] For example, if a user sends a chat message saying, "Please tell me how to use this product," the system will analyze the message and provide appropriate information. If the emotion engine detects anxiety in a message such as, "I'm in a lot of trouble," the system will pass that information on to the appropriate staff member, who will then respond promptly and courteously to the user.

[0835] In this way, this system aims to provide an online experience similar to face-to-face customer service in a physical store. Users can receive optimal service suggestions without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[0836] Examples of prompts include:

[0837] "If the message entered by the user is 'Please tell me how to use the recommended product,' generate an appropriate product description."

[0838] "If the emotion engine interprets the user's facial expression as serious, be a little more careful with the avatar's response tone."

[0839] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0840] Step 1:

[0841] The user opens a web browser and accesses the system's URL. The server sends the top page HTML, CSS, and JavaScript files to the user's device. The device receives these files and displays the top page. The user's input is the system's URL, and the output is the display of the top page. Specifically, the user enters the URL in the browser's address bar and presses the Enter key.

[0842] Step 2:

[0843] The user enters a user ID and password into the login form and clicks the submit button. The terminal sends this input information to the server. The server compares the received information with a database and performs authentication. If authentication is successful, the server starts a session for the user, generates a dashboard page, and sends it to the user's terminal. The terminal displays the dashboard page. The input is the user ID and password, and the output is the display of the dashboard page. The specific action is when the user clicks the "Login" button.

[0844] Step 3:

[0845] The user selects "Chat support" or "Avatar support" on the dashboard. The device sends the selection information to the server. The server allocates available resources (staff or avatars) based on the user's selection. The input is the selection information, and the output is the allocation of resources. The specific operation is when the user clicks the "Chat support" or "Avatar support" button.

[0846] Step 4:

[0847] (For chat support) The server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages and saves them as a log. The input is the chat message, and the output is the message saved as a log. The specific action is when the user clicks the "Send message" button.

[0848] Step 5:

[0849] (In the case of avatar support) The server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. When the user asks a question or makes a request, the server analyzes it and generates a response according to the avatar's scenario. The input is the user's question or request, and the output is the avatar's response. In concrete terms, the user speaks into the microphone.

[0850] Step 6:

[0851] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The optimal plan is analyzed and generated based on this data. The server then sends the generated plan information to the user's device, which displays it. The input is the user's needs and desires, and the output is a display of the optimal plan. A specific operation is when the user clicks the "Confirm Plan" button.

[0852] Step 7:

[0853] The system is equipped with an emotion engine that analyzes the user's text messages, voice, and facial expressions. The emotion engine recognizes the user's emotional state in real time and feeds that information back to the server. The server uses the feedback information to adjust the chat response and avatar's response method. The input is the user's message and facial expression, and the output is feedback on how to respond. Specifically, the emotion engine recognizes the user's facial expression.

[0854] Step 8:

[0855] When all dialogue and exchanges are complete, the user clicks the "Exit" button, and the terminal sends an end request to the server. The server ends the session and logs the user out. The terminal displays a logout completion screen. The input is the end request, and the output is a message indicating that the logout has been completed. The specific action is when the user clicks the "Exit" button.

[0856] (Application example 2)

[0857] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0858] Conventional online shopping systems struggle to provide the same sense of security and face-to-face customer service as brick-and-mortar stores, and lack the means to alleviate users' dissatisfaction and anxiety. They also lack the ability to understand users' emotional state and respond optimally accordingly. As a result, the user experience remains stagnant, leading to problems with declining customer satisfaction.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an account authentication means for a user to access the online platform and perform login authentication, a service selection means for the user to select from chat support or video and audio support using avatars, a means for generating a chat interface and corresponding resources for exchanging messages in real time, an emotion analysis means for analyzing emotions from the user's text messages and providing feedback, and a means for adjusting the chat support resources and avatar responses based on the emotion analysis results. This allows users to feel as secure online as if they were in a physical store, and receive optimal service in real time according to their emotions.

[0860] "Account authentication means" refers to the function by which a user accesses the online platform and enters an appropriate ID and password for authentication.

[0861] The "service selection means" is a function that provides an interface for the user to select either chat support or video and audio support using an avatar.

[0862] A "chat interface" is an interface that allows users and staff to exchange text messages in real time.

[0863] "Response resources" are staff or system resources assigned to provide services to users.

[0864] The "data analysis means" is a function that performs data analysis to propose the optimal plan based on the user's needs and requests.

[0865] The "plan generation means" is a function that generates an optimal proposed plan for the user based on the results of data analysis.

[0866] The "log storage means" is a function that stores the history of all conversations and interactions and makes them available for later reference as needed.

[0867] The "emotion analysis means" is a function that analyzes emotions from the user's text message and provides feedback on the results.

[0868] The "response adjustment means" is a function that adjusts the responses of chat resources and avatars based on the results of emotion analysis to provide an appropriate response.

[0869] A "chat room" is a dedicated virtual space where users and staff can exchange text messages in real time.

[0870] The "avatar generation means" is a function that generates an avatar and executes a scenario that responds to the user with audio and video.

[0871] The "scenario execution means" is a function that enables an avatar to respond to the user with voice and video based on a predefined response scenario.

[0872] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a real store, and also provides advanced services using an emotion engine. The system of the present invention operates as follows.

[0873] First, a user accesses the system's URL using a web browser and performs login authentication through the web browser interface. When the user enters their account information (user ID and password) and sends it to the server, the server checks it against a database, and if authentication is successful, the user's session begins. This authentication method prevents unauthorized access and ensures a secure connection.

[0874] After logging in, the user selects "chat support" or "avatar support" on the dashboard. The service selection means allows the user to select a chat interface for exchanging messages in real time as needed. Alternatively, the user can select voice and video responses from an avatar. The service selection means transmits the user's selection information to the server, and the server allocates available resources.

[0875] If the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the user's terminal and the terminal of the staff member in charge for display. Through this interface, the user and staff member exchange messages in real time, and the server saves all messages as a log. This log saving means can be used to prevent problems from occurring or as reference information later.

[0876] If the user selects video and audio support using an avatar, the server generates an avatar and executes a scenario that responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar. The avatar generation means and scenario execution means make it possible to provide a high level of interaction to the user.

[0877] Furthermore, in this embodiment, an emotion engine is added. The emotion analysis means analyzes emotions from the user's text message, voice, and facial expression. In the case of chat correspondence, the emotion engine analyzes the user's text message and feeds back the information to the server. The response adjustment means allows the server to provide an appropriate response method to the chat correspondence resource in real time based on the feedback information.

[0878] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions, sends the feedback information to the server, and the server adjusts the avatar's response method, thereby enabling the optimal response to the user's emotions.

[0879] As a specific example, if a user types into the chat interface, "I'm looking for a new smartphone. Which one do you recommend?", the emotion engine will analyze the emotion of the user's message, and based on that information, a staff member will respond by saying, "We will suggest the best options for you."

[0880] Examples of prompt sentences that may be used in the present invention include the following:

[0881] Prompt: Parse the user's message to recognize their sentiment and generate an appropriate response. Message: "I'm looking for a new smartphone. Which one would you recommend?"

[0882] The hardware and software used are as follows:

[0883] Web browser: displaying the user interface

[0884] Server: Provides API endpoints, manages sessions, operates the database, and runs the emotion engine

[0885] EmotionRecognition library: Analyzes user text messages and recognizes emotions in real time

[0886] Through these measures, users can feel as secure online as if they were in a physical store, and can receive optimal service tailored to their emotions in real time.

[0887] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0888] Step 1:

[0889] A user accesses the system's URL using a web browser. The terminal receives the top page HTML, CSS, and JavaScript files from the server and displays the top page, allowing the user to access the login screen.

[0890] Input: System URL

[0891] Output: Login screen displayed in web browser

[0892] What happens: A user enters a URL into the browser's address bar and loads a page.

[0893] Step 2:

[0894] The user enters account information (user ID and password) and clicks the submit button. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, the server starts the user's session, generates a dashboard page, sends it to the terminal, and the terminal displays it.

[0895] Input: User ID, Password

[0896] Output: Dashboard page upon successful authentication

[0897] Specific operation: The user enters their ID and password in the login form and presses the "Login" button.

[0898] Step 3:

[0899] The user selects either "chat support" or "avatar support" on the dashboard. The terminal sends the selection information to the server, and the server assigns available resources (staff or avatars).

[0900] Input: User selection information

[0901] Output: Resource allocation according to the selection

[0902] What happens: A user clicks on an option on a dashboard.

[0903] Step 4:

[0904] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[0905] Input: User's chat response selection, text message

[0906] Output: Chat interface, real-time message exchange, logs

[0907] Specific operation: The server creates a chat room, and users and staff members can input, send, and receive chat messages.

[0908] Step 5:

[0909] If the user selects avatar support, the server generates an avatar and sends the virtual support room interface to the user's device. The server analyzes the user's questions and requests, generates appropriate responses, and provides them through the avatar.

[0910] Input: User avatar-enabled selection, user question or request

[0911] Output: Virtual interaction room, avatar responses

[0912] Specific operation: The server generates an avatar and executes a scenario to respond to the user's questions.

[0913] Step 6:

[0914] The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes the user's text messages and feeds the information back to the server. Based on the feedback information, the server provides chat support resources with appropriate responses in real time.

[0915] Input: User's text message, voice, and facial expression information

[0916] Output: Sentiment analysis results, feedback information, and appropriate response methods

[0917] Specific behavior: The emotion engine analyzes messages and voice and provides appropriate feedback instantly.

[0918] Step 7:

[0919] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[0920] Input: User's voice and facial expression

[0921] Output: Sentiment analysis results, feedback information, appropriate avatar responses

[0922] Specific behavior: The emotion engine analyzes the user's voice and facial expressions and adjusts the avatar's response.

[0923] Step 8:

[0924] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0925] Input: User's exit request

[0926] Output: Interaction log, session termination, logout completion screen

[0927] Specific operation: The user presses the end button, and the server performs the session termination process.

[0928] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0929] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0930] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0931] [Third embodiment]

[0932] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0933] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0934] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0935] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0936] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0937] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0938] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0939] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0940] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0941] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0942] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0943] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0944] As an embodiment of the present invention, a system is provided that allows users to receive face-to-face customer service online with the same feeling as visiting a real store. Specific processing and operation of the system are described in detail below.

[0945] 1. User Access and Login

[0946] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the login button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[0947] 2. Service Selection

[0948] On the dashboard, the user selects either "chat support" or "avatar video and audio support." The device sends this selection information to the server, which then assigns available resources (staff or avatars) based on the user's selection.

[0949] 3. Customer service

[0950] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. These messages are sent to the server and exchanged in real time, and all messages are stored on the server as a log.

[0951] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[0952] 4. Confirmation of requests and provision of plans

[0953] When a user communicates their needs and desires through chat or interaction with an avatar, the server records this and stores it in a database. The server then analyzes the recorded information to determine the optimal plan and generates plan information. The generated plan information is then sent from the server to the user's device and displayed on the device.

[0954] 5. Logging and terminating the session

[0955] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[0956] As a concrete example, a user accesses a service site, logs in, selects chat options, exchanges messages with a staff member in real time, receives a service plan proposal, and finally ends the session. This allows the user to receive the most suitable service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[0957] The processing flow will be explained below.

[0958] Step 1:

[0959] A user accesses the system's URL using a web browser.

[0960] Step 2:

[0961] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[0962] Step 3:

[0963] The device displays the top page in the browser.

[0964] Step 4:

[0965] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[0966] Step 5:

[0967] The terminal sends the entered account information to the server as a POST request.

[0968] Step 6:

[0969] The server checks the received account information against user information in its database.

[0970] Step 7:

[0971] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[0972] Step 8:

[0973] The device displays the dashboard page in a browser.

[0974] Step 9:

[0975] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[0976] Step 10:

[0977] The terminal sends the selected information to the server as a POST request.

[0978] Step 11:

[0979] The server allocates available resources (staff or avatars) based on the user's selection.

[0980] Step 12:

[0981] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[0982] Step 13:

[0983] The terminal will display the chat interface.

[0984] Step 14:

[0985] The user types a message in the chat interface and clicks the send button.

[0986] Step 15:

[0987] The device sends a chat message to the server.

[0988] Step 16:

[0989] The server delivers the received message to the staff member in charge, and similarly delivers the staff member's reply to the user.

[0990] Step 17:

[0991] The server stores all message logs.

[0992] Step 18:

[0993] If the server supports avatars, it generates avatars and prepares the scenario.

[0994] Step 19:

[0995] The server transmits the interface of the virtual correspondence room to the user's terminal.

[0996] Step 20:

[0997] The device displays the virtual response room interface.

[0998] Step 21:

[0999] The server executes the scenario in which the avatar responds to the user with voice and video.

[1000] Step 22:

[1001] The server generates appropriate responses to the user's questions and requests and provides them through an avatar.

[1002] Step 23:

[1003] Users communicate their needs and desires through chat or interaction with an avatar.

[1004] Step 24:

[1005] The server stores the requests received from the user in a database.

[1006] Step 25:

[1007] The server analyzes and generates the information for the best plan based on the user's needs.

[1008] Step 26:

[1009] The server transmits the generated plan information to the user's terminal.

[1010] Step 27:

[1011] The device will display the plan information on the screen.

[1012] Step 28:

[1013] The server stores a log of all conversations and interactions in a database.

[1014] Step 29:

[1015] The user clicks the "Exit" button to end the session.

[1016] Step 30:

[1017] The terminal sends a termination request to the server.

[1018] Step 31:

[1019] The server ends the session and logs out the user.

[1020] Step 32:

[1021] The terminal displays the logout completion screen.

[1022] Example 1

[1023] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] Conventional online customer service systems make it difficult for users to experience the same sense of security as face-to-face customer service in a brick-and-mortar store, and they face challenges in quickly and appropriately responding to user needs and requests. Furthermore, there is a lack of effective ways to store and analyze conversation history and logs for future reference or to improve services. Furthermore, when using avatars, the technological means to provide accurate responses to user questions and requests are not yet fully developed.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1026] In this invention, the server includes an account authentication means for allowing a user to access the online platform and perform login authentication, a service selection means for selecting between chat support or avatar video and audio support from the user, a means for generating a chat interface and response resources for exchanging messages in real time, a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests, a means for saving a log of all dialogues and exchanges, and, in the case of avatar video and audio support, a means for using a generative AI model to analyze questions and requests from the user and generate appropriate responses. This allows users to receive reliable face-to-face customer service online just like in a physical store, provides optimal services tailored to the user's needs, and effectively saves and analyzes the history of all dialogues.

[1027] "Account authentication means" refers to the means by which a user accesses the online platform and performs login authentication.

[1028] The "service selection means" is a means for a user to select a service that supports chat or video and audio using an avatar.

[1029] The "chat interface" is an interface that allows users and staff to exchange messages in real time.

[1030] The "means for generating response resources" refers to a means for generating resources (staff or avatars) required for chat response or avatar response.

[1031] The "data analysis and plan generation means" is a means for analyzing data to propose an optimal plan based on the user's needs and requests, and generating a plan.

[1032] "Means for storing logs" refers to a means for storing logs of all conversations and interactions.

[1033] "Means for using a generative AI model" refers to means for using a generative AI model to analyze questions or requests from users and generate appropriate responses in the case of video and audio support using avatars.

[1034] MODE FOR CARRYING OUT THE INVENTION

[1035] As an embodiment of this invention, we provide a system that allows users to receive face-to-face customer service online with the same sense of security as in a physical store. The system is described in detail below. Specific names of the hardware and software used are also specified.

[1036] This system is based on the premise that users will use a web browser to access the service. The server runs on web server software such as Apache or Nginx. A relational database such as MySQL or PostgreSQL is used as the database. The front end that users access is built using web technologies such as HTML, CSS, and JavaScript.

[1037] 1. User Access and Login

[1038] When a user accesses the system's URL using a web browser (e.g., Chrome or Firefox), the server sends the top page HTML, CSS, and JavaScript files to the user's device. The user's device receives this and displays the top page in the browser. The user enters their account information (user ID and password) and clicks the login button. The device sends this information to the server, which checks it against a database for authentication. If authentication is successful, the server generates a session ID and sends the dashboard page to the user's device, which displays it.

[1039] 2. Service Selection

[1040] Users select the option "chat support" or "avatar video and audio support" on the dashboard. The user's device sends the selection to the server, which then allocates the necessary resources (staff or avatars) based on the selection.

[1041] 3. Customer service

[1042] If chat is selected, the server creates a dedicated chat room and sends the chat interface HTML and JavaScript to the user's and staff's devices. The devices display this, allowing for the exchange of text messages in real time. Text messages are exchanged via the server and are all saved as logs.

[1043] If avatar-based video and audio support is selected, the server generates a 3D avatar and prepares the scenario. The virtual support room interface is sent to the user's device, which displays it. The user's questions and requests are sent from the device to the server, which uses a generative AI model (e.g., ChatGPT) to generate appropriate responses and provides them through the avatar. All of these interactions are also logged.

[1044] 4. Confirmation of requests and provision of plans

[1045] The needs and requests communicated by users through chat or avatars are recorded on the server. The server analyzes the recorded data and generates an optimal plan. This plan information is sent from the server to the user's device and displayed on the device.

[1046] 5. Logging and terminating the session

[1047] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[1048] Examples of prompt statements

[1049] As a concrete example, let's assume that the system implements a series of steps in which a user accesses a service site, logs in, selects chat support, and exchanges messages with a staff member in real time. By inputting the following prompt sentences into the generative AI model, we can specifically explain how the system works.

[1050] Example prompt sentence:

[1051] "You have devised a system that allows users to safely receive face-to-face customer service online. Based on the specific scenario below, please write down how a user would use this service. The scenario includes accessing the service from a web browser, logging in, chatting, exchanging messages with a staff member, and receiving a service plan proposal."

[1052] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1053] Step 1: User accesses the system's URL in a web browser

[1054] A user accesses the system's URL using a web browser (e.g., Chrome or Firefox). The server generates the HTML, CSS, and JavaScript files for the top page and sends them to the user's device. The device receives these files and displays the top page in the browser.

[1055] Input: User enters a URL

[1056] Output: The HTML, CSS, and JavaScript files for the top page are sent to the terminal, and the top page is displayed in the browser.

[1057] Step 2: User enters account information and logs in

[1058] The user enters their account information (user ID and password) into the input fields on the top page and clicks the login button. The device uses JavaScript to perform basic format checks on the input data and then sends it to the server.

[1059] Input: Enter your account information (user ID, password)

[1060] Output: The input data is sent to the server

[1061] Step 3: The server authenticates the account

[1062] The server checks the received account information against its database. If authentication is successful, the server generates a session ID, dynamically generates a dashboard page with this information, and sends it to the user's device.

[1063] Input: Entered account information

[1064] Data processing and data calculation: Matching with database and generating session ID

[1065] Output: Dashboard page and session ID sent to device

[1066] Step 4: User selects service

[1067] The user selects "chat support" or "avatar video and audio support" from the options on the dashboard, and the device sends the selection information to the server.

[1068] Input: Select service (chat support, avatar support)

[1069] Output: Selection information is sent to the server

[1070] Step 5: Server allocates resources

[1071] Based on the received selection information, the server allocates the necessary resources (staff or avatars) and generates the necessary interfaces, such as chat rooms or virtual interaction rooms.

[1072] Input: Service selection information

[1073] Data processing and data calculation: Allocation of corresponding resources and generation of interfaces

[1074] Output: The appropriate interface is sent to the user's terminal.

[1075] Step 6: Start serving customers via chat

[1076] If the user selects chat support, a chat room is created and a chat interface is sent to the user's and staff's terminals.

[1077] Input: Chat response selection

[1078] Output: The chat room and chat interface are sent to the device.

[1079] Step 7: Exchange chat messages and save logs

[1080] Users and staff can type and send text messages, and the terminals send the messages to the server, which exchanges messages in real time and stores all messages as a log.

[1081] Input: Messages entered by users and staff

[1082] Data processing and data calculation: Message sending and receiving and log storage

[1083] Output: Real-time exchange of messages and storage of logs

[1084] Step 8: Start serving customers with avatar support

[1085] If the user selects avatar-based video and audio support, the server generates an avatar and prepares the scenario. It also generates a virtual support room interface and sends it to the user's device.

[1086] Input: Avatar-enabled selection

[1087] Output: The virtual room and avatar are sent to the user's device.

[1088] Step 9: Avatar response and logging

[1089] When a user types a question or request, the device sends it to the server, which uses a generative AI model to generate a response and delivers it through an avatar. All interactions are logged.

[1090] Input: User questions or requests

[1091] Data processing and data calculation: response generation and logging using generative AI models

[1092] Output: Avatar response and log storage

[1093] Step 10: Confirm your needs and provide a plan

[1094] The needs and requests communicated by the user are recorded on the server. The server analyzes the recorded data and generates an optimal plan. The generated plan information is sent to the user's device and displayed on the device.

[1095] Input: User needs and desires

[1096] Data processing and data calculations: Data analysis and plan generation

[1097] Output: Plan information is sent to the user's device

[1098] Step 11: Save the log and end the session

[1099] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[1100] Input: User request to terminate

[1101] Data processing and data calculation: Session termination processing and logout processing

[1102] Output: The logout completion screen is displayed on the terminal.

[1103] (Application example 1)

[1104] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1105] With the development of online shopping, users are increasingly seeking the same sense of security and experience as face-to-face customer service in a physical store. However, current online platforms make it difficult for users to receive immediate, personalized assistance, especially when they need detailed information about a product or are unsure about a purchase, resulting in a decline in satisfaction and a decrease in purchasing motivation. Additionally, there is a lack of mechanisms for appropriately logging and storing conversation content and customer needs, and utilizing them to improve services in the future.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1107] In this invention, the server includes a means for users to interact with avatars and specialized staff in real time through a shopping experience in a virtual store, an account authentication means, and a service selection means for users to choose between chat support or avatar video and audio support. This allows users to enjoy the same sense of security as face-to-face customer service in a physical store while online, and to receive optimal suggestions based on their individual needs. In addition, by storing a log of all conversations and interactions, this information can be used to improve services in the future.

[1108] "User" means any individual or group of people who accesses and uses the functionality of the system or platform.

[1109] "Online Platform" refers to software or websites provided over the Internet that offer specific services or functions to users.

[1110] "Account authentication means" refers to a mechanism for verifying that a user is a legitimate user when accessing a system or platform.

[1111] "Service selection means" refers to an interface or mechanism that allows a user to select one of multiple service options provided.

[1112] "Chat interface" refers to a screen or function that allows users and support staff to exchange text messages in real time.

[1113] "Supporting resources" refers to the technical elements such as servers, network equipment, and software required for chat support and avatar video and audio support.

[1114] "Data analysis and plan generation means" refers to a mechanism for collecting and analyzing data to propose an optimal service plan based on the user's needs and requests, and for generating a plan based on the results of the analysis.

[1115] "Log storage means" refers to a mechanism for storing records of all conversations and interactions so that they can be referenced at a later date.

[1116] A "virtual store" refers to a virtual store space provided on the Internet, where users can browse and purchase products just as if they were in a physical store.

[1117] An "avatar" is a virtual person or character used to interact with users and has the ability to respond through voice and video.

[1118] "Specialist staff" refers to personnel who have specialized knowledge about products and services and who provide assistance to users.

[1119] This invention is an online system that allows users to shop in a virtual store. This system provides users with a sense of security similar to face-to-face customer service in a real store, and includes a function that allows users to interact with avatars and professional staff in real time.

[1120] System Overview

[1121] The system mainly consists of the following components:

[1122] 1. Account Authentication Methods

[1123] 2. Service Selection Method

[1124] 3. Chat interface and corresponding resource generation means

[1125] 4. Data Analysis and Plan Generation Methods

[1126] 5. Log storage method

[1127] 6. Real-time interaction with avatars and expert staff

[1128] Account authentication method

[1129] It is a means for users to access online platforms and perform login authentication. It uses software such as Flask and SQLAlchemy to perform the user ID and password authentication process. This is the gateway for users to access the system and is a function to maintain security.

[1130] Service Selection Method

[1131] After logging in, users can choose between chat support or avatar video and audio support. This selection information is sent to the server, which allocates appropriate resources based on the user's selection. The user interface is built using HTML, CSS, and JavaScript.

[1132] Chat interface and corresponding resource generation means

[1133] If chat support is selected, the server generates a dedicated chat room and provides an interface for exchanging text messages in real time, using real-time communication technologies such as Flask-SocketIO. This allows users and expert staff to communicate in real time.

[1134] Data Analysis and Plan Generation Tools

[1135] It is a means of proposing optimal service plans based on the user's needs and requests. Here, machine learning models and generative AI models are used to collect and analyze data obtained from users, thereby generating and providing individually optimized plans for the user.

[1136] Log storage method

[1137] It is a way to store all conversations and interactions as logs, which can be referenced later and used to improve the service. The database is SQLite or another RDBMS.

[1138] Real-time interaction with avatars and expert staff

[1139] If the user selects avatar video and audio support, the server generates an avatar and executes a scenario that responds to the user with audio and video. This process utilizes text-to-speech (TTS) and video rendering technologies. The avatar's movements and responses are based on a generative AI model.

[1140] Examples of concrete examples and prompts

[1141] Specific examples

[1142] The user launches the virtual store app and logs in by entering their user ID and password on the login screen. They then enter a chat room and can ask questions about products in real time with a specialist staff member. An avatar will explain the product using video and audio based on the scenario, and if the user is satisfied, they can make a purchase.

[1143] Prompt Sentence Examples

[1144] User launches app and enters login details:

[1145] "Please enter your login information"

[1146] Join the chat room:

[1147] "Joined the room"

[1148] Chat message exchange:

[1149] User: "Please describe this product."

[1150] Staff: "Please wait. This product has the following features..."

[1151] Avatar response:

[1152] "Here are some products we recommend to you..."

[1153] This system allows users to enjoy the same high-quality service online as they would in a physical store, and also enables the storage of logs that will lead to improved service quality.

[1154] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1155] Step 1:

[1156] A user accesses the online platform and enters their user ID and password on the login screen for authentication. The server receives the entered user ID and password information and verifies them against the database. If authentication is successful, the server starts the user's session, generates a dashboard page, and sends it to the user's device.

[1157] Input: User ID, Password

[1158] Data processing: Matching with database, creating sessions

[1159] Output: Dashboard page displayed

[1160] Step 2:

[1161] The user selects the "chat support" or "avatar video and audio support" service on the dashboard. The device sends the user's selection information to the server, which then allocates the appropriate resource (specialist staff or avatar).

[1162] Input: User's service selection information

[1163] Data processing: receiving selection information, allocating resources

[1164] Output: Prepare corresponding resources

[1165] Step 3:

[1166] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The user and the staff exchange text messages in real time, and all messages are sent to the server and saved as a log.

[1167] Input: (Chat selection) User message, staff message

[1168] Data processing: Chat room creation, message exchange, message log storage

[1169] Output: Real-time chat

[1170] Step 4:

[1171] If the user selects video and audio support via an avatar, the server generates an avatar and executes a scenario in which the user responds via video and audio. When the user makes a question or request, the server analyzes it using a generative AI model, generates an appropriate response, and provides it through the avatar.

[1172] Input: (Avatar selection) User question or request

[1173] Data processing: Avatar generation, analysis using generative AI models, response generation

[1174] Output: Avatar video and audio response

[1175] Step 5:

[1176] The server analyzes the data based on the user's needs and requests and generates an optimal service plan. The generated service plan information is sent from the server to the user's terminal and displayed on the user's terminal.

[1177] Input: User needs and desires

[1178] Data processing: Data collection and analysis, service plan generation

[1179] Output: Display service plan information

[1180] Step 6:

[1181] By saving a log of all conversations and interactions, they can be managed in a form that can be referenced later. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs out the user.

[1182] Input: Content of conversation or exchange, end request

[1183] Data processing: saving logs, ending sessions

[1184] Output: Display of logout completion screen

[1185] In this way, a system is realized that allows users to receive real-time support throughout their shopping experience in a virtual store.

[1186] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1187] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a physical store, and that provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[1188] 1. User Access and Login

[1189] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the submit button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[1190] 2. Service Selection

[1191] On the dashboard, the user selects either "chat support" or "avatar support." The device sends the selection information to the server, which then assigns available resources (staff or avatars) according to the user's selection.

[1192] 3. Customer service

[1193] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1194] If the user selects avatar-based video and audio support, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[1195] 4. Confirmation of requests and provision of plans

[1196] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The server then analyzes the recorded information to find the optimal plan and generates plan information. The generated plan information is sent from the server to the user's device and displayed on the device.

[1197] 5. Introducing the Emotion Engine

[1198] In this embodiment, an emotion engine is added. The emotion engine analyzes the user's text message, voice, and facial expression to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes emotions from the user's text message and feeds the information back to the server. Based on the feedback information, the server provides a response method to the chat support resource in real time.

[1199] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1200] 6. Logging and terminating sessions

[1201] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[1202] As a specific example, a user accesses a service site, logs in, selects chat, and exchanges messages with a staff member in real time. The emotion engine analyzes the user's emotions and provides appropriate feedback to the staff member, enabling them to provide the most appropriate service to the user. There is also a series of steps that provide plan information generated based on the user's needs and requests. This allows the user to receive the most appropriate service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems from occurring.

[1203] The processing flow will be explained below.

[1204] Step 1:

[1205] A user accesses the system's URL using a web browser.

[1206] Step 2:

[1207] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[1208] Step 3:

[1209] The device displays the top page in the browser.

[1210] Step 4:

[1211] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[1212] Step 5:

[1213] The terminal sends the entered account information to the server as a POST request.

[1214] Step 6:

[1215] The server checks the received account information against user information in its database.

[1216] Step 7:

[1217] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[1218] Step 8:

[1219] The device displays the dashboard page in a browser.

[1220] Step 9:

[1221] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[1222] Step 10:

[1223] The terminal sends the selected information to the server as a POST request.

[1224] Step 11:

[1225] The server allocates available resources (staff or avatars) based on the user's selection.

[1226] Step 12:

[1227] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[1228] Step 13:

[1229] The terminal will display the chat interface.

[1230] Step 14:

[1231] The user types a message in the chat interface and clicks the send button.

[1232] Step 15:

[1233] The device sends a chat message to the server.

[1234] Step 16:

[1235] The server delivers the received message to the staff member in charge, and the staff member's reply is also delivered to the user. In addition, an emotion engine within the chat interface analyzes the user's message in real time and feeds the emotional information back to the server.

[1236] Step 17:

[1237] Based on the emotional feedback information, the server provides the appropriate response method to the staff member in charge, which is reflected on the chat interface.

[1238] Step 18:

[1239] The server stores all messages and sentiment analysis logs.

[1240] Step 19:

[1241] If the server supports avatars, it generates avatars and prepares the scenario.

[1242] Step 20:

[1243] The server transmits the interface of the virtual correspondence room to the user's terminal.

[1244] Step 21:

[1245] The device displays the virtual response room interface.

[1246] Step 22:

[1247] The server executes a scenario in which the avatar responds to the user with voice and video. The emotion engine within the avatar interface analyzes the user's voice and facial expressions in real time and feeds the emotion information back to the server.

[1248] Step 23:

[1249] The server adjusts the avatar's response method based on the emotional feedback information and provides it to the user.

[1250] Step 24:

[1251] Users communicate their needs and desires through chat or interaction with an avatar.

[1252] Step 25:

[1253] The server records and stores the user's requests and needs in a database.

[1254] Step 26:

[1255] The server analyzes the optimum plan based on the recorded information and generates plan information.

[1256] Step 27:

[1257] The server transmits the generated plan information to the user's terminal.

[1258] Step 28:

[1259] The device will display the plan information in a browser.

[1260] Step 29:

[1261] The server stores a log of all conversations and interactions in a database.

[1262] Step 30:

[1263] The user clicks the "Exit" button to end the session.

[1264] Step 31:

[1265] The terminal sends a termination request to the server.

[1266] Step 32:

[1267] The server ends the session and logs out the user.

[1268] Step 33:

[1269] The terminal displays the logout completion screen.

[1270] Example 2

[1271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1272] Modern online services face the challenge of making it difficult for users to obtain an experience equivalent to that of face-to-face customer service in a brick-and-mortar store. Furthermore, there is a need for systems that can accurately understand the user's emotional state and provide services based on that understanding, but conventional systems are inadequate in this regard. Furthermore, there is a need for systems that can log all conversations and interactions to prevent problems before they occur.

[1273] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1274] In this invention, the server includes an account authentication means for users to access the online platform and perform login authentication; a service selection means for users to select between chat support or avatar video and audio support; a means for generating a chat interface and response resources for exchanging messages in real time; a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; a means for saving logs of all conversations and interactions; a means including an emotion engine that analyzes the user's text messages, voice, and facial expressions to recognize their emotional state; and a means for providing a response method based on feedback information from the emotion engine. This allows for an online experience equivalent to face-to-face customer service in a physical store, enabling advanced responses based on the user's emotional state. Furthermore, by saving all conversations and interactions as logs, problems can be prevented before they occur.

[1275] "Account authentication means" refers to the means by which a user accesses an online platform, collates the authentication information (such as a user ID and password) required to log in, and verifies its legitimacy.

[1276] The "service selection means" is a means for transmitting information about whether a user selects chat support or video and audio support using an avatar to the server.

[1277] A "chat interface" is a user interface that allows users and staff to exchange text messages in real time.

[1278] The "means for generating corresponding resources" is a means for allocating and generating resources such as appropriate staff members and avatars based on the user's selection.

[1279] The "data analysis and plan generation means" is a means for analyzing data based on the user's needs and requests, and for designing and proposing the most suitable plan.

[1280] A "means for storing logs" is a means for recording all interactions and exchanges between a user and a system and storing them in a form that can be referenced at a later date.

[1281] An "emotion engine" is a technology that analyzes a user's text messages, voice, and facial expressions to recognize the user's emotional state in real time.

[1282] The "means for providing a response method based on feedback information from the emotion engine" is a means for appropriately adjusting the response method of the staff member or avatar based on the emotional state of the user analyzed by the emotion engine.

[1283] The present invention relates to a system that enables users to receive face-to-face customer service online in the same way as visiting a physical store, and provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[1284] This system provides an account authentication method for users to access the online platform and perform login authentication. When a user accesses the system using a web browser, the server sends the HTML, CSS, and JavaScript files for the top page to the user's device, which then displays the top page. When the user enters their user ID and password and clicks the submit button, the device sends this information to the server, which then checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, which the device displays.

[1285] Next, the user selects "chat support" or "avatar support" on the dashboard. The terminal sends this selection information to the server, and the server assigns available resources (staff or avatar) according to the user's selection. For example, if the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1286] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[1287] Furthermore, this system has an added emotion engine. The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. For example, in the case of chat support, the emotion engine analyzes the user's emotions from the text messages and feeds the information back to the server. The server provides a response method to the chat support resource in real time based on the feedback information. In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds the information back to the server. The server adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1288] For example, if a user sends a chat message saying, "Please tell me how to use this product," the system will analyze the message and provide appropriate information. If the emotion engine detects anxiety in a message such as, "I'm in a lot of trouble," the system will pass that information on to the appropriate staff member, who will then respond promptly and courteously to the user.

[1289] In this way, this system aims to provide an online experience similar to face-to-face customer service in a physical store. Users can receive optimal service suggestions without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[1290] Examples of prompts include:

[1291] "If the message entered by the user is 'Please tell me how to use the recommended product,' generate an appropriate product description."

[1292] "If the emotion engine interprets the user's facial expression as serious, be a little more careful with the avatar's response tone."

[1293] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1294] Step 1:

[1295] The user opens a web browser and accesses the system's URL. The server sends the top page HTML, CSS, and JavaScript files to the user's device. The device receives these files and displays the top page. The user's input is the system's URL, and the output is the display of the top page. Specifically, the user enters the URL in the browser's address bar and presses the Enter key.

[1296] Step 2:

[1297] The user enters a user ID and password into the login form and clicks the submit button. The terminal sends this input information to the server. The server compares the received information with a database and performs authentication. If authentication is successful, the server starts a session for the user, generates a dashboard page, and sends it to the user's terminal. The terminal displays the dashboard page. The input is the user ID and password, and the output is the display of the dashboard page. The specific action is when the user clicks the "Login" button.

[1298] Step 3:

[1299] The user selects "Chat support" or "Avatar support" on the dashboard. The device sends the selection information to the server. The server allocates available resources (staff or avatars) based on the user's selection. The input is the selection information, and the output is the allocation of resources. The specific operation is when the user clicks the "Chat support" or "Avatar support" button.

[1300] Step 4:

[1301] (For chat support) The server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages and saves them as a log. The input is the chat message, and the output is the message saved as a log. The specific action is when the user clicks the "Send message" button.

[1302] Step 5:

[1303] (In the case of avatar support) The server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. When the user asks a question or makes a request, the server analyzes it and generates a response according to the avatar's scenario. The input is the user's question or request, and the output is the avatar's response. In concrete terms, the user speaks into the microphone.

[1304] Step 6:

[1305] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The optimal plan is analyzed and generated based on this data. The server then sends the generated plan information to the user's device, which displays it. The input is the user's needs and desires, and the output is a display of the optimal plan. A specific operation is when the user clicks the "Confirm Plan" button.

[1306] Step 7:

[1307] The system is equipped with an emotion engine that analyzes the user's text messages, voice, and facial expressions. The emotion engine recognizes the user's emotional state in real time and feeds that information back to the server. The server uses the feedback information to adjust the chat response and avatar's response method. The input is the user's message and facial expression, and the output is feedback on how to respond. Specifically, the emotion engine recognizes the user's facial expression.

[1308] Step 8:

[1309] When all dialogue and exchanges are complete, the user clicks the "Exit" button, and the terminal sends an end request to the server. The server ends the session and logs the user out. The terminal displays a logout completion screen. The input is the end request, and the output is a message indicating that the logout has been completed. The specific action is when the user clicks the "Exit" button.

[1310] (Application example 2)

[1311] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1312] Conventional online shopping systems struggle to provide the same sense of security and face-to-face customer service as brick-and-mortar stores, and lack the means to alleviate users' dissatisfaction and anxiety. They also lack the ability to understand users' emotional state and respond optimally accordingly. As a result, the user experience remains stagnant, leading to problems with declining customer satisfaction.

[1313] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an account authentication means for a user to access the online platform and perform login authentication, a service selection means for the user to select from chat support or video and audio support using avatars, a means for generating a chat interface and corresponding resources for exchanging messages in real time, an emotion analysis means for analyzing emotions from the user's text messages and providing feedback, and a means for adjusting the chat support resources and avatar responses based on the emotion analysis results. This allows users to feel as secure online as if they were in a physical store, and receive optimal service in real time according to their emotions.

[1314] "Account authentication means" refers to the function by which a user accesses the online platform and enters an appropriate ID and password for authentication.

[1315] The "service selection means" is a function that provides an interface for the user to select either chat support or video and audio support using an avatar.

[1316] A "chat interface" is an interface that allows users and staff to exchange text messages in real time.

[1317] "Response resources" are staff or system resources assigned to provide services to users.

[1318] The "data analysis means" is a function that performs data analysis to propose the optimal plan based on the user's needs and requests.

[1319] The "plan generation means" is a function that generates an optimal proposed plan for the user based on the results of data analysis.

[1320] The "log storage means" is a function that stores the history of all conversations and interactions and makes them available for later reference as needed.

[1321] The "emotion analysis means" is a function that analyzes emotions from the user's text message and provides feedback on the results.

[1322] The "response adjustment means" is a function that adjusts the responses of chat resources and avatars based on the results of emotion analysis to provide an appropriate response.

[1323] A "chat room" is a dedicated virtual space where users and staff can exchange text messages in real time.

[1324] The "avatar generation means" is a function that generates an avatar and executes a scenario that responds to the user with audio and video.

[1325] The "scenario execution means" is a function that enables an avatar to respond to the user with voice and video based on a predefined response scenario.

[1326] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a real store, and also provides advanced services using an emotion engine. The system of the present invention operates as follows.

[1327] First, a user accesses the system's URL using a web browser and performs login authentication through the web browser interface. When the user enters their account information (user ID and password) and sends it to the server, the server checks it against a database, and if authentication is successful, the user's session begins. This authentication method prevents unauthorized access and ensures a secure connection.

[1328] After logging in, the user selects "chat support" or "avatar support" on the dashboard. The service selection means allows the user to select a chat interface for exchanging messages in real time as needed. Alternatively, the user can select voice and video responses from an avatar. The service selection means transmits the user's selection information to the server, and the server allocates available resources.

[1329] If the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the user's terminal and the terminal of the staff member in charge for display. Through this interface, the user and staff member exchange messages in real time, and the server saves all messages as a log. This log saving means can be used to prevent problems from occurring or as reference information later.

[1330] If the user selects video and audio support using an avatar, the server generates an avatar and executes a scenario that responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar. The avatar generation means and scenario execution means make it possible to provide a high level of interaction to the user.

[1331] Furthermore, in this embodiment, an emotion engine is added. The emotion analysis means analyzes emotions from the user's text message, voice, and facial expression. In the case of chat correspondence, the emotion engine analyzes the user's text message and feeds back the information to the server. The response adjustment means allows the server to provide an appropriate response method to the chat correspondence resource in real time based on the feedback information.

[1332] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions, sends the feedback information to the server, and the server adjusts the avatar's response method, thereby enabling the optimal response to the user's emotions.

[1333] As a specific example, if a user types into the chat interface, "I'm looking for a new smartphone. Which one do you recommend?", the emotion engine will analyze the emotion of the user's message, and based on that information, a staff member will respond by saying, "We will suggest the best options for you."

[1334] Examples of prompt sentences that may be used in the present invention include the following:

[1335] Prompt: Parse the user's message to recognize their sentiment and generate an appropriate response. Message: "I'm looking for a new smartphone. Which one would you recommend?"

[1336] The hardware and software used are as follows:

[1337] Web browser: displaying the user interface

[1338] Server: Provides API endpoints, manages sessions, operates the database, and runs the emotion engine

[1339] EmotionRecognition library: Analyzes user text messages and recognizes emotions in real time

[1340] Through these measures, users can feel as secure online as if they were in a physical store, and can receive optimal service tailored to their emotions in real time.

[1341] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1342] Step 1:

[1343] A user accesses the system's URL using a web browser. The terminal receives the top page HTML, CSS, and JavaScript files from the server and displays the top page, allowing the user to access the login screen.

[1344] Input: System URL

[1345] Output: Login screen displayed in web browser

[1346] What happens: A user enters a URL into the browser's address bar and loads a page.

[1347] Step 2:

[1348] The user enters account information (user ID and password) and clicks the submit button. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, the server starts the user's session, generates a dashboard page, sends it to the terminal, and the terminal displays it.

[1349] Input: User ID, Password

[1350] Output: Dashboard page upon successful authentication

[1351] Specific operation: The user enters their ID and password in the login form and presses the "Login" button.

[1352] Step 3:

[1353] The user selects either "chat support" or "avatar support" on the dashboard. The terminal sends the selection information to the server, and the server assigns available resources (staff or avatars).

[1354] Input: User selection information

[1355] Output: Resource allocation according to the selection

[1356] What happens: A user clicks on an option on a dashboard.

[1357] Step 4:

[1358] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1359] Input: User's chat response selection, text message

[1360] Output: Chat interface, real-time message exchange, logs

[1361] Specific operation: The server creates a chat room, and users and staff members can input, send, and receive chat messages.

[1362] Step 5:

[1363] If the user selects avatar support, the server generates an avatar and sends the virtual support room interface to the user's device. The server analyzes the user's questions and requests, generates appropriate responses, and provides them through the avatar.

[1364] Input: User avatar-enabled selection, user question or request

[1365] Output: Virtual interaction room, avatar responses

[1366] Specific operation: The server generates an avatar and executes a scenario to respond to the user's questions.

[1367] Step 6:

[1368] The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes the user's text messages and feeds the information back to the server. Based on the feedback information, the server provides chat support resources with appropriate responses in real time.

[1369] Input: User's text message, voice, and facial expression information

[1370] Output: Sentiment analysis results, feedback information, and appropriate response methods

[1371] Specific behavior: The emotion engine analyzes messages and voice and provides appropriate feedback instantly.

[1372] Step 7:

[1373] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1374] Input: User's voice and facial expression

[1375] Output: Sentiment analysis results, feedback information, appropriate avatar responses

[1376] Specific behavior: The emotion engine analyzes the user's voice and facial expressions and adjusts the avatar's response.

[1377] Step 8:

[1378] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[1379] Input: User's exit request

[1380] Output: Interaction log, session termination, logout completion screen

[1381] Specific operation: The user presses the end button, and the server performs the session termination process.

[1382] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1385] [Fourth embodiment]

[1386] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1387] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1389] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1393] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1394] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1397] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1398] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1399] As an embodiment of the present invention, a system is provided that allows users to receive face-to-face customer service online with the same feeling as visiting a real store. Specific processing and operation of the system are described in detail below.

[1400] 1. User Access and Login

[1401] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the login button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[1402] 2. Service Selection

[1403] On the dashboard, the user selects either "chat support" or "avatar video and audio support." The device sends this selection information to the server, which then assigns available resources (staff or avatars) based on the user's selection.

[1404] 3. Customer service

[1405] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. These messages are sent to the server and exchanged in real time, and all messages are stored on the server as a log.

[1406] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[1407] 4. Confirmation of requests and provision of plans

[1408] When a user communicates their needs and desires through chat or interaction with an avatar, the server records this and stores it in a database. The server then analyzes the recorded information to determine the optimal plan and generates plan information. The generated plan information is then sent from the server to the user's device and displayed on the device.

[1409] 5. Logging and terminating the session

[1410] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[1411] As a concrete example, a user accesses a service site, logs in, selects chat options, exchanges messages with a staff member in real time, receives a service plan proposal, and finally ends the session. This allows the user to receive the most suitable service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[1412] The processing flow will be explained below.

[1413] Step 1:

[1414] A user accesses the system's URL using a web browser.

[1415] Step 2:

[1416] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[1417] Step 3:

[1418] The device displays the top page in the browser.

[1419] Step 4:

[1420] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[1421] Step 5:

[1422] The terminal sends the entered account information to the server as a POST request.

[1423] Step 6:

[1424] The server checks the received account information against user information in its database.

[1425] Step 7:

[1426] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[1427] Step 8:

[1428] The device displays the dashboard page in a browser.

[1429] Step 9:

[1430] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[1431] Step 10:

[1432] The terminal sends the selected information to the server as a POST request.

[1433] Step 11:

[1434] The server allocates available resources (staff or avatars) based on the user's selection.

[1435] Step 12:

[1436] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[1437] Step 13:

[1438] The terminal will display the chat interface.

[1439] Step 14:

[1440] The user types a message in the chat interface and clicks the send button.

[1441] Step 15:

[1442] The device sends a chat message to the server.

[1443] Step 16:

[1444] The server delivers the received message to the staff member in charge, and similarly delivers the staff member's reply to the user.

[1445] Step 17:

[1446] The server stores all message logs.

[1447] Step 18:

[1448] If the server supports avatars, it generates avatars and prepares the scenario.

[1449] Step 19:

[1450] The server transmits the interface of the virtual correspondence room to the user's terminal.

[1451] Step 20:

[1452] The device displays the virtual response room interface.

[1453] Step 21:

[1454] The server executes the scenario in which the avatar responds to the user with voice and video.

[1455] Step 22:

[1456] The server generates appropriate responses to the user's questions and requests and provides them through an avatar.

[1457] Step 23:

[1458] Users communicate their needs and desires through chat or interaction with an avatar.

[1459] Step 24:

[1460] The server stores the requests received from the user in a database.

[1461] Step 25:

[1462] The server analyzes and generates the information for the best plan based on the user's needs.

[1463] Step 26:

[1464] The server transmits the generated plan information to the user's terminal.

[1465] Step 27:

[1466] The device will display the plan information on the screen.

[1467] Step 28:

[1468] The server stores a log of all conversations and interactions in a database.

[1469] Step 29:

[1470] The user clicks the "Exit" button to end the session.

[1471] Step 30:

[1472] The terminal sends a termination request to the server.

[1473] Step 31:

[1474] The server ends the session and logs out the user.

[1475] Step 32:

[1476] The terminal displays the logout completion screen.

[1477] Example 1

[1478] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1479] Conventional online customer service systems make it difficult for users to experience the same sense of security as face-to-face customer service in a brick-and-mortar store, and they face challenges in quickly and appropriately responding to user needs and requests. Furthermore, there is a lack of effective ways to store and analyze conversation history and logs for future reference or to improve services. Furthermore, when using avatars, the technological means to provide accurate responses to user questions and requests are not yet fully developed.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1481] In this invention, the server includes an account authentication means for allowing a user to access the online platform and perform login authentication, a service selection means for selecting between chat support or avatar video and audio support from the user, a means for generating a chat interface and response resources for exchanging messages in real time, a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests, a means for saving a log of all dialogues and exchanges, and, in the case of avatar video and audio support, a means for using a generative AI model to analyze questions and requests from the user and generate appropriate responses. This allows users to receive reliable face-to-face customer service online just like in a physical store, provides optimal services tailored to the user's needs, and effectively saves and analyzes the history of all dialogues.

[1482] "Account authentication means" refers to the means by which a user accesses the online platform and performs login authentication.

[1483] The "service selection means" is a means for a user to select a service that supports chat or video and audio using an avatar.

[1484] The "chat interface" is an interface that allows users and staff to exchange messages in real time.

[1485] The "means for generating response resources" refers to a means for generating resources (staff or avatars) required for chat response or avatar response.

[1486] The "data analysis and plan generation means" is a means for analyzing data to propose an optimal plan based on the user's needs and requests, and generating a plan.

[1487] "Means for storing logs" refers to a means for storing logs of all conversations and interactions.

[1488] "Means for using a generative AI model" refers to means for using a generative AI model to analyze questions or requests from users and generate appropriate responses in the case of video and audio support using avatars.

[1489] MODE FOR CARRYING OUT THE INVENTION

[1490] As an embodiment of this invention, we provide a system that allows users to receive face-to-face customer service online with the same sense of security as in a physical store. The system is described in detail below. Specific names of the hardware and software used are also specified.

[1491] This system is based on the premise that users will use a web browser to access the service. The server runs on web server software such as Apache or Nginx. A relational database such as MySQL or PostgreSQL is used as the database. The front end that users access is built using web technologies such as HTML, CSS, and JavaScript.

[1492] 1. User Access and Login

[1493] When a user accesses the system's URL using a web browser (e.g., Chrome or Firefox), the server sends the top page HTML, CSS, and JavaScript files to the user's device. The user's device receives this and displays the top page in the browser. The user enters their account information (user ID and password) and clicks the login button. The device sends this information to the server, which checks it against a database for authentication. If authentication is successful, the server generates a session ID and sends the dashboard page to the user's device, which displays it.

[1494] 2. Service Selection

[1495] Users select the option "chat support" or "avatar video and audio support" on the dashboard. The user's device sends the selection to the server, which then allocates the necessary resources (staff or avatars) based on the selection.

[1496] 3. Customer service

[1497] If chat is selected, the server creates a dedicated chat room and sends the chat interface HTML and JavaScript to the user's and staff's devices. The devices display this, allowing for the exchange of text messages in real time. Text messages are exchanged via the server and are all saved as logs.

[1498] If avatar-based video and audio support is selected, the server generates a 3D avatar and prepares the scenario. The virtual support room interface is sent to the user's device, which displays it. The user's questions and requests are sent from the device to the server, which uses a generative AI model (e.g., ChatGPT) to generate appropriate responses and provides them through the avatar. All of these interactions are also logged.

[1499] 4. Confirmation of requests and provision of plans

[1500] The needs and requests communicated by users through chat or avatars are recorded on the server. The server analyzes the recorded data and generates an optimal plan. This plan information is sent from the server to the user's device and displayed on the device.

[1501] 5. Logging and terminating the session

[1502] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[1503] Examples of prompt statements

[1504] As a concrete example, let's assume that the system implements a series of steps in which a user accesses a service site, logs in, selects chat support, and exchanges messages with a staff member in real time. By inputting the following prompt sentences into the generative AI model, we can specifically explain how the system works.

[1505] Example prompt sentence:

[1506] "You have devised a system that allows users to safely receive face-to-face customer service online. Based on the specific scenario below, please write down how a user would use this service. The scenario includes accessing the service from a web browser, logging in, chatting, exchanging messages with a staff member, and receiving a service plan proposal."

[1507] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1508] Step 1: User accesses the system's URL in a web browser

[1509] A user accesses the system's URL using a web browser (e.g., Chrome or Firefox). The server generates the HTML, CSS, and JavaScript files for the top page and sends them to the user's device. The device receives these files and displays the top page in the browser.

[1510] Input: User enters a URL

[1511] Output: The HTML, CSS, and JavaScript files for the top page are sent to the terminal, and the top page is displayed in the browser.

[1512] Step 2: User enters account information and logs in

[1513] The user enters their account information (user ID and password) into the input fields on the top page and clicks the login button. The device uses JavaScript to perform basic format checks on the input data and then sends it to the server.

[1514] Input: Enter your account information (user ID, password)

[1515] Output: The input data is sent to the server

[1516] Step 3: The server authenticates the account

[1517] The server checks the received account information against its database. If authentication is successful, the server generates a session ID, dynamically generates a dashboard page with this information, and sends it to the user's device.

[1518] Input: Entered account information

[1519] Data processing and data calculation: Matching with database and generating session ID

[1520] Output: Dashboard page and session ID sent to device

[1521] Step 4: User selects service

[1522] The user selects "chat support" or "avatar video and audio support" from the options on the dashboard, and the device sends the selection information to the server.

[1523] Input: Select service (chat support, avatar support)

[1524] Output: Selection information is sent to the server

[1525] Step 5: Server allocates resources

[1526] Based on the received selection information, the server allocates the necessary resources (staff or avatars) and generates the necessary interfaces, such as chat rooms or virtual interaction rooms.

[1527] Input: Service selection information

[1528] Data processing and data calculation: Allocation of corresponding resources and generation of interfaces

[1529] Output: The appropriate interface is sent to the user's terminal.

[1530] Step 6: Start serving customers via chat

[1531] If the user selects chat support, a chat room is created and a chat interface is sent to the user's and staff's terminals.

[1532] Input: Chat response selection

[1533] Output: The chat room and chat interface are sent to the device.

[1534] Step 7: Exchange chat messages and save logs

[1535] Users and staff can type and send text messages, and the terminals send the messages to the server, which exchanges messages in real time and stores all messages as a log.

[1536] Input: Messages entered by users and staff

[1537] Data processing and data calculation: Message sending and receiving and log storage

[1538] Output: Real-time exchange of messages and storage of logs

[1539] Step 8: Start serving customers with avatar support

[1540] If the user selects avatar-based video and audio support, the server generates an avatar and prepares the scenario. It also generates a virtual support room interface and sends it to the user's device.

[1541] Input: Avatar-enabled selection

[1542] Output: The virtual room and avatar are sent to the user's device.

[1543] Step 9: Avatar response and logging

[1544] When a user types a question or request, the device sends it to the server, which uses a generative AI model to generate a response and delivers it through an avatar. All interactions are logged.

[1545] Input: User questions or requests

[1546] Data processing and data calculation: response generation and logging using generative AI models

[1547] Output: Avatar response and log storage

[1548] Step 10: Confirm your needs and provide a plan

[1549] The needs and requests communicated by the user are recorded on the server. The server analyzes the recorded data and generates an optimal plan. The generated plan information is sent to the user's device and displayed on the device.

[1550] Input: User needs and desires

[1551] Data processing and data calculations: Data analysis and plan generation

[1552] Output: Plan information is sent to the user's device

[1553] Step 11: Save the log and end the session

[1554] All conversations and interactions are logged on the server and managed for future reference. When the user clicks the "Exit" button, the device sends an end request to the server, which then ends the session and logs the user out. The device then displays a logout completion screen, indicating that use of the system has ended.

[1555] Input: User request to terminate

[1556] Data processing and data calculation: Session termination processing and logout processing

[1557] Output: The logout completion screen is displayed on the terminal.

[1558] (Application example 1)

[1559] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1560] With the development of online shopping, users are increasingly seeking the same sense of security and experience as face-to-face customer service in a physical store. However, current online platforms make it difficult for users to receive immediate, personalized assistance, especially when they need detailed information about a product or are unsure about a purchase, resulting in a decline in satisfaction and a decrease in purchasing motivation. Additionally, there is a lack of mechanisms for appropriately logging and storing conversation content and customer needs, and utilizing them to improve services in the future.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1562] In this invention, the server includes a means for users to interact with avatars and specialized staff in real time through a shopping experience in a virtual store, an account authentication means, and a service selection means for users to choose between chat support or avatar video and audio support. This allows users to enjoy the same sense of security as face-to-face customer service in a physical store while online, and to receive optimal suggestions based on their individual needs. In addition, by storing a log of all conversations and interactions, this information can be used to improve services in the future.

[1563] "User" means any individual or group of people who accesses and uses the functionality of the system or platform.

[1564] "Online Platform" refers to software or websites provided over the Internet that offer specific services or functions to users.

[1565] "Account authentication means" refers to a mechanism for verifying that a user is a legitimate user when accessing a system or platform.

[1566] "Service selection means" refers to an interface or mechanism that allows a user to select one of multiple service options provided.

[1567] "Chat interface" refers to a screen or function that allows users and support staff to exchange text messages in real time.

[1568] "Supporting resources" refers to the technical elements such as servers, network equipment, and software required for chat support and avatar video and audio support.

[1569] "Data analysis and plan generation means" refers to a mechanism for collecting and analyzing data to propose an optimal service plan based on the user's needs and requests, and for generating a plan based on the results of the analysis.

[1570] "Log storage means" refers to a mechanism for storing records of all conversations and interactions so that they can be referenced at a later date.

[1571] A "virtual store" refers to a virtual store space provided on the Internet, where users can browse and purchase products just as if they were in a physical store.

[1572] An "avatar" is a virtual person or character used to interact with users and has the ability to respond through voice and video.

[1573] "Specialist staff" refers to personnel who have specialized knowledge about products and services and who provide assistance to users.

[1574] This invention is an online system that allows users to shop in a virtual store. This system provides users with a sense of security similar to face-to-face customer service in a real store, and includes a function that allows users to interact with avatars and professional staff in real time.

[1575] System Overview

[1576] The system mainly consists of the following components:

[1577] 1. Account Authentication Methods

[1578] 2. Service Selection Method

[1579] 3. Chat interface and corresponding resource generation means

[1580] 4. Data Analysis and Plan Generation Methods

[1581] 5. Log storage method

[1582] 6. Real-time interaction with avatars and expert staff

[1583] Account authentication method

[1584] It is a means for users to access online platforms and perform login authentication. It uses software such as Flask and SQLAlchemy to perform the user ID and password authentication process. This is the gateway for users to access the system and is a function to maintain security.

[1585] Service Selection Method

[1586] After logging in, users can choose between chat support or avatar video and audio support. This selection information is sent to the server, which allocates appropriate resources based on the user's selection. The user interface is built using HTML, CSS, and JavaScript.

[1587] Chat interface and corresponding resource generation means

[1588] If chat support is selected, the server generates a dedicated chat room and provides an interface for exchanging text messages in real time, using real-time communication technologies such as Flask-SocketIO. This allows users and expert staff to communicate in real time.

[1589] Data Analysis and Plan Generation Tools

[1590] It is a means of proposing optimal service plans based on the user's needs and requests. Here, machine learning models and generative AI models are used to collect and analyze data obtained from users, thereby generating and providing individually optimized plans for the user.

[1591] Log storage method

[1592] It is a way to store all conversations and interactions as logs, which can be referenced later and used to improve the service. The database is SQLite or another RDBMS.

[1593] Real-time interaction with avatars and expert staff

[1594] If the user selects avatar video and audio support, the server generates an avatar and executes a scenario that responds to the user with audio and video. This process utilizes text-to-speech (TTS) and video rendering technologies. The avatar's movements and responses are based on a generative AI model.

[1595] Examples of concrete examples and prompts

[1596] Specific examples

[1597] The user launches the virtual store app and logs in by entering their user ID and password on the login screen. They then enter a chat room and can ask questions about products in real time with a specialist staff member. An avatar will explain the product using video and audio based on the scenario, and if the user is satisfied, they can make a purchase.

[1598] Prompt Sentence Examples

[1599] User launches app and enters login details:

[1600] "Please enter your login information"

[1601] Join the chat room:

[1602] "Joined the room"

[1603] Chat message exchange:

[1604] User: "Please describe this product."

[1605] Staff: "Please wait. This product has the following features..."

[1606] Avatar response:

[1607] "Here are some products we recommend to you..."

[1608] This system allows users to enjoy the same high-quality service online as they would in a physical store, and also enables the storage of logs that will lead to improved service quality.

[1609] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1610] Step 1:

[1611] A user accesses the online platform and enters their user ID and password on the login screen for authentication. The server receives the entered user ID and password information and verifies them against the database. If authentication is successful, the server starts the user's session, generates a dashboard page, and sends it to the user's device.

[1612] Input: User ID, Password

[1613] Data processing: Matching with database, creating sessions

[1614] Output: Dashboard page displayed

[1615] Step 2:

[1616] The user selects the "chat support" or "avatar video and audio support" service on the dashboard. The device sends the user's selection information to the server, which then allocates the appropriate resource (specialist staff or avatar).

[1617] Input: User's service selection information

[1618] Data processing: receiving selection information, allocating resources

[1619] Output: Prepare corresponding resources

[1620] Step 3:

[1621] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The user and the staff exchange text messages in real time, and all messages are sent to the server and saved as a log.

[1622] Input: (Chat selection) User message, staff message

[1623] Data processing: Chat room creation, message exchange, message log storage

[1624] Output: Real-time chat

[1625] Step 4:

[1626] If the user selects video and audio support via an avatar, the server generates an avatar and executes a scenario in which the user responds via video and audio. When the user makes a question or request, the server analyzes it using a generative AI model, generates an appropriate response, and provides it through the avatar.

[1627] Input: (Avatar selection) User question or request

[1628] Data processing: Avatar generation, analysis using generative AI models, response generation

[1629] Output: Avatar video and audio response

[1630] Step 5:

[1631] The server analyzes the data based on the user's needs and requests and generates an optimal service plan. The generated service plan information is sent from the server to the user's terminal and displayed on the user's terminal.

[1632] Input: User needs and desires

[1633] Data processing: Data collection and analysis, service plan generation

[1634] Output: Display service plan information

[1635] Step 6:

[1636] By saving a log of all conversations and interactions, they can be managed in a form that can be referenced later. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs out the user.

[1637] Input: Content of conversation or exchange, end request

[1638] Data processing: saving logs, ending sessions

[1639] Output: Display of logout completion screen

[1640] In this way, a system is realized that allows users to receive real-time support throughout their shopping experience in a virtual store.

[1641] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1642] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a physical store, and that provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[1643] 1. User Access and Login

[1644] A user accesses the system's URL using a web browser. The server sends the top page HTML, CSS, and JavaScript files to the user's device, and the top page is displayed on the device. The user enters account information (user ID and password) and clicks the submit button. The device sends the entered account information to the server, which checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, and the device displays it.

[1645] 2. Service Selection

[1646] On the dashboard, the user selects either "chat support" or "avatar support." The device sends the selection information to the server, which then assigns available resources (staff or avatars) according to the user's selection.

[1647] 3. Customer service

[1648] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1649] If the user selects avatar-based video and audio support, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[1650] 4. Confirmation of requests and provision of plans

[1651] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The server then analyzes the recorded information to find the optimal plan and generates plan information. The generated plan information is sent from the server to the user's device and displayed on the device.

[1652] 5. Introducing the Emotion Engine

[1653] In this embodiment, an emotion engine is added. The emotion engine analyzes the user's text message, voice, and facial expression to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes emotions from the user's text message and feeds the information back to the server. Based on the feedback information, the server provides a response method to the chat support resource in real time.

[1654] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1655] 6. Logging and terminating sessions

[1656] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[1657] As a specific example, a user accesses a service site, logs in, selects chat, and exchanges messages with a staff member in real time. The emotion engine analyzes the user's emotions and provides appropriate feedback to the staff member, enabling them to provide the most appropriate service to the user. There is also a series of steps that provide plan information generated based on the user's needs and requests. This allows the user to receive the most appropriate service proposal without waiting, and all conversations are saved as logs, making it possible to prevent problems from occurring.

[1658] The processing flow will be explained below.

[1659] Step 1:

[1660] A user accesses the system's URL using a web browser.

[1661] Step 2:

[1662] The server sends the top page HTML, CSS, and JavaScript files to the terminal.

[1663] Step 3:

[1664] The device displays the top page in the browser.

[1665] Step 4:

[1666] The user enters account information (user ID and password) in the login form on the top page and clicks the submit button.

[1667] Step 5:

[1668] The terminal sends the entered account information to the server as a POST request.

[1669] Step 6:

[1670] The server checks the received account information against user information in its database.

[1671] Step 7:

[1672] If the server is successful in the authentication, it starts a session for the user, generates a dashboard page, and sends it to the terminal.

[1673] Step 8:

[1674] The device displays the dashboard page in a browser.

[1675] Step 9:

[1676] The user selects "Chat Enabled" or "Avatar Enabled" on the dashboard.

[1677] Step 10:

[1678] The terminal sends the selected information to the server as a POST request.

[1679] Step 11:

[1680] The server allocates available resources (staff or avatars) based on the user's selection.

[1681] Step 12:

[1682] If the server supports chat, it creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge.

[1683] Step 13:

[1684] The terminal will display the chat interface.

[1685] Step 14:

[1686] The user types a message in the chat interface and clicks the send button.

[1687] Step 15:

[1688] The device sends a chat message to the server.

[1689] Step 16:

[1690] The server delivers the received message to the staff member in charge, and the staff member's reply is also delivered to the user. In addition, an emotion engine within the chat interface analyzes the user's message in real time and feeds the emotional information back to the server.

[1691] Step 17:

[1692] Based on the emotional feedback information, the server provides the appropriate response method to the staff member in charge, which is reflected on the chat interface.

[1693] Step 18:

[1694] The server stores all messages and sentiment analysis logs.

[1695] Step 19:

[1696] If the server supports avatars, it generates avatars and prepares the scenario.

[1697] Step 20:

[1698] The server transmits the interface of the virtual correspondence room to the user's terminal.

[1699] Step 21:

[1700] The device displays the virtual response room interface.

[1701] Step 22:

[1702] The server executes a scenario in which the avatar responds to the user with voice and video. The emotion engine within the avatar interface analyzes the user's voice and facial expressions in real time and feeds the emotion information back to the server.

[1703] Step 23:

[1704] The server adjusts the avatar's response method based on the emotional feedback information and provides it to the user.

[1705] Step 24:

[1706] Users communicate their needs and desires through chat or interaction with an avatar.

[1707] Step 25:

[1708] The server records and stores the user's requests and needs in a database.

[1709] Step 26:

[1710] The server analyzes the optimum plan based on the recorded information and generates plan information.

[1711] Step 27:

[1712] The server transmits the generated plan information to the user's terminal.

[1713] Step 28:

[1714] The device will display the plan information in a browser.

[1715] Step 29:

[1716] The server stores a log of all conversations and interactions in a database.

[1717] Step 30:

[1718] The user clicks the "Exit" button to end the session.

[1719] Step 31:

[1720] The terminal sends a termination request to the server.

[1721] Step 32:

[1722] The server ends the session and logs out the user.

[1723] Step 33:

[1724] The terminal displays the logout completion screen.

[1725] Example 2

[1726] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1727] Modern online services face the challenge of making it difficult for users to obtain an experience equivalent to that of face-to-face customer service in a brick-and-mortar store. Furthermore, there is a need for systems that can accurately understand the user's emotional state and provide services based on that understanding, but conventional systems are inadequate in this regard. Furthermore, there is a need for systems that can log all conversations and interactions to prevent problems before they occur.

[1728] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1729] In this invention, the server includes an account authentication means for users to access the online platform and perform login authentication; a service selection means for users to select between chat support or avatar video and audio support; a means for generating a chat interface and response resources for exchanging messages in real time; a data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; a means for saving logs of all conversations and interactions; a means including an emotion engine that analyzes the user's text messages, voice, and facial expressions to recognize their emotional state; and a means for providing a response method based on feedback information from the emotion engine. This allows for an online experience equivalent to face-to-face customer service in a physical store, enabling advanced responses based on the user's emotional state. Furthermore, by saving all conversations and interactions as logs, problems can be prevented before they occur.

[1730] "Account authentication means" refers to the means by which a user accesses an online platform, collates the authentication information (such as a user ID and password) required to log in, and verifies its legitimacy.

[1731] The "service selection means" is a means for transmitting information about whether a user selects chat support or video and audio support using an avatar to the server.

[1732] A "chat interface" is a user interface that allows users and staff to exchange text messages in real time.

[1733] The "means for generating corresponding resources" is a means for allocating and generating resources such as appropriate staff members and avatars based on the user's selection.

[1734] The "data analysis and plan generation means" is a means for analyzing data based on the user's needs and requests, and for designing and proposing the most suitable plan.

[1735] A "means for storing logs" is a means for recording all interactions and exchanges between a user and a system and storing them in a form that can be referenced at a later date.

[1736] An "emotion engine" is a technology that analyzes a user's text messages, voice, and facial expressions to recognize the user's emotional state in real time.

[1737] The "means for providing a response method based on feedback information from the emotion engine" is a means for appropriately adjusting the response method of the staff member or avatar based on the emotional state of the user analyzed by the emotion engine.

[1738] The present invention relates to a system that enables users to receive face-to-face customer service online in the same way as visiting a physical store, and provides advanced services using an emotion engine. Specific processing and operation of the system are described in detail below.

[1739] This system provides an account authentication method for users to access the online platform and perform login authentication. When a user accesses the system using a web browser, the server sends the HTML, CSS, and JavaScript files for the top page to the user's device, which then displays the top page. When the user enters their user ID and password and clicks the submit button, the device sends this information to the server, which then checks it against a database. If authentication is successful, the server starts the user's session, generates and sends a dashboard page, which the device displays.

[1740] Next, the user selects "chat support" or "avatar support" on the dashboard. The terminal sends this selection information to the server, and the server assigns available resources (staff or avatar) according to the user's selection. For example, if the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1741] If the user selects video and audio support via an avatar, the server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. The server then executes a scenario in which the avatar responds to the user via audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar.

[1742] Furthermore, this system has an added emotion engine. The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. For example, in the case of chat support, the emotion engine analyzes the user's emotions from the text messages and feeds the information back to the server. The server provides a response method to the chat support resource in real time based on the feedback information. In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds the information back to the server. The server adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1743] For example, if a user sends a chat message saying, "Please tell me how to use this product," the system will analyze the message and provide appropriate information. If the emotion engine detects anxiety in a message such as, "I'm in a lot of trouble," the system will pass that information on to the appropriate staff member, who will then respond promptly and courteously to the user.

[1744] In this way, this system aims to provide an online experience similar to face-to-face customer service in a physical store. Users can receive optimal service suggestions without waiting, and all conversations are saved as logs, making it possible to prevent problems before they occur.

[1745] Examples of prompts include:

[1746] "If the message entered by the user is 'Please tell me how to use the recommended product,' generate an appropriate product description."

[1747] "If the emotion engine interprets the user's facial expression as serious, be a little more careful with the avatar's response tone."

[1748] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1749] Step 1:

[1750] The user opens a web browser and accesses the system's URL. The server sends the top page HTML, CSS, and JavaScript files to the user's device. The device receives these files and displays the top page. The user's input is the system's URL, and the output is the display of the top page. Specifically, the user enters the URL in the browser's address bar and presses the Enter key.

[1751] Step 2:

[1752] The user enters a user ID and password into the login form and clicks the submit button. The terminal sends this input information to the server. The server compares the received information with a database and performs authentication. If authentication is successful, the server starts a session for the user, generates a dashboard page, and sends it to the user's terminal. The terminal displays the dashboard page. The input is the user ID and password, and the output is the display of the dashboard page. The specific action is when the user clicks the "Login" button.

[1753] Step 3:

[1754] The user selects "Chat support" or "Avatar support" on the dashboard. The device sends the selection information to the server. The server allocates available resources (staff or avatars) based on the user's selection. The input is the selection information, and the output is the allocation of resources. The specific operation is when the user clicks the "Chat support" or "Avatar support" button.

[1755] Step 4:

[1756] (For chat support) The server creates a dedicated chat room and sends a chat interface to the terminals of the user and the staff member in charge. The terminal displays the chat interface, and the user and staff member exchange text messages in real time. The server receives these messages and saves them as a log. The input is the chat message, and the output is the message saved as a log. The specific action is when the user clicks the "Send message" button.

[1757] Step 5:

[1758] (In the case of avatar support) The server generates an avatar and prepares a scenario. The server sends the virtual support room interface to the user's device, which displays it. When the user asks a question or makes a request, the server analyzes it and generates a response according to the avatar's scenario. The input is the user's question or request, and the output is the avatar's response. In concrete terms, the user speaks into the microphone.

[1759] Step 6:

[1760] When a user communicates their needs and desires through chat or avatar interaction, the server records this and stores it in a database. The optimal plan is analyzed and generated based on this data. The server then sends the generated plan information to the user's device, which displays it. The input is the user's needs and desires, and the output is a display of the optimal plan. A specific operation is when the user clicks the "Confirm Plan" button.

[1761] Step 7:

[1762] The system is equipped with an emotion engine that analyzes the user's text messages, voice, and facial expressions. The emotion engine recognizes the user's emotional state in real time and feeds that information back to the server. The server uses the feedback information to adjust the chat response and avatar's response method. The input is the user's message and facial expression, and the output is feedback on how to respond. Specifically, the emotion engine recognizes the user's facial expression.

[1763] Step 8:

[1764] When all dialogue and exchanges are complete, the user clicks the "Exit" button, and the terminal sends an end request to the server. The server ends the session and logs the user out. The terminal displays a logout completion screen. The input is the end request, and the output is a message indicating that the logout has been completed. The specific action is when the user clicks the "Exit" button.

[1765] (Application example 2)

[1766] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1767] Conventional online shopping systems struggle to provide the same sense of security and face-to-face customer service as brick-and-mortar stores, and lack the means to alleviate users' dissatisfaction and anxiety. They also lack the ability to understand users' emotional state and respond optimally accordingly. As a result, the user experience remains stagnant, leading to problems with declining customer satisfaction.

[1768] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an account authentication means for a user to access the online platform and perform login authentication, a service selection means for the user to select from chat support or video and audio support using avatars, a means for generating a chat interface and corresponding resources for exchanging messages in real time, an emotion analysis means for analyzing emotions from the user's text messages and providing feedback, and a means for adjusting the chat support resources and avatar responses based on the emotion analysis results. This allows users to feel as secure online as if they were in a physical store, and receive optimal service in real time according to their emotions.

[1769] "Account authentication means" refers to the function by which a user accesses the online platform and enters an appropriate ID and password for authentication.

[1770] The "service selection means" is a function that provides an interface for the user to select either chat support or video and audio support using an avatar.

[1771] A "chat interface" is an interface that allows users and staff to exchange text messages in real time.

[1772] "Response resources" are staff or system resources assigned to provide services to users.

[1773] The "data analysis means" is a function that performs data analysis to propose the optimal plan based on the user's needs and requests.

[1774] The "plan generation means" is a function that generates an optimal proposed plan for the user based on the results of data analysis.

[1775] The "log storage means" is a function that stores the history of all conversations and interactions and makes them available for later reference as needed.

[1776] The "emotion analysis means" is a function that analyzes emotions from the user's text message and provides feedback on the results.

[1777] The "response adjustment means" is a function that adjusts the responses of chat resources and avatars based on the results of emotion analysis to provide an appropriate response.

[1778] A "chat room" is a dedicated virtual space where users and staff can exchange text messages in real time.

[1779] The "avatar generation means" is a function that generates an avatar and executes a scenario that responds to the user with audio and video.

[1780] The "scenario execution means" is a function that enables an avatar to respond to the user with voice and video based on a predefined response scenario.

[1781] As an embodiment of the present invention, a system is provided that enables users to receive face-to-face customer service online with the same feeling as visiting a real store, and also provides advanced services using an emotion engine. The system of the present invention operates as follows.

[1782] First, a user accesses the system's URL using a web browser and performs login authentication through the web browser interface. When the user enters their account information (user ID and password) and sends it to the server, the server checks it against a database, and if authentication is successful, the user's session begins. This authentication method prevents unauthorized access and ensures a secure connection.

[1783] After logging in, the user selects "chat support" or "avatar support" on the dashboard. The service selection means allows the user to select a chat interface for exchanging messages in real time as needed. Alternatively, the user can select voice and video responses from an avatar. The service selection means transmits the user's selection information to the server, and the server allocates available resources.

[1784] If the user selects chat support, the server creates a dedicated chat room and sends a chat interface to the user's terminal and the terminal of the staff member in charge for display. Through this interface, the user and staff member exchange messages in real time, and the server saves all messages as a log. This log saving means can be used to prevent problems from occurring or as reference information later.

[1785] If the user selects video and audio support using an avatar, the server generates an avatar and executes a scenario that responds to the user with audio and video. When the user asks a question or makes a request, the server analyzes it, generates an appropriate response, and provides it through the avatar. The avatar generation means and scenario execution means make it possible to provide a high level of interaction to the user.

[1786] Furthermore, in this embodiment, an emotion engine is added. The emotion analysis means analyzes emotions from the user's text message, voice, and facial expression. In the case of chat correspondence, the emotion engine analyzes the user's text message and feeds back the information to the server. The response adjustment means allows the server to provide an appropriate response method to the chat correspondence resource in real time based on the feedback information.

[1787] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions, sends the feedback information to the server, and the server adjusts the avatar's response method, thereby enabling the optimal response to the user's emotions.

[1788] As a specific example, if a user types into the chat interface, "I'm looking for a new smartphone. Which one do you recommend?", the emotion engine will analyze the emotion of the user's message, and based on that information, a staff member will respond by saying, "We will suggest the best options for you."

[1789] Examples of prompt sentences that may be used in the present invention include the following:

[1790] Prompt: Parse the user's message to recognize their sentiment and generate an appropriate response. Message: "I'm looking for a new smartphone. Which one would you recommend?"

[1791] The hardware and software used are as follows:

[1792] Web browser: displaying the user interface

[1793] Server: Provides API endpoints, manages sessions, operates the database, and runs the emotion engine

[1794] EmotionRecognition library: Analyzes user text messages and recognizes emotions in real time

[1795] Through these measures, users can feel as secure online as if they were in a physical store, and can receive optimal service tailored to their emotions in real time.

[1796] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1797] Step 1:

[1798] A user accesses the system's URL using a web browser. The terminal receives the top page HTML, CSS, and JavaScript files from the server and displays the top page, allowing the user to access the login screen.

[1799] Input: System URL

[1800] Output: Login screen displayed in web browser

[1801] What happens: A user enters a URL into the browser's address bar and loads a page.

[1802] Step 2:

[1803] The user enters account information (user ID and password) and clicks the submit button. The terminal sends this information to the server, which then authenticates it by checking it against a database. If authentication is successful, the server starts the user's session, generates a dashboard page, sends it to the terminal, and the terminal displays it.

[1804] Input: User ID, Password

[1805] Output: Dashboard page upon successful authentication

[1806] Specific operation: The user enters their ID and password in the login form and presses the "Login" button.

[1807] Step 3:

[1808] The user selects either "chat support" or "avatar support" on the dashboard. The terminal sends the selection information to the server, and the server assigns available resources (staff or avatars).

[1809] Input: User selection information

[1810] Output: Resource allocation according to the selection

[1811] What happens: A user clicks on an option on a dashboard.

[1812] Step 4:

[1813] If the user selects chat, the server creates a dedicated chat room and sends a chat interface to the user's and the staff's terminals. The terminals display the chat interface, and the user and staff exchange text messages in real time. The server receives these messages, exchanges them in real time, and saves all messages as a log.

[1814] Input: User's chat response selection, text message

[1815] Output: Chat interface, real-time message exchange, logs

[1816] Specific operation: The server creates a chat room, and users and staff members can input, send, and receive chat messages.

[1817] Step 5:

[1818] If the user selects avatar support, the server generates an avatar and sends the virtual support room interface to the user's device. The server analyzes the user's questions and requests, generates appropriate responses, and provides them through the avatar.

[1819] Input: User avatar-enabled selection, user question or request

[1820] Output: Virtual interaction room, avatar responses

[1821] Specific operation: The server generates an avatar and executes a scenario to respond to the user's questions.

[1822] Step 6:

[1823] The emotion engine analyzes the user's text messages, voice, and facial expressions to recognize the user's emotional state in real time. In the case of chat support, the emotion engine analyzes the user's text messages and feeds the information back to the server. Based on the feedback information, the server provides chat support resources with appropriate responses in real time.

[1824] Input: User's text message, voice, and facial expression information

[1825] Output: Sentiment analysis results, feedback information, and appropriate response methods

[1826] Specific behavior: The emotion engine analyzes messages and voice and provides appropriate feedback instantly.

[1827] Step 7:

[1828] In the case of avatar support, the emotion engine analyzes the user's voice and facial expressions and feeds that information back to the server, which then adjusts the avatar's response method based on the feedback information and provides an appropriate response to the user.

[1829] Input: User's voice and facial expression

[1830] Output: Sentiment analysis results, feedback information, appropriate avatar responses

[1831] Specific behavior: The emotion engine analyzes the user's voice and facial expressions and adjusts the avatar's response.

[1832] Step 8:

[1833] All conversations and exchanges are logged on the server and managed for future reference. When the user clicks the "Exit" button, an end request is sent from the terminal to the server, and the server ends the session and logs the user out. The terminal displays a logout completion screen, and the user's use of the system is terminated.

[1834] Input: User's exit request

[1835] Output: Interaction log, session termination, logout completion screen

[1836] Specific operation: The user presses the end button, and the server performs the session termination process.

[1837] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1838] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1839] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1840] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1841] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1842] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1843] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1844] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1845] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1846] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1847] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1848] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1849] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1850] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1851] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1852] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1853] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1854] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1855] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1856] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1857] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1858] The following is further disclosed regarding the above embodiment.

[1859] (Claim 1)

[1860] an account authentication means for users to access the online platform and perform login authentication;

[1861] A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user;

[1862] means for generating a chat interface and corresponding resources for exchanging messages in real time;

[1863] A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests;

[1864] A means of keeping a log of all conversations and interactions;

[1865] A system including:

[1866] (Claim 2)

[1867] 10. The system according to claim 1, further comprising means for generating a dedicated chat room in the case of chat support, for exchanging messages between the user and the staff member in real time.

[1868] (Claim 3)

[1869] 10. The system of claim 1, further comprising means for generating an avatar in the case of avatar video and audio support, and executing a scenario in which the avatar responds to the user with audio and video.

[1870] "Example 1"

[1871] (Claim 1)

[1872] an account authentication means for users to access the online platform and perform login authentication;

[1873] A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user;

[1874] means for generating a chat interface and corresponding resources for exchanging messages in real time;

[1875] A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests;

[1876] A means of keeping a log of all conversations and interactions;

[1877] In the case of avatar-based video and audio support, a means for using a generative AI model to analyze user questions and requests and generate appropriate responses;

[1878] A system including:

[1879] (Claim 2)

[1880] 10. The system according to claim 1, further comprising means for generating a dedicated chat room in the case of chat support, for exchanging messages between the user and the staff member in real time.

[1881] (Claim 3)

[1882] 10. The system of claim 1, further comprising means for generating an avatar in the case of avatar video and audio support, and executing a scenario in which the avatar responds to the user with audio and video.

[1883] "Application Example 1"

[1884] (Claim 1)

[1885] an account authentication means for users to access the online platform and perform login authentication;

[1886] A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user;

[1887] means for generating a chat interface and corresponding resources for exchanging messages in real time;

[1888] A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests;

[1889] A means of keeping a log of all conversations and interactions;

[1890] A means for users to interact with avatars and expert staff in real time throughout the virtual store shopping experience;

[1891] A system including:

[1892] (Claim 2)

[1893] 10. The system according to claim 1, further comprising means for generating a dedicated chat room in the case of chat support, for exchanging messages between the user and the staff member in real time.

[1894] (Claim 3)

[1895] 10. The system of claim 1, further comprising means for generating an avatar in the case of avatar video and audio support, and executing a scenario in which the avatar responds to the user with audio and video.

[1896] "Example 2: Combining Emotion Engines"

[1897] (Claim 1)

[1898] an account authentication means for users to access the online platform and perform login authentication;

[1899] A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user;

[1900] means for generating a chat interface and corresponding resources for exchanging messages in real time;

[1901] A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests;

[1902] A means of keeping a log of all conversations and interactions;

[1903] means including an emotion engine that analyzes a user's text messages, voice, and facial expressions to recognize the user's emotional state;

[1904] a means for providing a response method based on feedback information from the emotion engine;

[1905] A system including:

[1906] (Claim 2)

[1907] 10. The system according to claim 1, further comprising means for generating a dedicated chat room in the case of chat support, for exchanging messages between the user and the staff member in real time.

[1908] (Claim 3)

[1909] 10. The system of claim 1, further comprising means for generating an avatar in the case of avatar video and audio support and executing a scenario in which the avatar responds to the user with audio and video.

[1910] "Application example 2 when combining emotion engines"

[1911] (Claim 1)

[1912] an account authentication means for users to access the online platform and perform login authentication;

[1913] A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user;

[1914] means for generating a chat interface and corresponding resources for exchanging messages in real time;

[1915] A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests;

[1916] A means of keeping a log of all conversations and interactions;

[1917] emotion analysis means for analyzing emotions from a user's text message and providing feedback;

[1918] A means for adjusting chat resources and avatar responses based on the results of sentiment analysis;

[1919] A system including:

[1920] (Claim 2)

[1921] 10. The system according to claim 1, further comprising means for generating a dedicated chat room in the case of chat support, for e...

Claims

1. an account authentication means for users to access the online platform and perform login authentication; A service selection means for selecting chat support or video and audio support using an avatar, which can be selected by the user; means for generating a chat interface and corresponding resources for exchanging messages in real time; A data analysis and plan generation means for proposing an optimal plan based on the user's needs and requests; A means of keeping a log of all conversations and interactions; A system including:

2. 2. The system according to claim 1, further comprising means for generating a dedicated chat room in case of chat support, and exchanging messages between the user and the staff member in charge in real time.

3. 2. The system according to claim 1, further comprising means for generating an avatar in the case of avatar-based video and audio support, and executing a scenario in which the avatar responds to the user with audio and video.

Images