System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input consultation contents using a terminal; means for a server to analyze the consultation contents of the user using a natural language processing technique and extract important keywords and phrases; means for the server to arrange provisional arrangement information of the consultation contents according to a format and present the information to the terminal; means for the server to ask additional questions and arrange final information to generate a consultation contents sheet; and means for the server to store the final sheet in a database and notify a responder.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When waiting at stores, government offices, call centers, etc., customers tend to become emotional, and counter staff often feel stressed when dealing with them for the first time. Furthermore, if the initial interview is insufficient, it can take longer to resolve the problem, leading to lower customer satisfaction. To address these issues, there is a need for a system that can efficiently listen to the content of the consultation, organize it, and convey it to staff. [Means for solving the problem]

[0005] The system includes a means for a user to input the content of the consultation using a terminal, a means for a server to analyze the content of the user's consultation using natural language processing technology and extract important keywords and phrases, a means for the server to organize the provisional information of the consultation according to a format and present it on the terminal, a means for the server to ask additional questions, organize the final information and generate a consultation content sheet, and a means for the server to save the final sheet in a database and notify the person in charge, thereby making it possible to efficiently grasp the content of the user's consultation, reduce the stress of the person in charge when dealing with the situation for the first time, and improve work efficiency and customer satisfaction.

[0006] A "user" is someone who uses the system to input the details of a consultation and receive services.

[0007] A "terminal" is an electronic device used by a user that provides input and display means.

[0008] A "server" is a computer system that receives data sent by users and analyzes, processes, stores, and transmits the data to other devices.

[0009] "Natural language processing technology" is a technology for analyzing sentences entered by users, understanding their meaning, and extracting necessary information.

[0010] "Keywords" are words or phrases that are important for accurately understanding the content of the user's inquiry.

[0011] A "format" is a predetermined structure or arrangement for organizing and presenting information heard.

[0012] A "sheet" is a document format for organizing and recording the contents of a consultation, and is prepared so that the person providing the consultation can easily understand it.

[0013] "Database" means a system that stores information that has been heard and processed, and that can be searched and used as needed.

[0014] The "responder" is a staff member who checks the content of the user's consultation and provides a specific response.

[0015] "Notification" is a means for the server to notify the responder that a new consultation has been registered. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the contents of users' inquiries, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[0038] Overall system configuration

[0039] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, and a database that ultimately stores the data. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content sent from the device and analyzes it using natural language processing technology.

[0040] Program processing

[0041] 1. Starting an interactive session

[0042] A user starts a session with a conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends an initial message to the device.

[0043] 2. Hearing Phase

[0044] The user inputs the details of their consultation through their device. This input is then sent from the device to the server, which then analyzes the received user input and extracts important keywords and phrases.

[0045] 3. Preliminary organization of information

[0046] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[0047] 4. Confirmation and additional hearings

[0048] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[0049] 5. Creating a sheet

[0050] The server generates a sheet according to the format based on the final consultation content, and presents this sheet to the user for final confirmation.

[0051] 6. Check and correct

[0052] The user checks the sheet contents and makes any necessary corrections. Once the final sheet is confirmed, it is sent to the server via the terminal.

[0053] 7. Data Retention and Notification

[0054] The server saves the final generated sheet in the database and notifies the staff that a new consultation has been registered.

[0055] Specific examples

[0056] Scenario: Consultation about changing mobile phone plan

[0057] 1. Starting an interactive session

[0058] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0059] 2. Hearing Phase

[0060] The user inputs "I would like to know about mobile phone plans." The device sends this input to the server.

[0061] 3. Preliminary organization of information

[0062] The server extracts the keywords "price plan" and "what would you like to know" and tentatively organizes the consultation content.

[0063] 4. Confirmation and additional hearings

[0064] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[0065] 5. Final information organization

[0066] The server analyzes the additional information and organizes it according to the format: "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0067] 6. Creating a sheet

[0068] The server generates a sheet based on this information and prompts the user to confirm it.

[0069] 7. Check and correct

[0070] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[0071] 8. Data Retention and Notification

[0072] The finalized sheet is saved in the database and staff are notified of the new consultation details.

[0073] This allows users to smoothly communicate their concerns to staff members even when they meet for the first time, enabling staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[0077] Step 2:

[0078] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[0079] Step 3:

[0080] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[0081] Step 4:

[0082] The terminal transmits the user's input to the server.

[0083] Step 5:

[0084] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases, such as "price plan" and "please tell me."

[0085] Step 6:

[0086] The server then uses the extracted keywords to organize the provisional information about the consultation according to a format, such as "Consultation topic: Price plan" and "Specific content: Please let me know."

[0087] Step 7:

[0088] The server sends the preliminary information to the terminal and presents it to the user. The server also asks follow-up questions, such as "Which part specifically would you like to know about?"

[0089] Step 8:

[0090] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[0091] Step 9:

[0092] The terminal sends the user's additional input to the server.

[0093] Step 10:

[0094] The server analyzes the received additional information and organizes it into a format as the final consultation content, for example, "Consultation topic: Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[0095] Step 11:

[0096] The server generates a sheet in a predetermined format based on the final consultation content.

[0097] Step 12:

[0098] The server sends the generated sheet to the terminal for the user to check.

[0099] Step 13:

[0100] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[0101] Step 14:

[0102] The terminal sends the modifications to the server.

[0103] Step 15:

[0104] The server re-parses the modifications and regenerates the sheet.

[0105] Step 16:

[0106] The server saves the final sheet in the database and notifies the respondent that a new consultation has been registered.

[0107] Step 17:

[0108] The responder uses a dedicated terminal to check the pre-generated sheet and prepares to respond efficiently when the user visits.

[0109] This allows the user to smoothly communicate the content of the consultation, and the person in charge can also respond efficiently.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] In modern society, making effective use of waiting time at stores, government offices, call centers, etc., and efficiently listening to and organizing user inquiries is essential to improving business efficiency and customer satisfaction. However, there is a lack of appropriate systems to achieve this, and most responses are handled manually, resulting in a waste of time and effort. Therefore, there is a need for a system that can quickly and accurately receive, analyze, organize, and ultimately provide user inquiries to the person in charge.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for a user to input consultation details using an information processing device, means for a computer to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases, means for the computer to organize provisionally organized information of the consultation details according to a format and present it to the information processing device, means for the computer to generate an initial message and send it to the information processing device, means for the computer to ask additional questions, organize the final information, and generate a consultation details sheet, and means for the computer to save the final sheet in a database and notify the service provider. This makes it possible to efficiently hear the user's consultation details, organize them, and provide them to the person who will handle the consultation.

[0115] "User" refers to an individual or corporation who uses the service and inputs the details of their inquiry.

[0116] An "information processing device" is a device used by a user to input consultation details, and specifically refers to a smartphone, PC, tablet, etc.

[0117] "Server" refers to a computer system that receives data sent by users and analyzes, processes, stores, and notifies them.

[0118] "Natural language processing technology" refers to technology for analyzing user input text and extracting important keywords and phrases.

[0119] "Provisionally organized information" refers to information that temporarily organizes consultation content based on keywords and phrases extracted using natural language processing technology.

[0120] "Format" refers to a standard format or structure for organizing consultation content in a consistent manner.

[0121] "Initial message" refers to the first message that the server sends to the user at the beginning of a session.

[0122] "Sheet" refers to a document or data format that contains the final, organized content of the consultation.

[0123] "Database" refers to a data management system for storing the final generated sheets and other related data.

[0124] "Service provider" refers to an individual or organization that responds to and provides support based on the content of inquiries from users.

[0125] This invention relates to a system that effectively utilizes waiting time at stores, government offices, call centers, etc., and efficiently listens to and organizes the details of users' inquiries and provides them to staff. This system consists of an information processing device used by users, a server that receives and analyzes the details of the inquiries, and a database that stores the data.

[0126] A user initiates a session with a conversational AI using an information processing device such as a smartphone or PC. The server receives the user's consultation content sent from the terminal and analyzes it using natural language processing technology (e.g., Google Cloud NLP API). Specific embodiments for implementing this invention are described below.

[0127] When the server receives the consultation content entered by the user, it uses natural language processing technology to extract important keywords and phrases. This allows the server to accurately understand the user's intent and the content of the question. The extracted keywords and phrases are used to tentatively organize the consultation content.

[0128] The provisionally organized information is then organized into a standard format by the server and presented to the user via their terminal. This allows the user to easily confirm the content of their consultation. The server also asks additional questions at this stage to obtain more detailed information.

[0129] The information obtained from the additional interviews is analyzed in the same way and organized into a format as the final consultation content. The server generates a sheet based on this final information and asks the user to confirm it again. The user checks the sheet contents and corrects them as necessary to confirm the exact consultation content.

[0130] The server stores the confirmed consultation details in a database and sends a notification to the staff, enabling them to respond promptly to the newly registered consultation details.

[0131] Specific examples

[0132] Scenario: Consultation about changing mobile phone plan

[0133] 1. The user launches the smartphone app and taps the "Price Plan Consultation" button. The app then displays the message, "Please tell us your inquiry."

[0134] 2. The terminal sends a session initiation request to the server and sends the user ID.

[0135] 3. The server starts the session and sends an initial message to the terminal, for example, "Please tell us what you would like to discuss with us."

[0136] 4. The user types, "I'd like to know about mobile phone plans."

[0137] 5. The device sends the input information to the server.

[0138] 6. The server analyzes the received message using natural language processing technology and extracts important keywords such as "price plan" and "please tell me."

[0139] 7. The server generates provisional information such as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan" and sends it to the terminal.

[0140] 8. The terminal displays the provisional sorting information to the user.

[0141] 9. The user adds, "I want to know how to change from a smartphone to a tablet."

[0142] 10. The device sends the additional input to the server.

[0143] 11. The server analyzes again and generates "Consultation topic: Pricing plan, Details: How to change from smartphone to tablet."

[0144] 12. The server generates a sheet based on the final consultation content and sends it to the terminal.

[0145] 13. The terminal displays the sheet to the user, who then checks and modifies it.

[0146] 14. The user amends the question to say, "I want to change the Wi-Fi in addition to the tablet."

[0147] 15. The device sends the modified content to the server.

[0148] 16. The server generates a new sheet reflecting the changes and sends it to the terminal.

[0149] 17. The terminal displays the sheet again to the user for final confirmation.

[0150] 18. The server saves the confirmed sheet in the database and notifies the staff that a new consultation has been registered.

[0151] Prompt Sentence Examples

[0152] User prompt: "I want to change my mobile phone plan. How do I do this?"

[0153] Server's initial response: "Please tell us what you would like to discuss."

[0154] Additional prompt: "I want to know how to change from a smartphone to a tablet."

[0155] In this way, users can smoothly communicate their concerns to staff members they meet for the first time, enabling the staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Step 1:

[0158] Starting an interactive session

[0159] The user uses an information processing device (smartphone or PC) to start a session with the conversational AI.

[0160] Input: User clicks a "Start Session" button on an application or web page.

[0161] The terminal sends a session initiation request to the server, along with data including the user ID and a timestamp of the session initiation.

[0162] The server receives the request and performs processing to initialize the session, such as generating a new session ID and performing initial settings.

[0163] Output: The server generates an initial message and sends it to the user. The initial message, such as "Please tell us what you would like to discuss," is displayed on the terminal.

[0164] Step 2:

[0165] Hearing phase

[0166] The user inputs the content of their inquiry into the terminal. For example, they might input, "I'd like to know about mobile phone rate plans."

[0167] Input: A user enters text into an input field on an application.

[0168] The terminal sends the input text to the server.

[0169] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it uses the Google Cloud NLP API to extract important keywords from the text (e.g., "pricing plan" or "please tell me").

[0170] Output: Extracted keywords and analysis results.

[0171] Step 3:

[0172] Provisional organization of information

[0173] The server then organizes the consultation content into a temporary format based on the extracted keywords. For example, it could organize the content as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan."

[0174] Input: Keywords extracted by natural language processing and analysis results.

[0175] The server generates a message to present the provisional arrangement information to the user.

[0176] Output: Preliminary organized consultation message.

[0177] Step 4:

[0178] Confirmation and additional hearings

[0179] The user checks the provisional information provided by the server and enters additional information. For example, the user might enter, "I want to know how to change from a smartphone to a tablet."

[0180] Input: The user enters additional information and the device sends it to the server.

[0181] The server then analyzes the received data using natural language processing technology and formats the final consultation content based on the additional information.

[0182] Output: Information organized as the final consultation content.

[0183] Step 5:

[0184] Sheet Generation

[0185] The server then generates a sheet based on the final organized consultation content. For example, it creates a sheet with the following information: "Consultation topic: Pricing plan, Details: How to switch from smartphone to tablet."

[0186] Input: The final consultation details.

[0187] The server sends the generated sheet to the terminal for presentation to the user.

[0188] Output: The generated sheet.

[0189] Step 6:

[0190] Check and fix

[0191] The user checks the sheet displayed on the device and makes any necessary changes. For example, they might change the settings to "I want to change the Wi-Fi in addition to the tablet."

[0192] Input: The user inputs the correction information, and the terminal sends it to the server.

[0193] The server receives the revised content and updates the sheet again to generate the final version.

[0194] Output: The final sheet with the modifications reflected.

[0195] Step 7:

[0196] Data Retention and Notification

[0197] The server stores the final sheet in a database, including the consultation content, user identification information, time information, etc.

[0198] Input: Final sheet and associated metadata.

[0199] The server notifies the service provider that a new consultation has been registered. Notification methods include email and push notification.

[0200] Output: Information stored in the database and notification messages.

[0201] By this processing flow, the contents of the user's inquiry are efficiently and accurately heard and provided to the person who will ultimately handle the matter.

[0202] (Application example 1)

[0203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0204] In physical stores, there is a demand for a system that allows users to make effective use of their waiting time and efficiently consult with staff about product information and order details. In particular, it is necessary to improve in-store service by quickly and accurately conveying the content of users' inquiries to staff. There is also a demand for a means for users to smoothly convey the content of their inquiries to staff members they meet for the first time. Furthermore, there is a demand for improving store operational efficiency by using conversational AI to automatically organize the content of inquiries and respond in a timely manner.

[0205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0206] In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize the provisional consultation details according to a format and present it on the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for a user to input product information and order details using a smartphone at a physical store while waiting; a means for the server to analyze the user's input in real time using an interactive AI and generate a provisional format; a means for the server to reflect the user's confirmation and corrections and regenerate the final format; and a means for the server to notify staff at the physical store based on the final format. This allows users to effectively utilize their waiting time at the physical store and efficiently consult about product information and order details. Furthermore, the server automatically organizes the consultation details and quickly and accurately conveys them to staff, thereby improving in-store service and operational efficiency.

[0207] "User" refers to a person who uses the system to input the details of a consultation and receive services.

[0208] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0209] "Consultation content" refers to information related to in-store services, such as product information and order details entered by the user.

[0210] "Server" refers to a computer system that receives and analyzes input from users and notifies staff.

[0211] "Natural language processing technology" refers to the technology of analyzing text data entered by the user and extracting important keywords and phrases.

[0212] "Provisional sorting information" refers to information that sorts out the consultation content input by the user at an early stage.

[0213] "Format" refers to a structure or template for organizing consultation content in a certain format.

[0214] "Conversational AI" refers to an artificial intelligence system that engages in natural dialogue with users and collects and analyzes information.

[0215] "Waiting time" refers to the time a user waits to receive service at a physical store.

[0216] "Notification" refers to the action of informing staff of server-generated information.

[0217] "Physical store" refers to a store or service location that has a physical presence.

[0218] The "provisional format" refers to a format that is temporarily organized at the stage of initial analysis of the content of the user's inquiry.

[0219] The "final format" refers to the format of the consultation content that has been confirmed after being checked and corrected by the user.

[0220] The term "responder" refers to a store staff member who responds to the user upon receiving a notification from the server.

[0221] The present invention provides a system that enables users to effectively utilize their waiting time in a physical store and consult with them about product information and order details. Specific embodiments for carrying out the invention are described below.

[0222] Overall system configuration

[0223] The system mainly consists of the following components:

[0224] 1. Device: The device used by the user, such as a smartphone or tablet.

[0225] 2. Server: A computer system that is responsible for receiving user input, analyzing it, and notifying staff.

[0226] 3. Database: This is where the final consultation sheet is stored.

[0227] Hardware and software used

[0228] Hardware: Smartphones, tablets, server computers

[0229] Software: Python, Flask framework, SQLite database, OpenAI API

[0230] Explanation of program processing

[0231] Starting an interactive session

[0232] The user starts the smartphone app in the waiting area of ​​a physical store and starts a session with the conversational AI. The server then sends an initial message saying, "Please tell us what you would like to discuss," and the session begins.

[0233] Hearing phase

[0234] The user inputs the details of their consultation via their device. This input is then sent from the device to the server, which then analyzes the received user input and uses natural language processing technology to extract important keywords and phrases.

[0235] Generation of provisional arrangement information

[0236] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[0237] Confirmation and additional hearings

[0238] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[0239] Generating the final format

[0240] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet contents and makes corrections as necessary. Once the final sheet is confirmed, it is sent to the server via the terminal.

[0241] Data Retention and Notification

[0242] The server then saves the final generated sheet in a database. It also notifies the responder that a new consultation has been registered. This allows the staff member to understand the specific details of the user's consultation in advance, enabling them to respond quickly and appropriately.

[0243] Specific examples

[0244] For example, if a user types "Tell me about new smartphone models," the following prompt sentence is used:

[0245] Prompt Sentence Examples

[0246] Please organize the following information: I would like to know about the new smartphone model.

[0247] The server uses this prompt to call the generative AI model, extracting the keywords "new model" and "please tell me," and organizing them into a temporary format. This information is then presented to the user, who is then asked to enter additional details. Through this process, the final format is generated, and the user's inquiry is notified to the store staff.

[0248] According to the above embodiment, users can effectively use their waiting time to input the details of their consultation, and staff can efficiently respond to users' consultations.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] Starting an interactive session

[0252] The user starts a session with the conversational AI using a device. When the user launches the smartphone app, a request to start the session is sent from the device to the server. The server starts the session and sends the initial message "Please tell us your consultation." to the device.

[0253] Input: User's session initiation request

[0254] Output: Initial message from the server

[0255] Step 2:

[0256] Hearing phase

[0257] The user inputs their concerns through their device. This input is sent from the device to the server, which then receives it. The server uses a generative AI model and natural language processing technology to analyze the user's input and extract important keywords and phrases.

[0258] Input: User's consultation content input

[0259] Output: Extracted keywords and phrases

[0260] Step 3:

[0261] Generation of provisional arrangement information

[0262] The server organizes the consultation content into a provisional format based on the extracted keywords and phrases. This provisionally organized information is then presented to the user via their terminal.

[0263] Input: Extracted keywords or phrases

[0264] Output: Provisional arrangement information

[0265] Step 4:

[0266] Confirmation and additional hearings

[0267] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters specific additional information, which is then sent back to the server from the terminal. The server analyzes the additional information and organizes it into a format as the final consultation content.

[0268] Input: Additional hearing information for the user

[0269] Output: The final parsed consultation

[0270] Step 5:

[0271] Generating the final format

[0272] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet content and makes corrections as necessary.

[0273] Input: The final parsed consultation content

[0274] Output: The final sheet presented to the user

[0275] Step 6:

[0276] Check and fix

[0277] The user checks the sheet contents and makes any necessary corrections. The corrected contents are sent back to the server via the terminal, and the server regenerates the final sheet based on the updated information.

[0278] Input: User-modified information

[0279] Output: Updated final sheet

[0280] Step 7:

[0281] Data Retention and Notification

[0282] The server saves the final generated sheet in a database and notifies the responder that a new consultation has been registered, allowing the staff member to understand the specific details of the user's consultation in advance.

[0283] Input: Updated final sheet

[0284] Output: Sheets saved in the database and notifications to responders

[0285] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0286] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the user's consultation details and feelings, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[0287] Overall system configuration

[0288] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, a database that ultimately stores the data, and an emotion engine that recognizes the user's emotions. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device and analyzes each.

[0289] Program processing

[0290] 1. Starting an interactive session

[0291] The user starts a session with the conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends the initial message "Please tell us your consultation details" to the device.

[0292] 2. Hearing Phase

[0293] The user inputs the content of their inquiry through the device. For example, they might input, "I'd like to know about my mobile phone plan." The device then sends the user's input to the server. The emotion engine also analyzes the emotional data of the user's input.

[0294] 3. Preliminary organization of information

[0295] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases. For example, keywords such as "price plan" and "please tell me." The emotion engine recognizes the user's emotions (e.g., joy, anger, sadness, etc.) and sends that information to the server.

[0296] 4. Confirmation and additional hearings

[0297] The server takes into account the extracted keywords and the user's emotional information, and organizes the provisionally organized information in a format. This provisionally organized information is then presented to the user again via the terminal. Depending on the user's emotions, the server may send additional messages, such as a message to help them relax. The server may also ask additional questions, such as, "Which part specifically would you like to know more about?"

[0298] 5. Emotion-Based Regulation

[0299] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[0300] 6. Final information organization

[0301] The server then organizes the final consultation content into a format based on the additional information and emotion data received again. For example, it could organize the content into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0302] 7. Creating a sheet

[0303] The server generates a sheet in a predetermined format based on the final consultation content, which also includes the user's emotional data.

[0304] 8. Check and correct

[0305] The server sends the generated sheet to the terminal for the user to check. The user checks the sheet contents and makes corrections as necessary. These corrections are sent back to the server, and the sheet is regenerated.

[0306] 9. Data Retention and Notification

[0307] The server saves the final generated sheet in a database. It also notifies staff that a new consultation has been registered. Staff use dedicated terminals to check the pre-generated sheet and prepare to respond efficiently when the user visits. When responding, the server also takes into account the user's emotional information and provides an appropriate response.

[0308] Specific examples

[0309] Scenario: Consultation about changing mobile phone plan

[0310] 1. Starting an interactive session

[0311] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0312] 2. Hearing Phase

[0313] The user types, "Please tell me about my mobile phone plan." The device sends this input to the server. The emotion engine then analyzes the emotion (e.g., nervousness, interest, etc.) based on the user's input.

[0314] 3. Preliminary organization of information

[0315] The server extracts keywords such as "price plan" and "what would you like to know" and tentatively organizes the content of the consultation. It also incorporates emotional information obtained from the emotion engine.

[0316] 4. Confirmation and additional hearings

[0317] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[0318] 5. Emotion-Based Regulation

[0319] If the emotion engine detects that the user's emotions are a little unstable, the server will follow up with, "What are you worried about? Please tell us more."

[0320] 6. Final information organization

[0321] Based on the additional information and emotion data, the server updates the format and organizes it as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0322] 7. Creating a sheet

[0323] The server generates a sheet based on this information and prompts the user to confirm it.

[0324] 8. Check and correct

[0325] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[0326] 9. Data Retention and Notification

[0327] The finalized sheet is saved in a database, and the person in charge is notified of the new consultation content and emotional data. Staff members check the information in advance and prepare to respond appropriately when the patient visits.

[0328] This format allows users to smoothly communicate their concerns to staff members even when they meet for the first time, and allows staff members to respond efficiently while taking into consideration the user's feelings. The data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0329] The processing flow will be explained below.

[0330] Step 1:

[0331] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[0332] Step 2:

[0333] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[0334] Step 3:

[0335] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[0336] Step 4:

[0337] The terminal transmits the user's input to the server, and at the same time, the emotion engine collects the user's emotion data (voice tone, input speed, phrase selection, etc.) and transmits it to the server.

[0338] Step 5:

[0339] The server analyzes the received user input and uses natural language processing techniques to extract important keywords and phrases.

[0340] Step 6:

[0341] The emotion engine analyzes the received emotion data and identifies the user's emotional state (e.g., joy, anxiety, anger, etc.). Based on this information, the server tentatively organizes the user's consultation details.

[0342] Step 7:

[0343] The server organizes the provisionally organized information based on the extracted keywords and emotion data and displays it on the terminal. For example, the information might be "Consultation topic: Price plan," "Specific content: Please tell me," and "Emotional state: Anxiety."

[0344] Step 8:

[0345] The server can ask additional questions or make additional considerations based on the user's emotional state, for example, "If you would like to know more about our pricing plans, we can explain them in detail. Please let us know if you have any concerns."

[0346] Step 9:

[0347] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[0348] Step 10:

[0349] The terminal sends additional user input to the server, and the emotion engine collects new emotion data and sends it to the server.

[0350] Step 11:

[0351] The server analyzes the received additional information and formats it into the final consultation content. The emotion engine analyzes the user's emotional state again and includes the emotion data in the final sheet.

[0352] Step 12:

[0353] The server generates a final consultation content sheet and sends it to the terminal for the user to confirm. The sheet contains, for example, the following information:

[0354] Consultation topic: Pricing plan

[0355] Specific content: How to change from smartphone to tablet

[0356] Emotional state: Anxiety

[0357] Step 13:

[0358] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[0359] Step 14:

[0360] The terminal transmits the corrections to the server, and the emotion engine collects the user's emotion data again.

[0361] Step 15:

[0362] The server analyzes the modifications and regenerates the sheet, including the final emotion data from the emotion engine.

[0363] Step 16:

[0364] The server saves the finalized sheet in the database and notifies the person in charge that the new consultation content and emotional data have been registered.

[0365] Step 17:

[0366] The responder uses a dedicated terminal to check the pre-generated sheet and emotion data, and prepares to respond efficiently and appropriately when the user visits.

[0367] Example 2

[0368] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] In recent years, there has been a demand for stores, government offices, call centers, and other facilities to make effective use of users' waiting time and efficiently handle inquiries. However, current systems have difficulty grasping users' emotions and the details of their inquiries, which can lead to a decline in user satisfaction. Furthermore, they are unable to respond in a way that takes users' emotions into appropriate consideration, which can result in problems and dissatisfaction. It is necessary to solve these problems and provide a system that provides high satisfaction to both users and staff.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0371] In this invention, the server includes: means for a user to use a terminal to start a session with an interactive artificial intelligence; means for the server to receive the user's consultation content, analyze it using natural language processing technology, and extract important keywords and phrases; means for the server to analyze the user's emotional data using an emotion recognition engine and acquire emotional information; means for the server to create provisional organized information based on the extracted keywords and emotional information, organize it according to a format, and present it to the terminal; means for the server to ask the user additional questions, organize the final information, and generate a consultation content sheet; means for the server to store the final generated sheet in a database and notify the person handling the consultation; and means for the person handling the consultation to respond efficiently while taking the user's emotional information into consideration. This makes it possible to understand the user's emotions and detailed consultation content, and respond accurately and efficiently.

[0372] A "terminal" is an electronic device used by a user, which is a means for initiating a session with an interactive artificial intelligence and inputting and displaying consultation details.

[0373] "Conversational artificial intelligence" refers to an algorithm or system that allows a user to interact with it in natural language, analyzing input from the user and generating appropriate responses.

[0374] A "server" is a computer system that receives, processes, and analyzes data sent from terminals via a network.

[0375] "Natural language processing technology" is a general term for technology that analyzes, understands, and generates human language, and is used to extract important keywords and phrases.

[0376] An "emotion recognition engine" is software or an algorithm that analyzes emotions from user input data and obtains that information.

[0377] "Temporarily organized information" is information that has been temporarily organized based on keywords and emotional information extracted from the contents of the user's consultation.

[0378] A "format" is a standardized structure or form for organizing and presenting information.

[0379] A "consultation sheet" is a written or electronic form containing the final organized consultation and emotional information.

[0380] A "database" is a system for efficiently storing, managing, and searching accumulated data.

[0381] The "responder" is a person in charge of responding to the content of the inquiry from the user, and is a person who receives a notification from the server and takes an appropriate action.

[0382] This invention is a system in which a user inputs the details of a consultation using a terminal, a server analyzes and organizes the input, and generates a final consultation details sheet to notify the person handling the consultation. Specific embodiments for implementing this system are described below.

[0383] Overall system configuration

[0384] The system consists of a device used by the user, a server, a database, and an emotion recognition engine. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device, analyzes them, and organizes the user's consultation content.

[0385] System hardware and software examples

[0386] Device: A user device such as a smartphone, tablet, or PC.

[0387] Server: A high-performance computer system (e.g., a cloud server such as AWS or Azure).

[0388] Database: A SQL or NoSQL database (e.g. MySQL, MongoDB).

[0389] Emotion recognition engine: Software that analyzes emotions (e.g., Microsoft Azure Cognitive Services, IBM Watson, etc.).

[0390] Natural Language Processing (NLP): Technology for analyzing text (e.g., Google Natural Language API, spaCy).

[0391] Explanation of program processing

[0392] 1. Starting an interactive session

[0393] The user starts a session with the conversational AI using a terminal. The terminal sends a session start request to the server, and the server starts the session. The server sends the initial message "Please tell us your consultation details" to the terminal and displays it to the user.

[0394] 2. Hearing Phase

[0395] The user inputs the content of their inquiry through the device. For example, if they input "I'd like to know about my mobile phone plan," the device sends this input to the server. At the same time, the emotion recognition engine analyzes the user's emotional data.

[0396] 3. Preliminary organization of information

[0397] The server analyzes the received consultation content using natural language processing (NLP) technology to extract important keywords and phrases. It also obtains emotional data analyzed by an emotion recognition engine and creates provisional sorting information.

[0398] 4. Confirmation and additional hearings

[0399] The server then verifies the user based on the provisionally organized information and asks additional questions, such as, "Which part specifically would you like to know about?" The server then reorganizes the information based on the user's response.

[0400] 5. Emotion-Based Regulation

[0401] The server takes into account the user's emotional data and adjusts the content and order of follow-up questions, such as "What are you worried about?"

[0402] 6. Final information organization

[0403] The server then organizes the consultation details based on the information it finally obtains and writes the final information in a format, such as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0404] 7. Creating a sheet

[0405] The server then generates a sheet based on the final organized information, which also includes the user's emotional data.

[0406] 8. Check and correct

[0407] The sheet is sent to the terminal, where the user can check the contents and make corrections as necessary. The corrections are then sent back to the server, and the sheet is updated.

[0408] 9. Data Retention and Notification

[0409] The confirmed sheet is saved in the database, and the person in charge is notified of the new consultation content. The person in charge checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[0410] Examples and prompts

[0411] Scenario: Consultation about changing mobile phone plan

[0412] 1. Starting an interactive session

[0413] A user launches the app on their smartphone and starts chatting with a conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0414] 2. Hearing Phase

[0415] The user types "I'd like to know about mobile phone plans," and the device sends this information to the server. At the same time, the emotion recognition engine analyzes the user's emotions.

[0416] 3. Preliminary organization of information

[0417] The server extracts keywords such as "price plan" and "please tell me" and creates provisional sorting information along with emotional data.

[0418] 4. Confirmation and additional hearings

[0419] Based on the provisionally organized information, the server asks the user, "Which part specifically would you like to know?" The user enters, "I would like to know how to change from a smartphone to a tablet," and the device resends the request to the server.

[0420] 5. Emotion-Based Regulation

[0421] If the emotion recognition engine detects the user's anxiety, the server asks a follow-up question: "What are you worried about?"

[0422] 6. Final information organization

[0423] Based on the additional information and emotional data, the server organizes the final information into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0424] Example of input prompt for generative AI model

[0425] The functionality of this system can be simulated concretely by inputting the following prompt sentences into the generative AI model:

[0426] A user opens a smartphone app and types, "I'd like to know about my mobile phone plan." The system sends the initial message to the user, and the emotion engine analyzes the user's emotions. What action will the system take next?

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Program processing flow

[0429] Step 1:

[0430] A user starts an application on a device such as a smartphone or PC and starts a session with the conversational AI. The device sends a session start request to the server. The server receives the request, prepares to start the session, and generates an initial message, "Please tell us your consultation details," and sends it to the device. The device receives this message and displays it to the user.

[0431] Step 2:

[0432] The user inputs the content of their inquiry into an input field on the device. For example, they might input "Please tell me about my mobile phone plan." The device then sends the input content as text data to the server. The server receives this text data and transfers it to an emotion recognition engine. The server then begins to apply natural language processing (NLP) to the user's input content. The input data includes the user's text data and emotion data analyzed by the emotion recognition engine.

[0433] Step 3:

[0434] The server obtains emotional data (e.g., "anxiety," "interest") obtained by an emotion recognition engine, and important keywords and phrases (e.g., "price plan," "please tell me") extracted using natural language processing technology. Through this process, the server understands the user's basic inquiry content and emotional state regarding "price plan." For example, if the input data is "please tell me about mobile phone rate plans," and the emotional data is "anxiety," the keywords "price plan" and "anxiety" are extracted.

[0435] Step 4:

[0436] The server creates provisionally organized information based on the keywords and emotion information extracted by the server, and organizes it according to a format. This provisionally organized information is sent to the terminal for confirmation by the user, for example, as "You're asking about a pricing plan, right?" The terminal receives this information and displays it to the user. The user can confirm the content and enter more specific questions and answers.

[0437] Step 5:

[0438] The server receives the user's additional input and analyzes it again using natural language processing technology and an emotion recognition engine. For example, if the user inputs, "I want to know how to switch from a smartphone to a tablet," the server re-extracts the keywords "how to switch" and "I want to know" as well as the user's emotion data. Based on this additional information, the server creates more detailed provisionally organized information and sends it to the device.

[0439] Step 6:

[0440] The server adjusts the content of the answers and follow-up questions based on the user's emotional data. For example, if the user enters "I want to know how to switch from a smartphone to a tablet" and the emotional data indicates "anxiety," the server generates a follow-up message asking "What are you worried about?" and sends it to the device. The user then enters an additional answer and sends it again to the server.

[0441] Step 7:

[0442] The server organizes the final consultation content based on the additional information and emotional data. For example, it organizes the information according to a format such as "Consultation topic: Price plan," "Specific content: How to switch from a smartphone to a tablet," and "Emotional state: Anxiety." The server generates a sheet based on this information and sends it to the device.

[0443] Step 8:

[0444] The user can check the generated sheet on the terminal and make corrections as necessary. Once the corrections are sent to the server, the server updates the sheet again and sends it to the terminal.

[0445] Step 9:

[0446] The server saves the confirmed consultation content sheet in a database. It also notifies the person handling the consultation that a new consultation content has been registered. The person handling the consultation checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[0447] (Application example 2)

[0448] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0449] Conventional consultation systems systematically organize consultation content without considering the user's emotions, which results in an inability to fully grasp the user's true needs and concerns. Furthermore, in face-to-face consultations, it is difficult for staff to obtain emotional information in real time, which can prevent them from providing an appropriate response. In such situations, user satisfaction declines and efficient response becomes difficult.

[0450] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize provisional consultation details information according to a format and present it to the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for the terminal to start an interaction session and analyze the user's emotional data from the voice input in real time; a means for the smart glasses to recognize the customer's emotional state in real time and generate an appropriate response; a means for providing consultation response information to staff based on the final information; and a means for sharing information, such as emotional data, in real time so that staff can respond efficiently. This makes it possible to recognize emotions in real time through interaction with the user and provide appropriate responses.

[0451] The "means for a user to input the content of a consultation using a terminal" refers to an interface that allows a user to input the content of a consultation by text or voice using a mobile terminal, personal computer, or other device.

[0452] "Means for the server to use natural language processing technology to analyze the content of the user's inquiry and extract important keywords and phrases" refers to a technology in which the server uses natural language processing technology, a type of artificial intelligence technology, to analyze the content entered by the user and extract important information based on specific rules.

[0453] "Means for the server to organize provisional consultation information according to a format and present it on the terminal" refers to a technology in which the server provisionally organizes the consultation information based on keywords and phrases extracted by the server, compiles it according to a predetermined format, and displays the information on the user's terminal.

[0454] "Means for the server to ask additional questions, organize the final information, and generate a consultation content sheet" refers to a technology in which the server asks supplementary questions to the user, organizes the final consultation content based on the information obtained, formats the content, and generates it as a sheet.

[0455] "Means for the server to save the final sheet in a database and notify the responder" refers to a technology that saves the final consultation content sheet generated by the server in a database and notifies the staff member in charge of that information.

[0456] "Means for a terminal to initiate an interactive session and analyze user emotional data from voice input in real time" refers to a technology in which a user terminal initiates voice recognition and analyzes emotional data in real time based on information obtained from the user's voice input.

[0457] "A means for smart glasses to recognize a customer's emotional state in real time and generate an appropriate response" refers to a technology that uses an emotion recognition sensor built into smart glasses to analyze a customer's emotional state in real time from their facial expressions and voice, and generates an appropriate response based on that information.

[0458] "Means for providing staff with information on how to respond to inquiries based on final information" refers to a technique for providing staff with information that will enable them to respond appropriately to customers based on the final organized information on the content of the inquiries.

[0459] "Means for sharing information, such as emotional data, in real time to enable staff to respond efficiently" refers to technology that enables staff to share customer emotional data and other important information in real time, enabling efficient customer service.

[0460] The present invention relates to a system for efficiently responding to consultations while taking into consideration the user's emotions when making inquiries at stores, government offices, call centers, etc. Specific embodiments for carrying out the invention will be described below.

[0461] Overall system configuration

[0462] The system consists of the following main components:

[0463] 1. User terminal: A device such as a smartphone or personal computer (PC) that the user uses to input the details of their consultation.

[0464] 2. Server: Analyzes the user's inquiry using natural language processing technology, extracts important keywords and phrases, and organizes them.

[0465] 3. Database: Stores the final consultation sheet generated by the server.

[0466] 4. Emotion engine: Analyzes emotional data from the user's voice and facial expressions in real time.

[0467] 5. Smart Glasses: A device worn by staff that recognizes the emotional state of customers in real time and generates appropriate responses.

[0468] 6. Staff terminal: A device used by staff to check response information.

[0469] System Operation Overview

[0470] 1. Start an interactive session:

[0471] A user accesses the system using a smartphone or PC and starts a session with the conversational AI. The server receives the session start request and sends the initial message, "Please tell us your consultation details." to the user's device.

[0472] 2. Hearing Phase:

[0473] When the user inputs the content of their consultation, the device sends the input to the server. At the same time, the emotion engine analyzes the user's voice and facial expressions to obtain emotional data, and sends the results to the server.

[0474] 3. Preliminary organization of information:

[0475] The server analyzes the received user inquiry and uses natural language processing technology to extract important keywords and phrases. For example, it extracts keywords such as "price plan" and "please tell me." Based on this, it creates provisionally organized information and presents it to the user's device.

[0476] 4. Verification and Further Hearing:

[0477] The server then uses the preliminary information to ask additional questions, such as, "What specific part would you like to know about?" Any additional information entered by the user is analyzed in the same way.

[0478] 5. Emotion-based regulation:

[0479] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[0480] 6. Final information organization:

[0481] The server then organizes and formats the final consultation content based on the additional information and emotion data. For example, the consultation topic might be "Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[0482] 7. Generate and check the sheet:

[0483] The final organized information is generated as a correspondence sheet and presented to the user for confirmation. If the user enters any corrections, the information is reorganized and the sheet is updated.

[0484] 8. Data Retention and Notification:

[0485] The final sheet is saved in a database and sent to the customer service staff, who can then use the smart glasses to check the customer's emotional state in real time and respond accordingly.

[0486] Specific examples

[0487] As a concrete example, consider the following:

[0488] Scenario: Consultation about changing mobile phone plan

[0489] Example prompt:

[0490] "Please tell me about the administrative services available at the citizen service desk. Specifically, I would like information about tax consultations."

[0491] The system described above allows users to consult with confidence, and allows staff to take the user's feelings into consideration and respond more appropriately and quickly. Ultimately, the consultation content stored in the database will be used as learning data for future AI, which is expected to further improve the efficiency of problem-solving.

[0492] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0493] Step 1:

[0494] A user accesses the system using a smartphone or PC terminal and starts an interactive session. As input, the server receives a session start request from the user, and sends the initial message "Please tell us your consultation details" to the terminal. As output, the initial message is displayed on the user's terminal.

[0495] Step 2:

[0496] The user inputs the content of their consultation into their smartphone or PC terminal using voice or text. The consultation content data from the user is received as input and sent to the server. The consultation content data received by the server is passed to the natural language processing engine as output, and analysis begins.

[0497] Step 3:

[0498] The server uses natural language processing technology to analyze the user's consultation content and extract important keywords and phrases. The natural language processing engine analyzes the consultation content data received as input and extracts important information. The extracted keywords and phrases are saved as provisionally organized information as output.

[0499] Step 4:

[0500] The server uses an emotion engine to analyze emotion data from the user's voice and facial expressions in real time. The user's voice and facial expression data are sent to the emotion engine as input. Emotion data is generated as the analysis result as output and added to the provisional sorting information.

[0501] Step 5:

[0502] The server organizes the provisionally organized information according to a format and presents it to the user's terminal. As input, the provisionally organized information and emotion data are integrated within the server. As output, the formatted provisionally organized information is sent to the user's terminal and displayed to the user.

[0503] Step 6:

[0504] The server asks a follow-up question and receives the answer from the user again. As input, a follow-up question based on the provisional sorting information is sent to the user's terminal. The user enters the answer, and the data is sent to the server. As output, the follow-up information is saved on the server.

[0505] Step 7:

[0506] The server organizes and formats the final consultation content based on the additional information and emotion data. The additional information and the existing provisionally organized information are integrated as input. The final consultation content sheet is generated as output.

[0507] Step 8:

[0508] The server sends the generated final sheet to the user's terminal, where the user can review and modify it. As input, the final sheet is presented to the user's terminal. The user enters modifications, which are sent to the server. As output, the modified final sheet is regenerated.

[0509] Step 9:

[0510] The server saves the final consultation content sheet in the database and notifies the staff. As input, the final confirmation sheet is saved in the database. As output, the new consultation content is notified in real time to the staff terminal and smart glasses.

[0511] Step 10:

[0512] Staff use smart glasses to check and respond to customers' emotional states in real time. As input, the smart glasses receive information from the database and also obtain customer emotional data in real time. As output, an appropriate response is provided to the customer.

[0513] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0514] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0515] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0516] [Second embodiment]

[0517] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0518] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0519] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0520] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0521] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0522] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0523] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0524] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0525] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0526] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0527] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0528] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0529] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the contents of users' inquiries, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[0530] Overall system configuration

[0531] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, and a database that ultimately stores the data. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content sent from the device and analyzes it using natural language processing technology.

[0532] Program processing

[0533] 1. Starting an interactive session

[0534] A user starts a session with a conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends an initial message to the device.

[0535] 2. Hearing Phase

[0536] The user inputs the details of their consultation through their device. This input is then sent from the device to the server, which then analyzes the received user input and extracts important keywords and phrases.

[0537] 3. Preliminary organization of information

[0538] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[0539] 4. Confirmation and additional hearings

[0540] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[0541] 5. Creating a sheet

[0542] The server generates a sheet according to the format based on the final consultation content, and presents this sheet to the user for final confirmation.

[0543] 6. Check and correct

[0544] The user checks the sheet contents and makes any necessary corrections. Once the final sheet is confirmed, it is sent to the server via the terminal.

[0545] 7. Data Retention and Notification

[0546] The server saves the final generated sheet in the database and notifies the staff that a new consultation has been registered.

[0547] Specific examples

[0548] Scenario: Consultation about changing mobile phone plan

[0549] 1. Starting an interactive session

[0550] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0551] 2. Hearing Phase

[0552] The user inputs "I would like to know about mobile phone plans." The device sends this input to the server.

[0553] 3. Preliminary organization of information

[0554] The server extracts the keywords "price plan" and "what would you like to know" and tentatively organizes the consultation content.

[0555] 4. Confirmation and additional hearings

[0556] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[0557] 5. Final information organization

[0558] The server analyzes the additional information and organizes it according to the format: "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0559] 6. Creating a sheet

[0560] The server generates a sheet based on this information and prompts the user to confirm it.

[0561] 7. Check and correct

[0562] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[0563] 8. Data Retention and Notification

[0564] The finalized sheet is saved in the database and staff are notified of the new consultation details.

[0565] This allows users to smoothly communicate their concerns to staff members even when they meet for the first time, enabling staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0566] The processing flow will be explained below.

[0567] Step 1:

[0568] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[0569] Step 2:

[0570] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[0571] Step 3:

[0572] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[0573] Step 4:

[0574] The terminal transmits the user's input to the server.

[0575] Step 5:

[0576] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases, such as "price plan" and "please tell me."

[0577] Step 6:

[0578] The server then uses the extracted keywords to organize the provisional information about the consultation according to a format, such as "Consultation topic: Price plan" and "Specific content: Please let me know."

[0579] Step 7:

[0580] The server sends the preliminary information to the terminal and presents it to the user. The server also asks follow-up questions, such as "Which part specifically would you like to know about?"

[0581] Step 8:

[0582] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[0583] Step 9:

[0584] The terminal sends the user's additional input to the server.

[0585] Step 10:

[0586] The server analyzes the received additional information and organizes it into a format as the final consultation content, for example, "Consultation topic: Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[0587] Step 11:

[0588] The server generates a sheet in a predetermined format based on the final consultation content.

[0589] Step 12:

[0590] The server sends the generated sheet to the terminal for the user to check.

[0591] Step 13:

[0592] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[0593] Step 14:

[0594] The terminal sends the modifications to the server.

[0595] Step 15:

[0596] The server re-parses the modifications and regenerates the sheet.

[0597] Step 16:

[0598] The server saves the final sheet in the database and notifies the respondent that a new consultation has been registered.

[0599] Step 17:

[0600] The responder uses a dedicated terminal to check the pre-generated sheet and prepares to respond efficiently when the user visits.

[0601] This allows the user to smoothly communicate the content of the consultation, and the person in charge can also respond efficiently.

[0602] Example 1

[0603] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0604] In modern society, making effective use of waiting time at stores, government offices, call centers, etc., and efficiently listening to and organizing user inquiries is essential to improving business efficiency and customer satisfaction. However, there is a lack of appropriate systems to achieve this, and most responses are handled manually, resulting in a waste of time and effort. Therefore, there is a need for a system that can quickly and accurately receive, analyze, organize, and ultimately provide user inquiries to the person in charge.

[0605] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0606] In this invention, the server includes means for a user to input consultation details using an information processing device, means for a computer to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases, means for the computer to organize provisionally organized information of the consultation details according to a format and present it to the information processing device, means for the computer to generate an initial message and send it to the information processing device, means for the computer to ask additional questions, organize the final information, and generate a consultation details sheet, and means for the computer to save the final sheet in a database and notify the service provider. This makes it possible to efficiently hear the user's consultation details, organize them, and provide them to the person who will handle the consultation.

[0607] "User" refers to an individual or corporation who uses the service and inputs the details of their inquiry.

[0608] An "information processing device" is a device used by a user to input consultation details, and specifically refers to a smartphone, PC, tablet, etc.

[0609] "Server" refers to a computer system that receives data sent by users and analyzes, processes, stores, and notifies them.

[0610] "Natural language processing technology" refers to technology for analyzing user input text and extracting important keywords and phrases.

[0611] "Provisionally organized information" refers to information that temporarily organizes consultation content based on keywords and phrases extracted using natural language processing technology.

[0612] "Format" refers to a standard format or structure for organizing consultation content in a consistent manner.

[0613] "Initial message" refers to the first message that the server sends to the user at the beginning of a session.

[0614] "Sheet" refers to a document or data format that contains the final, organized content of the consultation.

[0615] "Database" refers to a data management system for storing the final generated sheets and other related data.

[0616] "Service provider" refers to an individual or organization that responds to and provides support based on the content of inquiries from users.

[0617] This invention relates to a system that effectively utilizes waiting time at stores, government offices, call centers, etc., and efficiently listens to and organizes the details of users' inquiries and provides them to staff. This system consists of an information processing device used by users, a server that receives and analyzes the details of the inquiries, and a database that stores the data.

[0618] A user initiates a session with a conversational AI using an information processing device such as a smartphone or PC. The server receives the user's consultation content sent from the terminal and analyzes it using natural language processing technology (e.g., Google Cloud NLP API). Specific embodiments for implementing this invention are described below.

[0619] When the server receives the consultation content entered by the user, it uses natural language processing technology to extract important keywords and phrases. This allows the server to accurately understand the user's intent and the content of the question. The extracted keywords and phrases are used to tentatively organize the consultation content.

[0620] The provisionally organized information is then organized into a standard format by the server and presented to the user via their terminal. This allows the user to easily confirm the content of their consultation. The server also asks additional questions at this stage to obtain more detailed information.

[0621] The information obtained from the additional interviews is analyzed in the same way and organized into a format as the final consultation content. The server generates a sheet based on this final information and asks the user to confirm it again. The user checks the sheet contents and corrects them as necessary to confirm the exact consultation content.

[0622] The server stores the confirmed consultation details in a database and sends a notification to the staff, enabling them to respond promptly to the newly registered consultation details.

[0623] Specific examples

[0624] Scenario: Consultation about changing mobile phone plan

[0625] 1. The user launches the smartphone app and taps the "Price Plan Consultation" button. The app then displays the message, "Please tell us your inquiry."

[0626] 2. The terminal sends a session initiation request to the server and sends the user ID.

[0627] 3. The server starts the session and sends an initial message to the terminal, for example, "Please tell us what you would like to discuss with us."

[0628] 4. The user types, "I'd like to know about mobile phone plans."

[0629] 5. The device sends the input information to the server.

[0630] 6. The server analyzes the received message using natural language processing technology and extracts important keywords such as "price plan" and "please tell me."

[0631] 7. The server generates provisional information such as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan" and sends it to the terminal.

[0632] 8. The terminal displays the provisional sorting information to the user.

[0633] 9. The user adds, "I want to know how to change from a smartphone to a tablet."

[0634] 10. The device sends the additional input to the server.

[0635] 11. The server analyzes again and generates "Consultation topic: Pricing plan, Details: How to change from smartphone to tablet."

[0636] 12. The server generates a sheet based on the final consultation content and sends it to the terminal.

[0637] 13. The terminal displays the sheet to the user, who then checks and modifies it.

[0638] 14. The user amends the question to say, "I want to change the Wi-Fi in addition to the tablet."

[0639] 15. The device sends the modified content to the server.

[0640] 16. The server generates a new sheet reflecting the changes and sends it to the terminal.

[0641] 17. The terminal displays the sheet again to the user for final confirmation.

[0642] 18. The server saves the confirmed sheet in the database and notifies the staff that a new consultation has been registered.

[0643] Prompt Sentence Examples

[0644] User prompt: "I want to change my mobile phone plan. How do I do this?"

[0645] Server's initial response: "Please tell us what you would like to discuss."

[0646] Additional prompt: "I want to know how to change from a smartphone to a tablet."

[0647] In this way, users can smoothly communicate their concerns to staff members they meet for the first time, enabling the staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0648] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0649] Step 1:

[0650] Starting an interactive session

[0651] The user uses an information processing device (smartphone or PC) to start a session with the conversational AI.

[0652] Input: User clicks a "Start Session" button on an application or web page.

[0653] The terminal sends a session initiation request to the server, along with data including the user ID and a timestamp of the session initiation.

[0654] The server receives the request and performs processing to initialize the session, such as generating a new session ID and performing initial settings.

[0655] Output: The server generates an initial message and sends it to the user. The initial message, such as "Please tell us what you would like to discuss," is displayed on the terminal.

[0656] Step 2:

[0657] Hearing phase

[0658] The user inputs the content of their inquiry into the terminal. For example, they might input, "I'd like to know about mobile phone rate plans."

[0659] Input: A user enters text into an input field on an application.

[0660] The terminal sends the input text to the server.

[0661] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it uses the Google Cloud NLP API to extract important keywords from the text (e.g., "pricing plan" or "please tell me").

[0662] Output: Extracted keywords and analysis results.

[0663] Step 3:

[0664] Provisional organization of information

[0665] The server then organizes the consultation content into a temporary format based on the extracted keywords. For example, it could organize the content as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan."

[0666] Input: Keywords extracted by natural language processing and analysis results.

[0667] The server generates a message to present the provisional arrangement information to the user.

[0668] Output: Preliminary organized consultation message.

[0669] Step 4:

[0670] Confirmation and additional hearings

[0671] The user checks the provisional information provided by the server and enters additional information. For example, the user might enter, "I want to know how to change from a smartphone to a tablet."

[0672] Input: The user enters additional information and the device sends it to the server.

[0673] The server then analyzes the received data using natural language processing technology and formats the final consultation content based on the additional information.

[0674] Output: Information organized as the final consultation content.

[0675] Step 5:

[0676] Sheet Generation

[0677] The server then generates a sheet based on the final organized consultation content. For example, it creates a sheet with the following information: "Consultation topic: Pricing plan, Details: How to switch from smartphone to tablet."

[0678] Input: The final consultation details.

[0679] The server sends the generated sheet to the terminal for presentation to the user.

[0680] Output: The generated sheet.

[0681] Step 6:

[0682] Check and fix

[0683] The user checks the sheet displayed on the device and makes any necessary changes. For example, they might change the settings to "I want to change the Wi-Fi in addition to the tablet."

[0684] Input: The user inputs the correction information, and the terminal sends it to the server.

[0685] The server receives the revised content and updates the sheet again to generate the final version.

[0686] Output: The final sheet with the modifications reflected.

[0687] Step 7:

[0688] Data Retention and Notification

[0689] The server stores the final sheet in a database, including the consultation content, user identification information, time information, etc.

[0690] Input: Final sheet and associated metadata.

[0691] The server notifies the service provider that a new consultation has been registered. Notification methods include email and push notification.

[0692] Output: Information stored in the database and notification messages.

[0693] By this processing flow, the contents of the user's inquiry are efficiently and accurately heard and provided to the person who will ultimately handle the matter.

[0694] (Application example 1)

[0695] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] In physical stores, there is a demand for a system that allows users to make effective use of their waiting time and efficiently consult with staff about product information and order details. In particular, it is necessary to improve in-store service by quickly and accurately conveying the content of users' inquiries to staff. There is also a demand for a means for users to smoothly convey the content of their inquiries to staff members they meet for the first time. Furthermore, there is a demand for improving store operational efficiency by using conversational AI to automatically organize the content of inquiries and respond in a timely manner.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0698] In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize the provisional consultation details according to a format and present it on the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for a user to input product information and order details using a smartphone at a physical store while waiting; a means for the server to analyze the user's input in real time using an interactive AI and generate a provisional format; a means for the server to reflect the user's confirmation and corrections and regenerate the final format; and a means for the server to notify staff at the physical store based on the final format. This allows users to effectively utilize their waiting time at the physical store and efficiently consult about product information and order details. Furthermore, the server automatically organizes the consultation details and quickly and accurately conveys them to staff, thereby improving in-store service and operational efficiency.

[0699] "User" refers to a person who uses the system to input the details of a consultation and receive services.

[0700] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0701] "Consultation content" refers to information related to in-store services, such as product information and order details entered by the user.

[0702] "Server" refers to a computer system that receives and analyzes input from users and notifies staff.

[0703] "Natural language processing technology" refers to the technology of analyzing text data entered by the user and extracting important keywords and phrases.

[0704] "Provisional sorting information" refers to information that sorts out the consultation content input by the user at an early stage.

[0705] "Format" refers to a structure or template for organizing consultation content in a certain format.

[0706] "Conversational AI" refers to an artificial intelligence system that engages in natural dialogue with users and collects and analyzes information.

[0707] "Waiting time" refers to the time a user waits to receive service at a physical store.

[0708] "Notification" refers to the action of informing staff of server-generated information.

[0709] "Physical store" refers to a store or service location that has a physical presence.

[0710] The "provisional format" refers to a format that is temporarily organized at the stage of initial analysis of the content of the user's inquiry.

[0711] The "final format" refers to the format of the consultation content that has been confirmed after being checked and corrected by the user.

[0712] The term "responder" refers to a store staff member who responds to the user upon receiving a notification from the server.

[0713] The present invention provides a system that enables users to effectively utilize their waiting time in a physical store and consult with them about product information and order details. Specific embodiments for carrying out the invention are described below.

[0714] Overall system configuration

[0715] The system mainly consists of the following components:

[0716] 1. Device: The device used by the user, such as a smartphone or tablet.

[0717] 2. Server: A computer system that is responsible for receiving user input, analyzing it, and notifying staff.

[0718] 3. Database: This is where the final consultation sheet is stored.

[0719] Hardware and software used

[0720] Hardware: Smartphones, tablets, server computers

[0721] Software: Python, Flask framework, SQLite database, OpenAI API

[0722] Explanation of program processing

[0723] Starting an interactive session

[0724] The user starts the smartphone app in the waiting area of ​​a physical store and starts a session with the conversational AI. The server then sends an initial message saying, "Please tell us what you would like to discuss," and the session begins.

[0725] Hearing phase

[0726] The user inputs the details of their consultation via their device. This input is then sent from the device to the server, which then analyzes the received user input and uses natural language processing technology to extract important keywords and phrases.

[0727] Generation of provisional arrangement information

[0728] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[0729] Confirmation and additional hearings

[0730] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[0731] Generating the final format

[0732] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet contents and makes corrections as necessary. Once the final sheet is confirmed, it is sent to the server via the terminal.

[0733] Data Retention and Notification

[0734] The server then saves the final generated sheet in a database. It also notifies the responder that a new consultation has been registered. This allows the staff member to understand the specific details of the user's consultation in advance, enabling them to respond quickly and appropriately.

[0735] Specific examples

[0736] For example, if a user types "Tell me about new smartphone models," the following prompt sentence is used:

[0737] Prompt Sentence Examples

[0738] Please organize the following information: I would like to know about the new smartphone model.

[0739] The server uses this prompt to call the generative AI model, extracting the keywords "new model" and "please tell me," and organizing them into a temporary format. This information is then presented to the user, who is then asked to enter additional details. Through this process, the final format is generated, and the user's inquiry is notified to the store staff.

[0740] According to the above embodiment, users can effectively use their waiting time to input the details of their consultation, and staff can efficiently respond to users' consultations.

[0741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] Starting an interactive session

[0744] The user starts a session with the conversational AI using a device. When the user launches the smartphone app, a request to start the session is sent from the device to the server. The server starts the session and sends the initial message "Please tell us your consultation." to the device.

[0745] Input: User's session initiation request

[0746] Output: Initial message from the server

[0747] Step 2:

[0748] Hearing phase

[0749] The user inputs their concerns through their device. This input is sent from the device to the server, which then receives it. The server uses a generative AI model and natural language processing technology to analyze the user's input and extract important keywords and phrases.

[0750] Input: User's consultation content input

[0751] Output: Extracted keywords and phrases

[0752] Step 3:

[0753] Generation of provisional arrangement information

[0754] The server organizes the consultation content into a provisional format based on the extracted keywords and phrases. This provisionally organized information is then presented to the user via their terminal.

[0755] Input: Extracted keywords or phrases

[0756] Output: Provisional arrangement information

[0757] Step 4:

[0758] Confirmation and additional hearings

[0759] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters specific additional information, which is then sent back to the server from the terminal. The server analyzes the additional information and organizes it into a format as the final consultation content.

[0760] Input: Additional hearing information for the user

[0761] Output: The final parsed consultation

[0762] Step 5:

[0763] Generating the final format

[0764] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet content and makes corrections as necessary.

[0765] Input: The final parsed consultation content

[0766] Output: The final sheet presented to the user

[0767] Step 6:

[0768] Check and fix

[0769] The user checks the sheet contents and makes any necessary corrections. The corrected contents are sent back to the server via the terminal, and the server regenerates the final sheet based on the updated information.

[0770] Input: User-modified information

[0771] Output: Updated final sheet

[0772] Step 7:

[0773] Data Retention and Notification

[0774] The server saves the final generated sheet in a database and notifies the responder that a new consultation has been registered, allowing the staff member to understand the specific details of the user's consultation in advance.

[0775] Input: Updated final sheet

[0776] Output: Sheets saved in the database and notifications to responders

[0777] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0778] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the user's consultation details and feelings, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[0779] Overall system configuration

[0780] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, a database that ultimately stores the data, and an emotion engine that recognizes the user's emotions. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device and analyzes each.

[0781] Program processing

[0782] 1. Starting an interactive session

[0783] The user starts a session with the conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends the initial message "Please tell us your consultation details" to the device.

[0784] 2. Hearing Phase

[0785] The user inputs the content of their inquiry through the device. For example, they might input, "I'd like to know about my mobile phone plan." The device then sends the user's input to the server. The emotion engine also analyzes the emotional data of the user's input.

[0786] 3. Preliminary organization of information

[0787] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases. For example, keywords such as "price plan" and "please tell me." The emotion engine recognizes the user's emotions (e.g., joy, anger, sadness, etc.) and sends that information to the server.

[0788] 4. Confirmation and additional hearings

[0789] The server takes into account the extracted keywords and the user's emotional information, and organizes the provisionally organized information in a format. This provisionally organized information is then presented to the user again via the terminal. Depending on the user's emotions, the server may send additional messages, such as a message to help them relax. The server may also ask additional questions, such as, "Which part specifically would you like to know more about?"

[0790] 5. Emotion-Based Regulation

[0791] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[0792] 6. Final information organization

[0793] The server then organizes the final consultation content into a format based on the additional information and emotion data received again. For example, it could organize the content into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0794] 7. Creating a sheet

[0795] The server generates a sheet in a predetermined format based on the final consultation content, which also includes the user's emotional data.

[0796] 8. Check and correct

[0797] The server sends the generated sheet to the terminal for the user to check. The user checks the sheet contents and makes corrections as necessary. These corrections are sent back to the server, and the sheet is regenerated.

[0798] 9. Data Retention and Notification

[0799] The server saves the final generated sheet in a database. It also notifies staff that a new consultation has been registered. Staff use dedicated terminals to check the pre-generated sheet and prepare to respond efficiently when the user visits. When responding, the server also takes into account the user's emotional information and provides an appropriate response.

[0800] Specific examples

[0801] Scenario: Consultation about changing mobile phone plan

[0802] 1. Starting an interactive session

[0803] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0804] 2. Hearing Phase

[0805] The user types, "Please tell me about my mobile phone plan." The device sends this input to the server. The emotion engine then analyzes the emotion (e.g., nervousness, interest, etc.) based on the user's input.

[0806] 3. Preliminary organization of information

[0807] The server extracts keywords such as "price plan" and "what would you like to know" and tentatively organizes the content of the consultation. It also incorporates emotional information obtained from the emotion engine.

[0808] 4. Confirmation and additional hearings

[0809] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[0810] 5. Emotion-Based Regulation

[0811] If the emotion engine detects that the user's emotions are a little unstable, the server will follow up with, "What are you worried about? Please tell us more."

[0812] 6. Final information organization

[0813] Based on the additional information and emotion data, the server updates the format and organizes it as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0814] 7. Creating a sheet

[0815] The server generates a sheet based on this information and prompts the user to confirm it.

[0816] 8. Check and correct

[0817] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[0818] 9. Data Retention and Notification

[0819] The finalized sheet is saved in a database, and the person in charge is notified of the new consultation content and emotional data. Staff members check the information in advance and prepare to respond appropriately when the patient visits.

[0820] This format allows users to smoothly communicate their concerns to staff members even when they meet for the first time, and allows staff members to respond efficiently while taking into consideration the user's feelings. The data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[0821] The processing flow will be explained below.

[0822] Step 1:

[0823] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[0824] Step 2:

[0825] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[0826] Step 3:

[0827] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[0828] Step 4:

[0829] The terminal transmits the user's input to the server, and at the same time, the emotion engine collects the user's emotion data (voice tone, input speed, phrase selection, etc.) and transmits it to the server.

[0830] Step 5:

[0831] The server analyzes the received user input and uses natural language processing techniques to extract important keywords and phrases.

[0832] Step 6:

[0833] The emotion engine analyzes the received emotion data and identifies the user's emotional state (e.g., joy, anxiety, anger, etc.). Based on this information, the server tentatively organizes the user's consultation details.

[0834] Step 7:

[0835] The server organizes the provisionally organized information based on the extracted keywords and emotion data and displays it on the terminal. For example, the information might be "Consultation topic: Price plan," "Specific content: Please tell me," and "Emotional state: Anxiety."

[0836] Step 8:

[0837] The server can ask additional questions or make additional considerations based on the user's emotional state, for example, "If you would like to know more about our pricing plans, we can explain them in detail. Please let us know if you have any concerns."

[0838] Step 9:

[0839] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[0840] Step 10:

[0841] The terminal sends additional user input to the server, and the emotion engine collects new emotion data and sends it to the server.

[0842] Step 11:

[0843] The server analyzes the received additional information and formats it into the final consultation content. The emotion engine analyzes the user's emotional state again and includes the emotion data in the final sheet.

[0844] Step 12:

[0845] The server generates a final consultation content sheet and sends it to the terminal for the user to confirm. The sheet contains, for example, the following information:

[0846] Consultation topic: Pricing plan

[0847] Specific content: How to change from smartphone to tablet

[0848] Emotional state: Anxiety

[0849] Step 13:

[0850] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[0851] Step 14:

[0852] The terminal transmits the corrections to the server, and the emotion engine collects the user's emotion data again.

[0853] Step 15:

[0854] The server analyzes the modifications and regenerates the sheet, including the final emotion data from the emotion engine.

[0855] Step 16:

[0856] The server saves the finalized sheet in the database and notifies the person in charge that the new consultation content and emotional data have been registered.

[0857] Step 17:

[0858] The responder uses a dedicated terminal to check the pre-generated sheet and emotion data, and prepares to respond efficiently and appropriately when the user visits.

[0859] Example 2

[0860] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0861] In recent years, there has been a demand for stores, government offices, call centers, and other facilities to make effective use of users' waiting time and efficiently handle inquiries. However, current systems have difficulty grasping users' emotions and the details of their inquiries, which can lead to a decline in user satisfaction. Furthermore, they are unable to respond in a way that takes users' emotions into appropriate consideration, which can result in problems and dissatisfaction. It is necessary to solve these problems and provide a system that provides high satisfaction to both users and staff.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0863] In this invention, the server includes: means for a user to use a terminal to start a session with an interactive artificial intelligence; means for the server to receive the user's consultation content, analyze it using natural language processing technology, and extract important keywords and phrases; means for the server to analyze the user's emotional data using an emotion recognition engine and acquire emotional information; means for the server to create provisional organized information based on the extracted keywords and emotional information, organize it according to a format, and present it to the terminal; means for the server to ask the user additional questions, organize the final information, and generate a consultation content sheet; means for the server to store the final generated sheet in a database and notify the person handling the consultation; and means for the person handling the consultation to respond efficiently while taking the user's emotional information into consideration. This makes it possible to understand the user's emotions and detailed consultation content, and respond accurately and efficiently.

[0864] A "terminal" is an electronic device used by a user, which is a means for initiating a session with an interactive artificial intelligence and inputting and displaying consultation details.

[0865] "Conversational artificial intelligence" refers to an algorithm or system that allows a user to interact with it in natural language, analyzing input from the user and generating appropriate responses.

[0866] A "server" is a computer system that receives, processes, and analyzes data sent from terminals via a network.

[0867] "Natural language processing technology" is a general term for technology that analyzes, understands, and generates human language, and is used to extract important keywords and phrases.

[0868] An "emotion recognition engine" is software or an algorithm that analyzes emotions from user input data and obtains that information.

[0869] "Temporarily organized information" is information that has been temporarily organized based on keywords and emotional information extracted from the contents of the user's consultation.

[0870] A "format" is a standardized structure or form for organizing and presenting information.

[0871] A "consultation sheet" is a written or electronic form containing the final organized consultation and emotional information.

[0872] A "database" is a system for efficiently storing, managing, and searching accumulated data.

[0873] The "responder" is a person in charge of responding to the content of the inquiry from the user, and is a person who receives a notification from the server and takes an appropriate action.

[0874] This invention is a system in which a user inputs the details of a consultation using a terminal, a server analyzes and organizes the input, and generates a final consultation details sheet to notify the person handling the consultation. Specific embodiments for implementing this system are described below.

[0875] Overall system configuration

[0876] The system consists of a device used by the user, a server, a database, and an emotion recognition engine. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device, analyzes them, and organizes the user's consultation content.

[0877] System hardware and software examples

[0878] Device: A user device such as a smartphone, tablet, or PC.

[0879] Server: A high-performance computer system (e.g., a cloud server such as AWS or Azure).

[0880] Database: A SQL or NoSQL database (e.g. MySQL, MongoDB).

[0881] Emotion recognition engine: Software that analyzes emotions (e.g., Microsoft Azure Cognitive Services, IBM Watson, etc.).

[0882] Natural Language Processing (NLP): Technology for analyzing text (e.g., Google Natural Language API, spaCy).

[0883] Explanation of program processing

[0884] 1. Starting an interactive session

[0885] The user starts a session with the conversational AI using a terminal. The terminal sends a session start request to the server, and the server starts the session. The server sends the initial message "Please tell us your consultation details" to the terminal and displays it to the user.

[0886] 2. Hearing Phase

[0887] The user inputs the content of their inquiry through the device. For example, if they input "I'd like to know about my mobile phone plan," the device sends this input to the server. At the same time, the emotion recognition engine analyzes the user's emotional data.

[0888] 3. Preliminary organization of information

[0889] The server analyzes the received consultation content using natural language processing (NLP) technology to extract important keywords and phrases. It also obtains emotional data analyzed by an emotion recognition engine and creates provisional sorting information.

[0890] 4. Confirmation and additional hearings

[0891] The server then verifies the user based on the provisionally organized information and asks additional questions, such as, "Which part specifically would you like to know about?" The server then reorganizes the information based on the user's response.

[0892] 5. Emotion-Based Regulation

[0893] The server takes into account the user's emotional data and adjusts the content and order of follow-up questions, such as "What are you worried about?"

[0894] 6. Final information organization

[0895] The server then organizes the consultation details based on the information it finally obtains and writes the final information in a format, such as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0896] 7. Creating a sheet

[0897] The server then generates a sheet based on the final organized information, which also includes the user's emotional data.

[0898] 8. Check and correct

[0899] The sheet is sent to the terminal, where the user can check the contents and make corrections as necessary. The corrections are then sent back to the server, and the sheet is updated.

[0900] 9. Data Retention and Notification

[0901] The confirmed sheet is saved in the database, and the person in charge is notified of the new consultation content. The person in charge checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[0902] Examples and prompts

[0903] Scenario: Consultation about changing mobile phone plan

[0904] 1. Starting an interactive session

[0905] A user launches the app on their smartphone and starts chatting with a conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[0906] 2. Hearing Phase

[0907] The user types "I'd like to know about mobile phone plans," and the device sends this information to the server. At the same time, the emotion recognition engine analyzes the user's emotions.

[0908] 3. Preliminary organization of information

[0909] The server extracts keywords such as "price plan" and "please tell me" and creates provisional sorting information along with emotional data.

[0910] 4. Confirmation and additional hearings

[0911] Based on the provisionally organized information, the server asks the user, "Which part specifically would you like to know?" The user enters, "I would like to know how to change from a smartphone to a tablet," and the device resends the request to the server.

[0912] 5. Emotion-Based Regulation

[0913] If the emotion recognition engine detects the user's anxiety, the server asks a follow-up question: "What are you worried about?"

[0914] 6. Final information organization

[0915] Based on the additional information and emotional data, the server organizes the final information into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[0916] Example of input prompt for generative AI model

[0917] The functionality of this system can be simulated concretely by inputting the following prompt sentences into the generative AI model:

[0918] A user opens a smartphone app and types, "I'd like to know about my mobile phone plan." The system sends the initial message to the user, and the emotion engine analyzes the user's emotions. What action will the system take next?

[0919] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0920] Program processing flow

[0921] Step 1:

[0922] A user starts an application on a device such as a smartphone or PC and starts a session with the conversational AI. The device sends a session start request to the server. The server receives the request, prepares to start the session, and generates an initial message, "Please tell us your consultation details," and sends it to the device. The device receives this message and displays it to the user.

[0923] Step 2:

[0924] The user inputs the content of their inquiry into an input field on the device. For example, they might input "Please tell me about my mobile phone plan." The device then sends the input content as text data to the server. The server receives this text data and transfers it to an emotion recognition engine. The server then begins to apply natural language processing (NLP) to the user's input content. The input data includes the user's text data and emotion data analyzed by the emotion recognition engine.

[0925] Step 3:

[0926] The server obtains emotional data (e.g., "anxiety," "interest") obtained by an emotion recognition engine, and important keywords and phrases (e.g., "price plan," "please tell me") extracted using natural language processing technology. Through this process, the server understands the user's basic inquiry content and emotional state regarding "price plan." For example, if the input data is "please tell me about mobile phone rate plans," and the emotional data is "anxiety," the keywords "price plan" and "anxiety" are extracted.

[0927] Step 4:

[0928] The server creates provisionally organized information based on the keywords and emotion information extracted by the server, and organizes it according to a format. This provisionally organized information is sent to the terminal for confirmation by the user, for example, as "You're asking about a pricing plan, right?" The terminal receives this information and displays it to the user. The user can confirm the content and enter more specific questions and answers.

[0929] Step 5:

[0930] The server receives the user's additional input and analyzes it again using natural language processing technology and an emotion recognition engine. For example, if the user inputs, "I want to know how to switch from a smartphone to a tablet," the server re-extracts the keywords "how to switch" and "I want to know" as well as the user's emotion data. Based on this additional information, the server creates more detailed provisionally organized information and sends it to the device.

[0931] Step 6:

[0932] The server adjusts the content of the answers and follow-up questions based on the user's emotional data. For example, if the user enters "I want to know how to switch from a smartphone to a tablet" and the emotional data indicates "anxiety," the server generates a follow-up message asking "What are you worried about?" and sends it to the device. The user then enters an additional answer and sends it again to the server.

[0933] Step 7:

[0934] The server organizes the final consultation content based on the additional information and emotional data. For example, it organizes the information according to a format such as "Consultation topic: Price plan," "Specific content: How to switch from a smartphone to a tablet," and "Emotional state: Anxiety." The server generates a sheet based on this information and sends it to the device.

[0935] Step 8:

[0936] The user can check the generated sheet on the terminal and make corrections as necessary. Once the corrections are sent to the server, the server updates the sheet again and sends it to the terminal.

[0937] Step 9:

[0938] The server saves the confirmed consultation content sheet in a database. It also notifies the person handling the consultation that a new consultation content has been registered. The person handling the consultation checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[0939] (Application example 2)

[0940] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0941] Conventional consultation systems systematically organize consultation content without considering the user's emotions, which results in an inability to fully grasp the user's true needs and concerns. Furthermore, in face-to-face consultations, it is difficult for staff to obtain emotional information in real time, which can prevent them from providing an appropriate response. In such situations, user satisfaction declines and efficient response becomes difficult.

[0942] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize provisional consultation details information according to a format and present it to the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for the terminal to start an interaction session and analyze the user's emotional data from the voice input in real time; a means for the smart glasses to recognize the customer's emotional state in real time and generate an appropriate response; a means for providing consultation response information to staff based on the final information; and a means for sharing information, such as emotional data, in real time so that staff can respond efficiently. This makes it possible to recognize emotions in real time through interaction with the user and provide appropriate responses.

[0943] The "means for a user to input the content of a consultation using a terminal" refers to an interface that allows a user to input the content of a consultation by text or voice using a mobile terminal, personal computer, or other device.

[0944] "Means for the server to use natural language processing technology to analyze the content of the user's inquiry and extract important keywords and phrases" refers to a technology in which the server uses natural language processing technology, a type of artificial intelligence technology, to analyze the content entered by the user and extract important information based on specific rules.

[0945] "Means for the server to organize provisional consultation information according to a format and present it on the terminal" refers to a technology in which the server provisionally organizes the consultation information based on keywords and phrases extracted by the server, compiles it according to a predetermined format, and displays the information on the user's terminal.

[0946] "Means for the server to ask additional questions, organize the final information, and generate a consultation content sheet" refers to a technology in which the server asks supplementary questions to the user, organizes the final consultation content based on the information obtained, formats the content, and generates it as a sheet.

[0947] "Means for the server to save the final sheet in a database and notify the responder" refers to a technology that saves the final consultation content sheet generated by the server in a database and notifies the staff member in charge of that information.

[0948] "Means for a terminal to initiate an interactive session and analyze user emotional data from voice input in real time" refers to a technology in which a user terminal initiates voice recognition and analyzes emotional data in real time based on information obtained from the user's voice input.

[0949] "A means for smart glasses to recognize a customer's emotional state in real time and generate an appropriate response" refers to a technology that uses an emotion recognition sensor built into smart glasses to analyze a customer's emotional state in real time from their facial expressions and voice, and generates an appropriate response based on that information.

[0950] "Means for providing staff with information on how to respond to inquiries based on final information" refers to a technique for providing staff with information that will enable them to respond appropriately to customers based on the final organized information on the content of the inquiries.

[0951] "Means for sharing information, such as emotional data, in real time to enable staff to respond efficiently" refers to technology that enables staff to share customer emotional data and other important information in real time, enabling efficient customer service.

[0952] The present invention relates to a system for efficiently responding to consultations while taking into consideration the user's emotions when making inquiries at stores, government offices, call centers, etc. Specific embodiments for carrying out the invention will be described below.

[0953] Overall system configuration

[0954] The system consists of the following main components:

[0955] 1. User terminal: A device such as a smartphone or personal computer (PC) that the user uses to input the details of their consultation.

[0956] 2. Server: Analyzes the user's inquiry using natural language processing technology, extracts important keywords and phrases, and organizes them.

[0957] 3. Database: Stores the final consultation sheet generated by the server.

[0958] 4. Emotion engine: Analyzes emotional data from the user's voice and facial expressions in real time.

[0959] 5. Smart Glasses: A device worn by staff that recognizes the emotional state of customers in real time and generates appropriate responses.

[0960] 6. Staff terminal: A device used by staff to check response information.

[0961] System Operation Overview

[0962] 1. Start an interactive session:

[0963] A user accesses the system using a smartphone or PC and starts a session with the conversational AI. The server receives the session start request and sends the initial message, "Please tell us your consultation details." to the user's device.

[0964] 2. Hearing Phase:

[0965] When the user inputs the content of their consultation, the device sends the input to the server. At the same time, the emotion engine analyzes the user's voice and facial expressions to obtain emotional data, and sends the results to the server.

[0966] 3. Preliminary organization of information:

[0967] The server analyzes the received user inquiry and uses natural language processing technology to extract important keywords and phrases. For example, it extracts keywords such as "price plan" and "please tell me." Based on this, it creates provisionally organized information and presents it to the user's device.

[0968] 4. Verification and Further Hearing:

[0969] The server then uses the preliminary information to ask additional questions, such as, "What specific part would you like to know about?" Any additional information entered by the user is analyzed in the same way.

[0970] 5. Emotion-based regulation:

[0971] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[0972] 6. Final information organization:

[0973] The server then organizes and formats the final consultation content based on the additional information and emotion data. For example, the consultation topic might be "Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[0974] 7. Generate and check the sheet:

[0975] The final organized information is generated as a correspondence sheet and presented to the user for confirmation. If the user enters any corrections, the information is reorganized and the sheet is updated.

[0976] 8. Data Retention and Notification:

[0977] The final sheet is saved in a database and sent to the customer service staff, who can then use the smart glasses to check the customer's emotional state in real time and respond accordingly.

[0978] Specific examples

[0979] As a concrete example, consider the following:

[0980] Scenario: Consultation about changing mobile phone plan

[0981] Example prompt:

[0982] "Please tell me about the administrative services available at the citizen service desk. Specifically, I would like information about tax consultations."

[0983] The system described above allows users to consult with confidence, and allows staff to take the user's feelings into consideration and respond more appropriately and quickly. Ultimately, the consultation content stored in the database will be used as learning data for future AI, which is expected to further improve the efficiency of problem-solving.

[0984] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0985] Step 1:

[0986] A user accesses the system using a smartphone or PC terminal and starts an interactive session. As input, the server receives a session start request from the user, and sends the initial message "Please tell us your consultation details" to the terminal. As output, the initial message is displayed on the user's terminal.

[0987] Step 2:

[0988] The user inputs the content of their consultation into their smartphone or PC terminal using voice or text. The consultation content data from the user is received as input and sent to the server. The consultation content data received by the server is passed to the natural language processing engine as output, and analysis begins.

[0989] Step 3:

[0990] The server uses natural language processing technology to analyze the user's consultation content and extract important keywords and phrases. The natural language processing engine analyzes the consultation content data received as input and extracts important information. The extracted keywords and phrases are saved as provisionally organized information as output.

[0991] Step 4:

[0992] The server uses an emotion engine to analyze emotion data from the user's voice and facial expressions in real time. The user's voice and facial expression data are sent to the emotion engine as input. Emotion data is generated as the analysis result as output and added to the provisional sorting information.

[0993] Step 5:

[0994] The server organizes the provisionally organized information according to a format and presents it to the user's terminal. As input, the provisionally organized information and emotion data are integrated within the server. As output, the formatted provisionally organized information is sent to the user's terminal and displayed to the user.

[0995] Step 6:

[0996] The server asks a follow-up question and receives the answer from the user again. As input, a follow-up question based on the provisional sorting information is sent to the user's terminal. The user enters the answer, and the data is sent to the server. As output, the follow-up information is saved on the server.

[0997] Step 7:

[0998] The server organizes and formats the final consultation content based on the additional information and emotion data. The additional information and the existing provisionally organized information are integrated as input. The final consultation content sheet is generated as output.

[0999] Step 8:

[1000] The server sends the generated final sheet to the user's terminal, where the user can review and modify it. As input, the final sheet is presented to the user's terminal. The user enters modifications, which are sent to the server. As output, the modified final sheet is regenerated.

[1001] Step 9:

[1002] The server saves the final consultation content sheet in the database and notifies the staff. As input, the final confirmation sheet is saved in the database. As output, the new consultation content is notified in real time to the staff terminal and smart glasses.

[1003] Step 10:

[1004] Staff use smart glasses to check and respond to customers' emotional states in real time. As input, the smart glasses receive information from the database and also obtain customer emotional data in real time. As output, an appropriate response is provided to the customer.

[1005] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1006] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1007] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1008] [Third embodiment]

[1009] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1010] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1011] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1012] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1013] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1014] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1015] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1016] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1017] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1018] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1019] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1020] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1021] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the contents of users' inquiries, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[1022] Overall system configuration

[1023] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, and a database that ultimately stores the data. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content sent from the device and analyzes it using natural language processing technology.

[1024] Program processing

[1025] 1. Starting an interactive session

[1026] A user starts a session with a conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends an initial message to the device.

[1027] 2. Hearing Phase

[1028] The user inputs the details of their consultation through their device. This input is then sent from the device to the server, which then analyzes the received user input and extracts important keywords and phrases.

[1029] 3. Preliminary organization of information

[1030] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[1031] 4. Confirmation and additional hearings

[1032] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[1033] 5. Creating a sheet

[1034] The server generates a sheet according to the format based on the final consultation content, and presents this sheet to the user for final confirmation.

[1035] 6. Check and correct

[1036] The user checks the sheet contents and makes any necessary corrections. Once the final sheet is confirmed, it is sent to the server via the terminal.

[1037] 7. Data Retention and Notification

[1038] The server saves the final generated sheet in the database and notifies the staff that a new consultation has been registered.

[1039] Specific examples

[1040] Scenario: Consultation about changing mobile phone plan

[1041] 1. Starting an interactive session

[1042] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1043] 2. Hearing Phase

[1044] The user inputs "I would like to know about mobile phone plans." The device sends this input to the server.

[1045] 3. Preliminary organization of information

[1046] The server extracts the keywords "price plan" and "what would you like to know" and tentatively organizes the consultation content.

[1047] 4. Confirmation and additional hearings

[1048] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[1049] 5. Final information organization

[1050] The server analyzes the additional information and organizes it according to the format: "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1051] 6. Creating a sheet

[1052] The server generates a sheet based on this information and prompts the user to confirm it.

[1053] 7. Check and correct

[1054] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[1055] 8. Data Retention and Notification

[1056] The finalized sheet is saved in the database and staff are notified of the new consultation details.

[1057] This allows users to smoothly communicate their concerns to staff members even when they meet for the first time, enabling staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1058] The processing flow will be explained below.

[1059] Step 1:

[1060] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[1061] Step 2:

[1062] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[1063] Step 3:

[1064] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[1065] Step 4:

[1066] The terminal transmits the user's input to the server.

[1067] Step 5:

[1068] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases, such as "price plan" and "please tell me."

[1069] Step 6:

[1070] The server then uses the extracted keywords to organize the provisional information about the consultation according to a format, such as "Consultation topic: Price plan" and "Specific content: Please let me know."

[1071] Step 7:

[1072] The server sends the preliminary information to the terminal and presents it to the user. The server also asks follow-up questions, such as "Which part specifically would you like to know about?"

[1073] Step 8:

[1074] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[1075] Step 9:

[1076] The terminal sends the user's additional input to the server.

[1077] Step 10:

[1078] The server analyzes the received additional information and organizes it into a format as the final consultation content, for example, "Consultation topic: Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[1079] Step 11:

[1080] The server generates a sheet in a predetermined format based on the final consultation content.

[1081] Step 12:

[1082] The server sends the generated sheet to the terminal for the user to check.

[1083] Step 13:

[1084] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[1085] Step 14:

[1086] The terminal sends the modifications to the server.

[1087] Step 15:

[1088] The server re-parses the modifications and regenerates the sheet.

[1089] Step 16:

[1090] The server saves the final sheet in the database and notifies the respondent that a new consultation has been registered.

[1091] Step 17:

[1092] The responder uses a dedicated terminal to check the pre-generated sheet and prepares to respond efficiently when the user visits.

[1093] This allows the user to smoothly communicate the content of the consultation, and the person in charge can also respond efficiently.

[1094] Example 1

[1095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1096] In modern society, making effective use of waiting time at stores, government offices, call centers, etc., and efficiently listening to and organizing user inquiries is essential to improving business efficiency and customer satisfaction. However, there is a lack of appropriate systems to achieve this, and most responses are handled manually, resulting in a waste of time and effort. Therefore, there is a need for a system that can quickly and accurately receive, analyze, organize, and ultimately provide user inquiries to the person in charge.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1098] In this invention, the server includes means for a user to input consultation details using an information processing device, means for a computer to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases, means for the computer to organize provisionally organized information of the consultation details according to a format and present it to the information processing device, means for the computer to generate an initial message and send it to the information processing device, means for the computer to ask additional questions, organize the final information, and generate a consultation details sheet, and means for the computer to save the final sheet in a database and notify the service provider. This makes it possible to efficiently hear the user's consultation details, organize them, and provide them to the person who will handle the consultation.

[1099] "User" refers to an individual or corporation who uses the service and inputs the details of their inquiry.

[1100] An "information processing device" is a device used by a user to input consultation details, and specifically refers to a smartphone, PC, tablet, etc.

[1101] "Server" refers to a computer system that receives data sent by users and analyzes, processes, stores, and notifies them.

[1102] "Natural language processing technology" refers to technology for analyzing user input text and extracting important keywords and phrases.

[1103] "Provisionally organized information" refers to information that temporarily organizes consultation content based on keywords and phrases extracted using natural language processing technology.

[1104] "Format" refers to a standard format or structure for organizing consultation content in a consistent manner.

[1105] "Initial message" refers to the first message that the server sends to the user at the beginning of a session.

[1106] "Sheet" refers to a document or data format that contains the final, organized content of the consultation.

[1107] "Database" refers to a data management system for storing the final generated sheets and other related data.

[1108] "Service provider" refers to an individual or organization that responds to and provides support based on the content of inquiries from users.

[1109] This invention relates to a system that effectively utilizes waiting time at stores, government offices, call centers, etc., and efficiently listens to and organizes the details of users' inquiries and provides them to staff. This system consists of an information processing device used by users, a server that receives and analyzes the details of the inquiries, and a database that stores the data.

[1110] A user initiates a session with a conversational AI using an information processing device such as a smartphone or PC. The server receives the user's consultation content sent from the terminal and analyzes it using natural language processing technology (e.g., Google Cloud NLP API). Specific embodiments for implementing this invention are described below.

[1111] When the server receives the consultation content entered by the user, it uses natural language processing technology to extract important keywords and phrases. This allows the server to accurately understand the user's intent and the content of the question. The extracted keywords and phrases are used to tentatively organize the consultation content.

[1112] The provisionally organized information is then organized into a standard format by the server and presented to the user via their terminal. This allows the user to easily confirm the content of their consultation. The server also asks additional questions at this stage to obtain more detailed information.

[1113] The information obtained from the additional interviews is analyzed in the same way and organized into a format as the final consultation content. The server generates a sheet based on this final information and asks the user to confirm it again. The user checks the sheet contents and corrects them as necessary to confirm the exact consultation content.

[1114] The server stores the confirmed consultation details in a database and sends a notification to the staff, enabling them to respond promptly to the newly registered consultation details.

[1115] Specific examples

[1116] Scenario: Consultation about changing mobile phone plan

[1117] 1. The user launches the smartphone app and taps the "Price Plan Consultation" button. The app then displays the message, "Please tell us your inquiry."

[1118] 2. The terminal sends a session initiation request to the server and sends the user ID.

[1119] 3. The server starts the session and sends an initial message to the terminal, for example, "Please tell us what you would like to discuss with us."

[1120] 4. The user types, "I'd like to know about mobile phone plans."

[1121] 5. The device sends the input information to the server.

[1122] 6. The server analyzes the received message using natural language processing technology and extracts important keywords such as "price plan" and "please tell me."

[1123] 7. The server generates provisional information such as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan" and sends it to the terminal.

[1124] 8. The terminal displays the provisional sorting information to the user.

[1125] 9. The user adds, "I want to know how to change from a smartphone to a tablet."

[1126] 10. The device sends the additional input to the server.

[1127] 11. The server analyzes again and generates "Consultation topic: Pricing plan, Details: How to change from smartphone to tablet."

[1128] 12. The server generates a sheet based on the final consultation content and sends it to the terminal.

[1129] 13. The terminal displays the sheet to the user, who then checks and modifies it.

[1130] 14. The user amends the question to say, "I want to change the Wi-Fi in addition to the tablet."

[1131] 15. The device sends the modified content to the server.

[1132] 16. The server generates a new sheet reflecting the changes and sends it to the terminal.

[1133] 17. The terminal displays the sheet again to the user for final confirmation.

[1134] 18. The server saves the confirmed sheet in the database and notifies the staff that a new consultation has been registered.

[1135] Prompt Sentence Examples

[1136] User prompt: "I want to change my mobile phone plan. How do I do this?"

[1137] Server's initial response: "Please tell us what you would like to discuss."

[1138] Additional prompt: "I want to know how to change from a smartphone to a tablet."

[1139] In this way, users can smoothly communicate their concerns to staff members they meet for the first time, enabling the staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1141] Step 1:

[1142] Starting an interactive session

[1143] The user uses an information processing device (smartphone or PC) to start a session with the conversational AI.

[1144] Input: User clicks a "Start Session" button on an application or web page.

[1145] The terminal sends a session initiation request to the server, along with data including the user ID and a timestamp of the session initiation.

[1146] The server receives the request and performs processing to initialize the session, such as generating a new session ID and performing initial settings.

[1147] Output: The server generates an initial message and sends it to the user. The initial message, such as "Please tell us what you would like to discuss," is displayed on the terminal.

[1148] Step 2:

[1149] Hearing phase

[1150] The user inputs the content of their inquiry into the terminal. For example, they might input, "I'd like to know about mobile phone rate plans."

[1151] Input: A user enters text into an input field on an application.

[1152] The terminal sends the input text to the server.

[1153] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it uses the Google Cloud NLP API to extract important keywords from the text (e.g., "pricing plan" or "please tell me").

[1154] Output: Extracted keywords and analysis results.

[1155] Step 3:

[1156] Provisional organization of information

[1157] The server then organizes the consultation content into a temporary format based on the extracted keywords. For example, it could organize the content as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan."

[1158] Input: Keywords extracted by natural language processing and analysis results.

[1159] The server generates a message to present the provisional arrangement information to the user.

[1160] Output: Preliminary organized consultation message.

[1161] Step 4:

[1162] Confirmation and additional hearings

[1163] The user checks the provisional information provided by the server and enters additional information. For example, the user might enter, "I want to know how to change from a smartphone to a tablet."

[1164] Input: The user enters additional information and the device sends it to the server.

[1165] The server then analyzes the received data using natural language processing technology and formats the final consultation content based on the additional information.

[1166] Output: Information organized as the final consultation content.

[1167] Step 5:

[1168] Sheet Generation

[1169] The server then generates a sheet based on the final organized consultation content. For example, it creates a sheet with the following information: "Consultation topic: Pricing plan, Details: How to switch from smartphone to tablet."

[1170] Input: The final consultation details.

[1171] The server sends the generated sheet to the terminal for presentation to the user.

[1172] Output: The generated sheet.

[1173] Step 6:

[1174] Check and fix

[1175] The user checks the sheet displayed on the device and makes any necessary changes. For example, they might change the settings to "I want to change the Wi-Fi in addition to the tablet."

[1176] Input: The user inputs the correction information, and the terminal sends it to the server.

[1177] The server receives the revised content and updates the sheet again to generate the final version.

[1178] Output: The final sheet with the modifications reflected.

[1179] Step 7:

[1180] Data Retention and Notification

[1181] The server stores the final sheet in a database, including the consultation content, user identification information, time information, etc.

[1182] Input: Final sheet and associated metadata.

[1183] The server notifies the service provider that a new consultation has been registered. Notification methods include email and push notification.

[1184] Output: Information stored in the database and notification messages.

[1185] By this processing flow, the contents of the user's inquiry are efficiently and accurately heard and provided to the person who will ultimately handle the matter.

[1186] (Application example 1)

[1187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1188] In physical stores, there is a demand for a system that allows users to make effective use of their waiting time and efficiently consult with staff about product information and order details. In particular, it is necessary to improve in-store service by quickly and accurately conveying the content of users' inquiries to staff. There is also a demand for a means for users to smoothly convey the content of their inquiries to staff members they meet for the first time. Furthermore, there is a demand for improving store operational efficiency by using conversational AI to automatically organize the content of inquiries and respond in a timely manner.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1190] In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize the provisional consultation details according to a format and present it on the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for a user to input product information and order details using a smartphone at a physical store while waiting; a means for the server to analyze the user's input in real time using an interactive AI and generate a provisional format; a means for the server to reflect the user's confirmation and corrections and regenerate the final format; and a means for the server to notify staff at the physical store based on the final format. This allows users to effectively utilize their waiting time at the physical store and efficiently consult about product information and order details. Furthermore, the server automatically organizes the consultation details and quickly and accurately conveys them to staff, thereby improving in-store service and operational efficiency.

[1191] "User" refers to a person who uses the system to input the details of a consultation and receive services.

[1192] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1193] "Consultation content" refers to information related to in-store services, such as product information and order details entered by the user.

[1194] "Server" refers to a computer system that receives and analyzes input from users and notifies staff.

[1195] "Natural language processing technology" refers to the technology of analyzing text data entered by the user and extracting important keywords and phrases.

[1196] "Provisional sorting information" refers to information that sorts out the consultation content input by the user at an early stage.

[1197] "Format" refers to a structure or template for organizing consultation content in a certain format.

[1198] "Conversational AI" refers to an artificial intelligence system that engages in natural dialogue with users and collects and analyzes information.

[1199] "Waiting time" refers to the time a user waits to receive service at a physical store.

[1200] "Notification" refers to the action of informing staff of server-generated information.

[1201] "Physical store" refers to a store or service location that has a physical presence.

[1202] The "provisional format" refers to a format that is temporarily organized at the stage of initial analysis of the content of the user's inquiry.

[1203] The "final format" refers to the format of the consultation content that has been confirmed after being checked and corrected by the user.

[1204] The term "responder" refers to a store staff member who responds to the user upon receiving a notification from the server.

[1205] The present invention provides a system that enables users to effectively utilize their waiting time in a physical store and consult with them about product information and order details. Specific embodiments for carrying out the invention are described below.

[1206] Overall system configuration

[1207] The system mainly consists of the following components:

[1208] 1. Device: The device used by the user, such as a smartphone or tablet.

[1209] 2. Server: A computer system that is responsible for receiving user input, analyzing it, and notifying staff.

[1210] 3. Database: This is where the final consultation sheet is stored.

[1211] Hardware and software used

[1212] Hardware: Smartphones, tablets, server computers

[1213] Software: Python, Flask framework, SQLite database, OpenAI API

[1214] Explanation of program processing

[1215] Starting an interactive session

[1216] The user starts the smartphone app in the waiting area of ​​a physical store and starts a session with the conversational AI. The server then sends an initial message saying, "Please tell us what you would like to discuss," and the session begins.

[1217] Hearing phase

[1218] The user inputs the details of their consultation via their device. This input is then sent from the device to the server, which then analyzes the received user input and uses natural language processing technology to extract important keywords and phrases.

[1219] Generation of provisional arrangement information

[1220] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[1221] Confirmation and additional hearings

[1222] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[1223] Generating the final format

[1224] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet contents and makes corrections as necessary. Once the final sheet is confirmed, it is sent to the server via the terminal.

[1225] Data Retention and Notification

[1226] The server then saves the final generated sheet in a database. It also notifies the responder that a new consultation has been registered. This allows the staff member to understand the specific details of the user's consultation in advance, enabling them to respond quickly and appropriately.

[1227] Specific examples

[1228] For example, if a user types "Tell me about new smartphone models," the following prompt sentence is used:

[1229] Prompt Sentence Examples

[1230] Please organize the following information: I would like to know about the new smartphone model.

[1231] The server uses this prompt to call the generative AI model, extracting the keywords "new model" and "please tell me," and organizing them into a temporary format. This information is then presented to the user, who is then asked to enter additional details. Through this process, the final format is generated, and the user's inquiry is notified to the store staff.

[1232] According to the above embodiment, users can effectively use their waiting time to input the details of their consultation, and staff can efficiently respond to users' consultations.

[1233] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1234] Step 1:

[1235] Starting an interactive session

[1236] The user starts a session with the conversational AI using a device. When the user launches the smartphone app, a request to start the session is sent from the device to the server. The server starts the session and sends the initial message "Please tell us your consultation." to the device.

[1237] Input: User's session initiation request

[1238] Output: Initial message from the server

[1239] Step 2:

[1240] Hearing phase

[1241] The user inputs their concerns through their device. This input is sent from the device to the server, which then receives it. The server uses a generative AI model and natural language processing technology to analyze the user's input and extract important keywords and phrases.

[1242] Input: User's consultation content input

[1243] Output: Extracted keywords and phrases

[1244] Step 3:

[1245] Generation of provisional arrangement information

[1246] The server organizes the consultation content into a provisional format based on the extracted keywords and phrases. This provisionally organized information is then presented to the user via their terminal.

[1247] Input: Extracted keywords or phrases

[1248] Output: Provisional arrangement information

[1249] Step 4:

[1250] Confirmation and additional hearings

[1251] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters specific additional information, which is then sent back to the server from the terminal. The server analyzes the additional information and organizes it into a format as the final consultation content.

[1252] Input: Additional hearing information for the user

[1253] Output: The final parsed consultation

[1254] Step 5:

[1255] Generating the final format

[1256] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet content and makes corrections as necessary.

[1257] Input: The final parsed consultation content

[1258] Output: The final sheet presented to the user

[1259] Step 6:

[1260] Check and fix

[1261] The user checks the sheet contents and makes any necessary corrections. The corrected contents are sent back to the server via the terminal, and the server regenerates the final sheet based on the updated information.

[1262] Input: User-modified information

[1263] Output: Updated final sheet

[1264] Step 7:

[1265] Data Retention and Notification

[1266] The server saves the final generated sheet in a database and notifies the responder that a new consultation has been registered, allowing the staff member to understand the specific details of the user's consultation in advance.

[1267] Input: Updated final sheet

[1268] Output: Sheets saved in the database and notifications to responders

[1269] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1270] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the user's consultation details and feelings, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[1271] Overall system configuration

[1272] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, a database that ultimately stores the data, and an emotion engine that recognizes the user's emotions. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device and analyzes each.

[1273] Program processing

[1274] 1. Starting an interactive session

[1275] The user starts a session with the conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends the initial message "Please tell us your consultation details" to the device.

[1276] 2. Hearing Phase

[1277] The user inputs the content of their inquiry through the device. For example, they might input, "I'd like to know about my mobile phone plan." The device then sends the user's input to the server. The emotion engine also analyzes the emotional data of the user's input.

[1278] 3. Preliminary organization of information

[1279] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases. For example, keywords such as "price plan" and "please tell me." The emotion engine recognizes the user's emotions (e.g., joy, anger, sadness, etc.) and sends that information to the server.

[1280] 4. Confirmation and additional hearings

[1281] The server takes into account the extracted keywords and the user's emotional information, and organizes the provisionally organized information in a format. This provisionally organized information is then presented to the user again via the terminal. Depending on the user's emotions, the server may send additional messages, such as a message to help them relax. The server may also ask additional questions, such as, "Which part specifically would you like to know more about?"

[1282] 5. Emotion-Based Regulation

[1283] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[1284] 6. Final information organization

[1285] The server then organizes the final consultation content into a format based on the additional information and emotion data received again. For example, it could organize the content into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1286] 7. Creating a sheet

[1287] The server generates a sheet in a predetermined format based on the final consultation content, which also includes the user's emotional data.

[1288] 8. Check and correct

[1289] The server sends the generated sheet to the terminal for the user to check. The user checks the sheet contents and makes corrections as necessary. These corrections are sent back to the server, and the sheet is regenerated.

[1290] 9. Data Retention and Notification

[1291] The server saves the final generated sheet in a database. It also notifies staff that a new consultation has been registered. Staff use dedicated terminals to check the pre-generated sheet and prepare to respond efficiently when the user visits. When responding, the server also takes into account the user's emotional information and provides an appropriate response.

[1292] Specific examples

[1293] Scenario: Consultation about changing mobile phone plan

[1294] 1. Starting an interactive session

[1295] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1296] 2. Hearing Phase

[1297] The user types, "Please tell me about my mobile phone plan." The device sends this input to the server. The emotion engine then analyzes the emotion (e.g., nervousness, interest, etc.) based on the user's input.

[1298] 3. Preliminary organization of information

[1299] The server extracts keywords such as "price plan" and "what would you like to know" and tentatively organizes the content of the consultation. It also incorporates emotional information obtained from the emotion engine.

[1300] 4. Confirmation and additional hearings

[1301] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[1302] 5. Emotion-Based Regulation

[1303] If the emotion engine detects that the user's emotions are a little unstable, the server will follow up with, "What are you worried about? Please tell us more."

[1304] 6. Final information organization

[1305] Based on the additional information and emotion data, the server updates the format and organizes it as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1306] 7. Creating a sheet

[1307] The server generates a sheet based on this information and prompts the user to confirm it.

[1308] 8. Check and correct

[1309] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[1310] 9. Data Retention and Notification

[1311] The finalized sheet is saved in a database, and the person in charge is notified of the new consultation content and emotional data. Staff members check the information in advance and prepare to respond appropriately when the patient visits.

[1312] This format allows users to smoothly communicate their concerns to staff members even when they meet for the first time, and allows staff members to respond efficiently while taking into consideration the user's feelings. The data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1313] The processing flow will be explained below.

[1314] Step 1:

[1315] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[1316] Step 2:

[1317] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[1318] Step 3:

[1319] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[1320] Step 4:

[1321] The terminal transmits the user's input to the server, and at the same time, the emotion engine collects the user's emotion data (voice tone, input speed, phrase selection, etc.) and transmits it to the server.

[1322] Step 5:

[1323] The server analyzes the received user input and uses natural language processing techniques to extract important keywords and phrases.

[1324] Step 6:

[1325] The emotion engine analyzes the received emotion data and identifies the user's emotional state (e.g., joy, anxiety, anger, etc.). Based on this information, the server tentatively organizes the user's consultation details.

[1326] Step 7:

[1327] The server organizes the provisionally organized information based on the extracted keywords and emotion data and displays it on the terminal. For example, the information might be "Consultation topic: Price plan," "Specific content: Please tell me," and "Emotional state: Anxiety."

[1328] Step 8:

[1329] The server can ask additional questions or make additional considerations based on the user's emotional state, for example, "If you would like to know more about our pricing plans, we can explain them in detail. Please let us know if you have any concerns."

[1330] Step 9:

[1331] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[1332] Step 10:

[1333] The terminal sends additional user input to the server, and the emotion engine collects new emotion data and sends it to the server.

[1334] Step 11:

[1335] The server analyzes the received additional information and formats it into the final consultation content. The emotion engine analyzes the user's emotional state again and includes the emotion data in the final sheet.

[1336] Step 12:

[1337] The server generates a final consultation content sheet and sends it to the terminal for the user to confirm. The sheet contains, for example, the following information:

[1338] Consultation topic: Pricing plan

[1339] Specific content: How to change from smartphone to tablet

[1340] Emotional state: Anxiety

[1341] Step 13:

[1342] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[1343] Step 14:

[1344] The terminal transmits the corrections to the server, and the emotion engine collects the user's emotion data again.

[1345] Step 15:

[1346] The server analyzes the modifications and regenerates the sheet, including the final emotion data from the emotion engine.

[1347] Step 16:

[1348] The server saves the finalized sheet in the database and notifies the person in charge that the new consultation content and emotional data have been registered.

[1349] Step 17:

[1350] The responder uses a dedicated terminal to check the pre-generated sheet and emotion data, and prepares to respond efficiently and appropriately when the user visits.

[1351] Example 2

[1352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1353] In recent years, there has been a demand for stores, government offices, call centers, and other facilities to make effective use of users' waiting time and efficiently handle inquiries. However, current systems have difficulty grasping users' emotions and the details of their inquiries, which can lead to a decline in user satisfaction. Furthermore, they are unable to respond in a way that takes users' emotions into appropriate consideration, which can result in problems and dissatisfaction. It is necessary to solve these problems and provide a system that provides high satisfaction to both users and staff.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1355] In this invention, the server includes: means for a user to use a terminal to start a session with an interactive artificial intelligence; means for the server to receive the user's consultation content, analyze it using natural language processing technology, and extract important keywords and phrases; means for the server to analyze the user's emotional data using an emotion recognition engine and acquire emotional information; means for the server to create provisional organized information based on the extracted keywords and emotional information, organize it according to a format, and present it to the terminal; means for the server to ask the user additional questions, organize the final information, and generate a consultation content sheet; means for the server to store the final generated sheet in a database and notify the person handling the consultation; and means for the person handling the consultation to respond efficiently while taking the user's emotional information into consideration. This makes it possible to understand the user's emotions and detailed consultation content, and respond accurately and efficiently.

[1356] A "terminal" is an electronic device used by a user, which is a means for initiating a session with an interactive artificial intelligence and inputting and displaying consultation details.

[1357] "Conversational artificial intelligence" refers to an algorithm or system that allows a user to interact with it in natural language, analyzing input from the user and generating appropriate responses.

[1358] A "server" is a computer system that receives, processes, and analyzes data sent from terminals via a network.

[1359] "Natural language processing technology" is a general term for technology that analyzes, understands, and generates human language, and is used to extract important keywords and phrases.

[1360] An "emotion recognition engine" is software or an algorithm that analyzes emotions from user input data and obtains that information.

[1361] "Temporarily organized information" is information that has been temporarily organized based on keywords and emotional information extracted from the contents of the user's consultation.

[1362] A "format" is a standardized structure or form for organizing and presenting information.

[1363] A "consultation sheet" is a written or electronic form containing the final organized consultation and emotional information.

[1364] A "database" is a system for efficiently storing, managing, and searching accumulated data.

[1365] The "responder" is a person in charge of responding to the content of the inquiry from the user, and is a person who receives a notification from the server and takes an appropriate action.

[1366] This invention is a system in which a user inputs the details of a consultation using a terminal, a server analyzes and organizes the input, and generates a final consultation details sheet to notify the person handling the consultation. Specific embodiments for implementing this system are described below.

[1367] Overall system configuration

[1368] The system consists of a device used by the user, a server, a database, and an emotion recognition engine. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device, analyzes them, and organizes the user's consultation content.

[1369] System hardware and software examples

[1370] Device: A user device such as a smartphone, tablet, or PC.

[1371] Server: A high-performance computer system (e.g., a cloud server such as AWS or Azure).

[1372] Database: A SQL or NoSQL database (e.g. MySQL, MongoDB).

[1373] Emotion recognition engine: Software that analyzes emotions (e.g., Microsoft Azure Cognitive Services, IBM Watson, etc.).

[1374] Natural Language Processing (NLP): Technology for analyzing text (e.g., Google Natural Language API, spaCy).

[1375] Explanation of program processing

[1376] 1. Starting an interactive session

[1377] The user starts a session with the conversational AI using a terminal. The terminal sends a session start request to the server, and the server starts the session. The server sends the initial message "Please tell us your consultation details" to the terminal and displays it to the user.

[1378] 2. Hearing Phase

[1379] The user inputs the content of their inquiry through the device. For example, if they input "I'd like to know about my mobile phone plan," the device sends this input to the server. At the same time, the emotion recognition engine analyzes the user's emotional data.

[1380] 3. Preliminary organization of information

[1381] The server analyzes the received consultation content using natural language processing (NLP) technology to extract important keywords and phrases. It also obtains emotional data analyzed by an emotion recognition engine and creates provisional sorting information.

[1382] 4. Confirmation and additional hearings

[1383] The server then verifies the user based on the provisionally organized information and asks additional questions, such as, "Which part specifically would you like to know about?" The server then reorganizes the information based on the user's response.

[1384] 5. Emotion-Based Regulation

[1385] The server takes into account the user's emotional data and adjusts the content and order of follow-up questions, such as "What are you worried about?"

[1386] 6. Final information organization

[1387] The server then organizes the consultation details based on the information it finally obtains and writes the final information in a format, such as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1388] 7. Creating a sheet

[1389] The server then generates a sheet based on the final organized information, which also includes the user's emotional data.

[1390] 8. Check and correct

[1391] The sheet is sent to the terminal, where the user can check the contents and make corrections as necessary. The corrections are then sent back to the server, and the sheet is updated.

[1392] 9. Data Retention and Notification

[1393] The confirmed sheet is saved in the database, and the person in charge is notified of the new consultation content. The person in charge checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[1394] Examples and prompts

[1395] Scenario: Consultation about changing mobile phone plan

[1396] 1. Starting an interactive session

[1397] A user launches the app on their smartphone and starts chatting with a conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1398] 2. Hearing Phase

[1399] The user types "I'd like to know about mobile phone plans," and the device sends this information to the server. At the same time, the emotion recognition engine analyzes the user's emotions.

[1400] 3. Preliminary organization of information

[1401] The server extracts keywords such as "price plan" and "please tell me" and creates provisional sorting information along with emotional data.

[1402] 4. Confirmation and additional hearings

[1403] Based on the provisionally organized information, the server asks the user, "Which part specifically would you like to know?" The user enters, "I would like to know how to change from a smartphone to a tablet," and the device resends the request to the server.

[1404] 5. Emotion-Based Regulation

[1405] If the emotion recognition engine detects the user's anxiety, the server asks a follow-up question: "What are you worried about?"

[1406] 6. Final information organization

[1407] Based on the additional information and emotional data, the server organizes the final information into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1408] Example of input prompt for generative AI model

[1409] The functionality of this system can be simulated concretely by inputting the following prompt sentences into the generative AI model:

[1410] A user opens a smartphone app and types, "I'd like to know about my mobile phone plan." The system sends the initial message to the user, and the emotion engine analyzes the user's emotions. What action will the system take next?

[1411] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1412] Program processing flow

[1413] Step 1:

[1414] A user starts an application on a device such as a smartphone or PC and starts a session with the conversational AI. The device sends a session start request to the server. The server receives the request, prepares to start the session, and generates an initial message, "Please tell us your consultation details," and sends it to the device. The device receives this message and displays it to the user.

[1415] Step 2:

[1416] The user inputs the content of their inquiry into an input field on the device. For example, they might input "Please tell me about my mobile phone plan." The device then sends the input content as text data to the server. The server receives this text data and transfers it to an emotion recognition engine. The server then begins to apply natural language processing (NLP) to the user's input content. The input data includes the user's text data and emotion data analyzed by the emotion recognition engine.

[1417] Step 3:

[1418] The server obtains emotional data (e.g., "anxiety," "interest") obtained by an emotion recognition engine, and important keywords and phrases (e.g., "price plan," "please tell me") extracted using natural language processing technology. Through this process, the server understands the user's basic inquiry content and emotional state regarding "price plan." For example, if the input data is "please tell me about mobile phone rate plans," and the emotional data is "anxiety," the keywords "price plan" and "anxiety" are extracted.

[1419] Step 4:

[1420] The server creates provisionally organized information based on the keywords and emotion information extracted by the server, and organizes it according to a format. This provisionally organized information is sent to the terminal for confirmation by the user, for example, as "You're asking about a pricing plan, right?" The terminal receives this information and displays it to the user. The user can confirm the content and enter more specific questions and answers.

[1421] Step 5:

[1422] The server receives the user's additional input and analyzes it again using natural language processing technology and an emotion recognition engine. For example, if the user inputs, "I want to know how to switch from a smartphone to a tablet," the server re-extracts the keywords "how to switch" and "I want to know" as well as the user's emotion data. Based on this additional information, the server creates more detailed provisionally organized information and sends it to the device.

[1423] Step 6:

[1424] The server adjusts the content of the answers and follow-up questions based on the user's emotional data. For example, if the user enters "I want to know how to switch from a smartphone to a tablet" and the emotional data indicates "anxiety," the server generates a follow-up message asking "What are you worried about?" and sends it to the device. The user then enters an additional answer and sends it again to the server.

[1425] Step 7:

[1426] The server organizes the final consultation content based on the additional information and emotional data. For example, it organizes the information according to a format such as "Consultation topic: Price plan," "Specific content: How to switch from a smartphone to a tablet," and "Emotional state: Anxiety." The server generates a sheet based on this information and sends it to the device.

[1427] Step 8:

[1428] The user can check the generated sheet on the terminal and make corrections as necessary. Once the corrections are sent to the server, the server updates the sheet again and sends it to the terminal.

[1429] Step 9:

[1430] The server saves the confirmed consultation content sheet in a database. It also notifies the person handling the consultation that a new consultation content has been registered. The person handling the consultation checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[1431] (Application example 2)

[1432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1433] Conventional consultation systems systematically organize consultation content without considering the user's emotions, which results in an inability to fully grasp the user's true needs and concerns. Furthermore, in face-to-face consultations, it is difficult for staff to obtain emotional information in real time, which can prevent them from providing an appropriate response. In such situations, user satisfaction declines and efficient response becomes difficult.

[1434] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize provisional consultation details information according to a format and present it to the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for the terminal to start an interaction session and analyze the user's emotional data from the voice input in real time; a means for the smart glasses to recognize the customer's emotional state in real time and generate an appropriate response; a means for providing consultation response information to staff based on the final information; and a means for sharing information, such as emotional data, in real time so that staff can respond efficiently. This makes it possible to recognize emotions in real time through interaction with the user and provide appropriate responses.

[1435] The "means for a user to input the content of a consultation using a terminal" refers to an interface that allows a user to input the content of a consultation by text or voice using a mobile terminal, personal computer, or other device.

[1436] "Means for the server to use natural language processing technology to analyze the content of the user's inquiry and extract important keywords and phrases" refers to a technology in which the server uses natural language processing technology, a type of artificial intelligence technology, to analyze the content entered by the user and extract important information based on specific rules.

[1437] "Means for the server to organize provisional consultation information according to a format and present it on the terminal" refers to a technology in which the server provisionally organizes the consultation information based on keywords and phrases extracted by the server, compiles it according to a predetermined format, and displays the information on the user's terminal.

[1438] "Means for the server to ask additional questions, organize the final information, and generate a consultation content sheet" refers to a technology in which the server asks supplementary questions to the user, organizes the final consultation content based on the information obtained, formats the content, and generates it as a sheet.

[1439] "Means for the server to save the final sheet in a database and notify the responder" refers to a technology that saves the final consultation content sheet generated by the server in a database and notifies the staff member in charge of that information.

[1440] "Means for a terminal to initiate an interactive session and analyze user emotional data from voice input in real time" refers to a technology in which a user terminal initiates voice recognition and analyzes emotional data in real time based on information obtained from the user's voice input.

[1441] "A means for smart glasses to recognize a customer's emotional state in real time and generate an appropriate response" refers to a technology that uses an emotion recognition sensor built into smart glasses to analyze a customer's emotional state in real time from their facial expressions and voice, and generates an appropriate response based on that information.

[1442] "Means for providing staff with information on how to respond to inquiries based on final information" refers to a technique for providing staff with information that will enable them to respond appropriately to customers based on the final organized information on the content of the inquiries.

[1443] "Means for sharing information, such as emotional data, in real time to enable staff to respond efficiently" refers to technology that enables staff to share customer emotional data and other important information in real time, enabling efficient customer service.

[1444] The present invention relates to a system for efficiently responding to consultations while taking into consideration the user's emotions when making inquiries at stores, government offices, call centers, etc. Specific embodiments for carrying out the invention will be described below.

[1445] Overall system configuration

[1446] The system consists of the following main components:

[1447] 1. User terminal: A device such as a smartphone or personal computer (PC) that the user uses to input the details of their consultation.

[1448] 2. Server: Analyzes the user's inquiry using natural language processing technology, extracts important keywords and phrases, and organizes them.

[1449] 3. Database: Stores the final consultation sheet generated by the server.

[1450] 4. Emotion engine: Analyzes emotional data from the user's voice and facial expressions in real time.

[1451] 5. Smart Glasses: A device worn by staff that recognizes the emotional state of customers in real time and generates appropriate responses.

[1452] 6. Staff terminal: A device used by staff to check response information.

[1453] System Operation Overview

[1454] 1. Start an interactive session:

[1455] A user accesses the system using a smartphone or PC and starts a session with the conversational AI. The server receives the session start request and sends the initial message, "Please tell us your consultation details." to the user's device.

[1456] 2. Hearing Phase:

[1457] When the user inputs the content of their consultation, the device sends the input to the server. At the same time, the emotion engine analyzes the user's voice and facial expressions to obtain emotional data, and sends the results to the server.

[1458] 3. Preliminary organization of information:

[1459] The server analyzes the received user inquiry and uses natural language processing technology to extract important keywords and phrases. For example, it extracts keywords such as "price plan" and "please tell me." Based on this, it creates provisionally organized information and presents it to the user's device.

[1460] 4. Verification and Further Hearing:

[1461] The server then uses the preliminary information to ask additional questions, such as, "What specific part would you like to know about?" Any additional information entered by the user is analyzed in the same way.

[1462] 5. Emotion-based regulation:

[1463] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[1464] 6. Final information organization:

[1465] The server then organizes and formats the final consultation content based on the additional information and emotion data. For example, the consultation topic might be "Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[1466] 7. Generate and check the sheet:

[1467] The final organized information is generated as a correspondence sheet and presented to the user for confirmation. If the user enters any corrections, the information is reorganized and the sheet is updated.

[1468] 8. Data Retention and Notification:

[1469] The final sheet is saved in a database and sent to the customer service staff, who can then use the smart glasses to check the customer's emotional state in real time and respond accordingly.

[1470] Specific examples

[1471] As a concrete example, consider the following:

[1472] Scenario: Consultation about changing mobile phone plan

[1473] Example prompt:

[1474] "Please tell me about the administrative services available at the citizen service desk. Specifically, I would like information about tax consultations."

[1475] The system described above allows users to consult with confidence, and allows staff to take the user's feelings into consideration and respond more appropriately and quickly. Ultimately, the consultation content stored in the database will be used as learning data for future AI, which is expected to further improve the efficiency of problem-solving.

[1476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1477] Step 1:

[1478] A user accesses the system using a smartphone or PC terminal and starts an interactive session. As input, the server receives a session start request from the user, and sends the initial message "Please tell us your consultation details" to the terminal. As output, the initial message is displayed on the user's terminal.

[1479] Step 2:

[1480] The user inputs the content of their consultation into their smartphone or PC terminal using voice or text. The consultation content data from the user is received as input and sent to the server. The consultation content data received by the server is passed to the natural language processing engine as output, and analysis begins.

[1481] Step 3:

[1482] The server uses natural language processing technology to analyze the user's consultation content and extract important keywords and phrases. The natural language processing engine analyzes the consultation content data received as input and extracts important information. The extracted keywords and phrases are saved as provisionally organized information as output.

[1483] Step 4:

[1484] The server uses an emotion engine to analyze emotion data from the user's voice and facial expressions in real time. The user's voice and facial expression data are sent to the emotion engine as input. Emotion data is generated as the analysis result as output and added to the provisional sorting information.

[1485] Step 5:

[1486] The server organizes the provisionally organized information according to a format and presents it to the user's terminal. As input, the provisionally organized information and emotion data are integrated within the server. As output, the formatted provisionally organized information is sent to the user's terminal and displayed to the user.

[1487] Step 6:

[1488] The server asks a follow-up question and receives the answer from the user again. As input, a follow-up question based on the provisional sorting information is sent to the user's terminal. The user enters the answer, and the data is sent to the server. As output, the follow-up information is saved on the server.

[1489] Step 7:

[1490] The server organizes and formats the final consultation content based on the additional information and emotion data. The additional information and the existing provisionally organized information are integrated as input. The final consultation content sheet is generated as output.

[1491] Step 8:

[1492] The server sends the generated final sheet to the user's terminal, where the user can review and modify it. As input, the final sheet is presented to the user's terminal. The user enters modifications, which are sent to the server. As output, the modified final sheet is regenerated.

[1493] Step 9:

[1494] The server saves the final consultation content sheet in the database and notifies the staff. As input, the final confirmation sheet is saved in the database. As output, the new consultation content is notified in real time to the staff terminal and smart glasses.

[1495] Step 10:

[1496] Staff use smart glasses to check and respond to customers' emotional states in real time. As input, the smart glasses receive information from the database and also obtain customer emotional data in real time. As output, an appropriate response is provided to the customer.

[1497] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1499] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1500] [Fourth embodiment]

[1501] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1502] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1503] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1504] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1505] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1506] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1507] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1508] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1509] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1510] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1511] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1512] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1513] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1514] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the contents of users' inquiries, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[1515] Overall system configuration

[1516] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, and a database that ultimately stores the data. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content sent from the device and analyzes it using natural language processing technology.

[1517] Program processing

[1518] 1. Starting an interactive session

[1519] A user starts a session with a conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends an initial message to the device.

[1520] 2. Hearing Phase

[1521] The user inputs the details of their consultation through their device. This input is then sent from the device to the server, which then analyzes the received user input and extracts important keywords and phrases.

[1522] 3. Preliminary organization of information

[1523] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[1524] 4. Confirmation and additional hearings

[1525] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[1526] 5. Creating a sheet

[1527] The server generates a sheet according to the format based on the final consultation content, and presents this sheet to the user for final confirmation.

[1528] 6. Check and correct

[1529] The user checks the sheet contents and makes any necessary corrections. Once the final sheet is confirmed, it is sent to the server via the terminal.

[1530] 7. Data Retention and Notification

[1531] The server saves the final generated sheet in the database and notifies the staff that a new consultation has been registered.

[1532] Specific examples

[1533] Scenario: Consultation about changing mobile phone plan

[1534] 1. Starting an interactive session

[1535] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1536] 2. Hearing Phase

[1537] The user inputs "I would like to know about mobile phone plans." The device sends this input to the server.

[1538] 3. Preliminary organization of information

[1539] The server extracts the keywords "price plan" and "what would you like to know" and tentatively organizes the consultation content.

[1540] 4. Confirmation and additional hearings

[1541] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[1542] 5. Final information organization

[1543] The server analyzes the additional information and organizes it according to the format: "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1544] 6. Creating a sheet

[1545] The server generates a sheet based on this information and prompts the user to confirm it.

[1546] 7. Check and correct

[1547] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[1548] 8. Data Retention and Notification

[1549] The finalized sheet is saved in the database and staff are notified of the new consultation details.

[1550] This allows users to smoothly communicate their concerns to staff members even when they meet for the first time, enabling staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1551] The processing flow will be explained below.

[1552] Step 1:

[1553] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[1554] Step 2:

[1555] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[1556] Step 3:

[1557] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[1558] Step 4:

[1559] The terminal transmits the user's input to the server.

[1560] Step 5:

[1561] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases, such as "price plan" and "please tell me."

[1562] Step 6:

[1563] The server then uses the extracted keywords to organize the provisional information about the consultation according to a format, such as "Consultation topic: Price plan" and "Specific content: Please let me know."

[1564] Step 7:

[1565] The server sends the preliminary information to the terminal and presents it to the user. The server also asks follow-up questions, such as "Which part specifically would you like to know about?"

[1566] Step 8:

[1567] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[1568] Step 9:

[1569] The terminal sends the user's additional input to the server.

[1570] Step 10:

[1571] The server analyzes the received additional information and organizes it into a format as the final consultation content, for example, "Consultation topic: Price plan" and "Specific content: How to switch from a smartphone to a tablet."

[1572] Step 11:

[1573] The server generates a sheet in a predetermined format based on the final consultation content.

[1574] Step 12:

[1575] The server sends the generated sheet to the terminal for the user to check.

[1576] Step 13:

[1577] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[1578] Step 14:

[1579] The terminal sends the modifications to the server.

[1580] Step 15:

[1581] The server re-parses the modifications and regenerates the sheet.

[1582] Step 16:

[1583] The server saves the final sheet in the database and notifies the respondent that a new consultation has been registered.

[1584] Step 17:

[1585] The responder uses a dedicated terminal to check the pre-generated sheet and prepares to respond efficiently when the user visits.

[1586] This allows the user to smoothly communicate the content of the consultation, and the person in charge can also respond efficiently.

[1587] Example 1

[1588] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1589] In modern society, making effective use of waiting time at stores, government offices, call centers, etc., and efficiently listening to and organizing user inquiries is essential to improving business efficiency and customer satisfaction. However, there is a lack of appropriate systems to achieve this, and most responses are handled manually, resulting in a waste of time and effort. Therefore, there is a need for a system that can quickly and accurately receive, analyze, organize, and ultimately provide user inquiries to the person in charge.

[1590] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1591] In this invention, the server includes means for a user to input consultation details using an information processing device, means for a computer to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases, means for the computer to organize provisionally organized information of the consultation details according to a format and present it to the information processing device, means for the computer to generate an initial message and send it to the information processing device, means for the computer to ask additional questions, organize the final information, and generate a consultation details sheet, and means for the computer to save the final sheet in a database and notify the service provider. This makes it possible to efficiently hear the user's consultation details, organize them, and provide them to the person who will handle the consultation.

[1592] "User" refers to an individual or corporation who uses the service and inputs the details of their inquiry.

[1593] An "information processing device" is a device used by a user to input consultation details, and specifically refers to a smartphone, PC, tablet, etc.

[1594] "Server" refers to a computer system that receives data sent by users and analyzes, processes, stores, and notifies them.

[1595] "Natural language processing technology" refers to technology for analyzing user input text and extracting important keywords and phrases.

[1596] "Provisionally organized information" refers to information that temporarily organizes consultation content based on keywords and phrases extracted using natural language processing technology.

[1597] "Format" refers to a standard format or structure for organizing consultation content in a consistent manner.

[1598] "Initial message" refers to the first message that the server sends to the user at the beginning of a session.

[1599] "Sheet" refers to a document or data format that contains the final, organized content of the consultation.

[1600] "Database" refers to a data management system for storing the final generated sheets and other related data.

[1601] "Service provider" refers to an individual or organization that responds to and provides support based on the content of inquiries from users.

[1602] This invention relates to a system that effectively utilizes waiting time at stores, government offices, call centers, etc., and efficiently listens to and organizes the details of users' inquiries and provides them to staff. This system consists of an information processing device used by users, a server that receives and analyzes the details of the inquiries, and a database that stores the data.

[1603] A user initiates a session with a conversational AI using an information processing device such as a smartphone or PC. The server receives the user's consultation content sent from the terminal and analyzes it using natural language processing technology (e.g., Google Cloud NLP API). Specific embodiments for implementing this invention are described below.

[1604] When the server receives the consultation content entered by the user, it uses natural language processing technology to extract important keywords and phrases. This allows the server to accurately understand the user's intent and the content of the question. The extracted keywords and phrases are used to tentatively organize the consultation content.

[1605] The provisionally organized information is then organized into a standard format by the server and presented to the user via their terminal. This allows the user to easily confirm the content of their consultation. The server also asks additional questions at this stage to obtain more detailed information.

[1606] The information obtained from the additional interviews is analyzed in the same way and organized into a format as the final consultation content. The server generates a sheet based on this final information and asks the user to confirm it again. The user checks the sheet contents and corrects them as necessary to confirm the exact consultation content.

[1607] The server stores the confirmed consultation details in a database and sends a notification to the staff, enabling them to respond promptly to the newly registered consultation details.

[1608] Specific examples

[1609] Scenario: Consultation about changing mobile phone plan

[1610] 1. The user launches the smartphone app and taps the "Price Plan Consultation" button. The app then displays the message, "Please tell us your inquiry."

[1611] 2. The terminal sends a session initiation request to the server and sends the user ID.

[1612] 3. The server starts the session and sends an initial message to the terminal, for example, "Please tell us what you would like to discuss with us."

[1613] 4. The user types, "I'd like to know about mobile phone plans."

[1614] 5. The device sends the input information to the server.

[1615] 6. The server analyzes the received message using natural language processing technology and extracts important keywords such as "price plan" and "please tell me."

[1616] 7. The server generates provisional information such as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan" and sends it to the terminal.

[1617] 8. The terminal displays the provisional sorting information to the user.

[1618] 9. The user adds, "I want to know how to change from a smartphone to a tablet."

[1619] 10. The device sends the additional input to the server.

[1620] 11. The server analyzes again and generates "Consultation topic: Pricing plan, Details: How to change from smartphone to tablet."

[1621] 12. The server generates a sheet based on the final consultation content and sends it to the terminal.

[1622] 13. The terminal displays the sheet to the user, who then checks and modifies it.

[1623] 14. The user amends the question to say, "I want to change the Wi-Fi in addition to the tablet."

[1624] 15. The device sends the modified content to the server.

[1625] 16. The server generates a new sheet reflecting the changes and sends it to the terminal.

[1626] 17. The terminal displays the sheet again to the user for final confirmation.

[1627] 18. The server saves the confirmed sheet in the database and notifies the staff that a new consultation has been registered.

[1628] Prompt Sentence Examples

[1629] User prompt: "I want to change my mobile phone plan. How do I do this?"

[1630] Server's initial response: "Please tell us what you would like to discuss."

[1631] Additional prompt: "I want to know how to change from a smartphone to a tablet."

[1632] In this way, users can smoothly communicate their concerns to staff members they meet for the first time, enabling the staff members to respond efficiently. In addition, the data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1633] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1634] Step 1:

[1635] Starting an interactive session

[1636] The user uses an information processing device (smartphone or PC) to start a session with the conversational AI.

[1637] Input: User clicks a "Start Session" button on an application or web page.

[1638] The terminal sends a session initiation request to the server, along with data including the user ID and a timestamp of the session initiation.

[1639] The server receives the request and performs processing to initialize the session, such as generating a new session ID and performing initial settings.

[1640] Output: The server generates an initial message and sends it to the user. The initial message, such as "Please tell us what you would like to discuss," is displayed on the terminal.

[1641] Step 2:

[1642] Hearing phase

[1643] The user inputs the content of their inquiry into the terminal. For example, they might input, "I'd like to know about mobile phone rate plans."

[1644] Input: A user enters text into an input field on an application.

[1645] The terminal sends the input text to the server.

[1646] The server analyzes the received text data using natural language processing (NLP) technology. Specifically, it uses the Google Cloud NLP API to extract important keywords from the text (e.g., "pricing plan" or "please tell me").

[1647] Output: Extracted keywords and analysis results.

[1648] Step 3:

[1649] Provisional organization of information

[1650] The server then organizes the consultation content into a temporary format based on the extracted keywords. For example, it could organize the content as "Consultation topic: Pricing plan, Details: Please tell me about the pricing plan."

[1651] Input: Keywords extracted by natural language processing and analysis results.

[1652] The server generates a message to present the provisional arrangement information to the user.

[1653] Output: Preliminary organized consultation message.

[1654] Step 4:

[1655] Confirmation and additional hearings

[1656] The user checks the provisional information provided by the server and enters additional information. For example, the user might enter, "I want to know how to change from a smartphone to a tablet."

[1657] Input: The user enters additional information and the device sends it to the server.

[1658] The server then analyzes the received data using natural language processing technology and formats the final consultation content based on the additional information.

[1659] Output: Information organized as the final consultation content.

[1660] Step 5:

[1661] Sheet Generation

[1662] The server then generates a sheet based on the final organized consultation content. For example, it creates a sheet with the following information: "Consultation topic: Pricing plan, Details: How to switch from smartphone to tablet."

[1663] Input: The final consultation details.

[1664] The server sends the generated sheet to the terminal for presentation to the user.

[1665] Output: The generated sheet.

[1666] Step 6:

[1667] Check and fix

[1668] The user checks the sheet displayed on the device and makes any necessary changes. For example, they might change the settings to "I want to change the Wi-Fi in addition to the tablet."

[1669] Input: The user inputs the correction information, and the terminal sends it to the server.

[1670] The server receives the revised content and updates the sheet again to generate the final version.

[1671] Output: The final sheet with the modifications reflected.

[1672] Step 7:

[1673] Data Retention and Notification

[1674] The server stores the final sheet in a database, including the consultation content, user identification information, time information, etc.

[1675] Input: Final sheet and associated metadata.

[1676] The server notifies the service provider that a new consultation has been registered. Notification methods include email and push notification.

[1677] Output: Information stored in the database and notification messages.

[1678] By this processing flow, the contents of the user's inquiry are efficiently and accurately heard and provided to the person who will ultimately handle the matter.

[1679] (Application example 1)

[1680] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1681] In physical stores, there is a demand for a system that allows users to make effective use of their waiting time and efficiently consult with staff about product information and order details. In particular, it is necessary to improve in-store service by quickly and accurately conveying the content of users' inquiries to staff. There is also a demand for a means for users to smoothly convey the content of their inquiries to staff members they meet for the first time. Furthermore, there is a demand for improving store operational efficiency by using conversational AI to automatically organize the content of inquiries and respond in a timely manner.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1683] In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize the provisional consultation details according to a format and present it on the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for a user to input product information and order details using a smartphone at a physical store while waiting; a means for the server to analyze the user's input in real time using an interactive AI and generate a provisional format; a means for the server to reflect the user's confirmation and corrections and regenerate the final format; and a means for the server to notify staff at the physical store based on the final format. This allows users to effectively utilize their waiting time at the physical store and efficiently consult about product information and order details. Furthermore, the server automatically organizes the consultation details and quickly and accurately conveys them to staff, thereby improving in-store service and operational efficiency.

[1684] "User" refers to a person who uses the system to input the details of a consultation and receive services.

[1685] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1686] "Consultation content" refers to information related to in-store services, such as product information and order details entered by the user.

[1687] "Server" refers to a computer system that receives and analyzes input from users and notifies staff.

[1688] "Natural language processing technology" refers to the technology of analyzing text data entered by the user and extracting important keywords and phrases.

[1689] "Provisional sorting information" refers to information that sorts out the consultation content input by the user at an early stage.

[1690] "Format" refers to a structure or template for organizing consultation content in a certain format.

[1691] "Conversational AI" refers to an artificial intelligence system that engages in natural dialogue with users and collects and analyzes information.

[1692] "Waiting time" refers to the time a user waits to receive service at a physical store.

[1693] "Notification" refers to the action of informing staff of server-generated information.

[1694] "Physical store" refers to a store or service location that has a physical presence.

[1695] The "provisional format" refers to a format that is temporarily organized at the stage of initial analysis of the content of the user's inquiry.

[1696] The "final format" refers to the format of the consultation content that has been confirmed after being checked and corrected by the user.

[1697] The term "responder" refers to a store staff member who responds to the user upon receiving a notification from the server.

[1698] The present invention provides a system that enables users to effectively utilize their waiting time in a physical store and consult with them about product information and order details. Specific embodiments for carrying out the invention are described below.

[1699] Overall system configuration

[1700] The system mainly consists of the following components:

[1701] 1. Device: The device used by the user, such as a smartphone or tablet.

[1702] 2. Server: A computer system that is responsible for receiving user input, analyzing it, and notifying staff.

[1703] 3. Database: This is where the final consultation sheet is stored.

[1704] Hardware and software used

[1705] Hardware: Smartphones, tablets, server computers

[1706] Software: Python, Flask framework, SQLite database, OpenAI API

[1707] Explanation of program processing

[1708] Starting an interactive session

[1709] The user starts the smartphone app in the waiting area of ​​a physical store and starts a session with the conversational AI. The server then sends an initial message saying, "Please tell us what you would like to discuss," and the session begins.

[1710] Hearing phase

[1711] The user inputs the details of their consultation via their device. This input is then sent from the device to the server, which then analyzes the received user input and uses natural language processing technology to extract important keywords and phrases.

[1712] Generation of provisional arrangement information

[1713] The server then organizes the consultation details into a provisional format based on the extracted information. This provisionally organized information is then presented to the user via the terminal.

[1714] Confirmation and additional hearings

[1715] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters the additional details and sends them back to the server from their device. The server analyzes the received additional information and organizes it into a format as the final consultation content.

[1716] Generating the final format

[1717] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet contents and makes corrections as necessary. Once the final sheet is confirmed, it is sent to the server via the terminal.

[1718] Data Retention and Notification

[1719] The server then saves the final generated sheet in a database. It also notifies the responder that a new consultation has been registered. This allows the staff member to understand the specific details of the user's consultation in advance, enabling them to respond quickly and appropriately.

[1720] Specific examples

[1721] For example, if a user types "Tell me about new smartphone models," the following prompt sentence is used:

[1722] Prompt Sentence Examples

[1723] Please organize the following information: I would like to know about the new smartphone model.

[1724] The server uses this prompt to call the generative AI model, extracting the keywords "new model" and "please tell me," and organizing them into a temporary format. This information is then presented to the user, who is then asked to enter additional details. Through this process, the final format is generated, and the user's inquiry is notified to the store staff.

[1725] According to the above embodiment, users can effectively use their waiting time to input the details of their consultation, and staff can efficiently respond to users' consultations.

[1726] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1727] Step 1:

[1728] Starting an interactive session

[1729] The user starts a session with the conversational AI using a device. When the user launches the smartphone app, a request to start the session is sent from the device to the server. The server starts the session and sends the initial message "Please tell us your consultation." to the device.

[1730] Input: User's session initiation request

[1731] Output: Initial message from the server

[1732] Step 2:

[1733] Hearing phase

[1734] The user inputs their concerns through their device. This input is sent from the device to the server, which then receives it. The server uses a generative AI model and natural language processing technology to analyze the user's input and extract important keywords and phrases.

[1735] Input: User's consultation content input

[1736] Output: Extracted keywords and phrases

[1737] Step 3:

[1738] Generation of provisional arrangement information

[1739] The server organizes the consultation content into a provisional format based on the extracted keywords and phrases. This provisionally organized information is then presented to the user via their terminal.

[1740] Input: Extracted keywords or phrases

[1741] Output: Provisional arrangement information

[1742] Step 4:

[1743] Confirmation and additional hearings

[1744] The server presents the provisionally organized information to the user and asks for additional detailed information. The user enters specific additional information, which is then sent back to the server from the terminal. The server analyzes the additional information and organizes it into a format as the final consultation content.

[1745] Input: Additional hearing information for the user

[1746] Output: The final parsed consultation

[1747] Step 5:

[1748] Generating the final format

[1749] The server generates a sheet according to the format based on the final consultation content. This sheet is presented to the user for final confirmation. The user checks the sheet content and makes corrections as necessary.

[1750] Input: The final parsed consultation content

[1751] Output: The final sheet presented to the user

[1752] Step 6:

[1753] Check and fix

[1754] The user checks the sheet contents and makes any necessary corrections. The corrected contents are sent back to the server via the terminal, and the server regenerates the final sheet based on the updated information.

[1755] Input: User-modified information

[1756] Output: Updated final sheet

[1757] Step 7:

[1758] Data Retention and Notification

[1759] The server saves the final generated sheet in a database and notifies the responder that a new consultation has been registered, allowing the staff member to understand the specific details of the user's consultation in advance.

[1760] Input: Updated final sheet

[1761] Output: Sheets saved in the database and notifications to responders

[1762] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1763] The present invention is a system for effectively utilizing waiting time at stores, government offices, call centers, etc., by efficiently listening to the user's consultation details and feelings, organizing them, and providing them to staff. Specific embodiments for carrying out the present invention will be described below.

[1764] Overall system configuration

[1765] The system mainly consists of a device used by the user, a server that receives and analyzes the consultation content, a database that ultimately stores the data, and an emotion engine that recognizes the user's emotions. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device and analyzes each.

[1766] Program processing

[1767] 1. Starting an interactive session

[1768] The user starts a session with the conversational AI using a device. The device sends a request to start the session to the server. The server starts the session and sends the initial message "Please tell us your consultation details" to the device.

[1769] 2. Hearing Phase

[1770] The user inputs the content of their inquiry through the device. For example, they might input, "I'd like to know about my mobile phone plan." The device then sends the user's input to the server. The emotion engine also analyzes the emotional data of the user's input.

[1771] 3. Preliminary organization of information

[1772] The server analyzes the received user input and uses natural language processing technology to extract important keywords and phrases. For example, keywords such as "price plan" and "please tell me." The emotion engine recognizes the user's emotions (e.g., joy, anger, sadness, etc.) and sends that information to the server.

[1773] 4. Confirmation and additional hearings

[1774] The server takes into account the extracted keywords and the user's emotional information, and organizes the provisionally organized information in a format. This provisionally organized information is then presented to the user again via the terminal. Depending on the user's emotions, the server may send additional messages, such as a message to help them relax. The server may also ask additional questions, such as, "Which part specifically would you like to know more about?"

[1775] 5. Emotion-Based Regulation

[1776] Based on the user's emotional data, the server will ask additional questions and adjust the content presented. For example, if the user is feeling anxious, the server will ask a follow-up question such as, "What are you worried about?"

[1777] 6. Final information organization

[1778] The server then organizes the final consultation content into a format based on the additional information and emotion data received again. For example, it could organize the content into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1779] 7. Creating a sheet

[1780] The server generates a sheet in a predetermined format based on the final consultation content, which also includes the user's emotional data.

[1781] 8. Check and correct

[1782] The server sends the generated sheet to the terminal for the user to check. The user checks the sheet contents and makes corrections as necessary. These corrections are sent back to the server, and the sheet is regenerated.

[1783] 9. Data Retention and Notification

[1784] The server saves the final generated sheet in a database. It also notifies staff that a new consultation has been registered. Staff use dedicated terminals to check the pre-generated sheet and prepare to respond efficiently when the user visits. When responding, the server also takes into account the user's emotional information and provides an appropriate response.

[1785] Specific examples

[1786] Scenario: Consultation about changing mobile phone plan

[1787] 1. Starting an interactive session

[1788] The user opens the smartphone app and starts chatting with the conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1789] 2. Hearing Phase

[1790] The user types, "Please tell me about my mobile phone plan." The device sends this input to the server. The emotion engine then analyzes the emotion (e.g., nervousness, interest, etc.) based on the user's input.

[1791] 3. Preliminary organization of information

[1792] The server extracts keywords such as "price plan" and "what would you like to know" and tentatively organizes the content of the consultation. It also incorporates emotional information obtained from the emotion engine.

[1793] 4. Confirmation and additional hearings

[1794] The server presents the provisional information to the user and asks, "You would like to know about the pricing plan. What specific details would you like to know?" The user enters, "I would like to know how to switch from a smartphone to a tablet." This information is again sent to the server via the device.

[1795] 5. Emotion-Based Regulation

[1796] If the emotion engine detects that the user's emotions are a little unstable, the server will follow up with, "What are you worried about? Please tell us more."

[1797] 6. Final information organization

[1798] Based on the additional information and emotion data, the server updates the format and organizes it as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1799] 7. Creating a sheet

[1800] The server generates a sheet based on this information and prompts the user to confirm it.

[1801] 8. Check and correct

[1802] The user checks the sheet and enters a correction, saying, "I want to change the Wi-Fi in addition to the tablet." This correction information is sent again to the server, and the sheet is regenerated.

[1803] 9. Data Retention and Notification

[1804] The finalized sheet is saved in a database, and the person in charge is notified of the new consultation content and emotional data. Staff members check the information in advance and prepare to respond appropriately when the patient visits.

[1805] This format allows users to smoothly communicate their concerns to staff members even when they meet for the first time, and allows staff members to respond efficiently while taking into consideration the user's feelings. The data accumulated in the system will be useful for future AI learning, which is expected to shorten the time it takes to resolve problems.

[1806] The processing flow will be explained below.

[1807] Step 1:

[1808] A user initiates a session with a conversational AI using a device, which sends a request to start the session to the server.

[1809] Step 2:

[1810] The server starts the session and sends the initial message "Please tell us what you would like to discuss" to the user's terminal.

[1811] Step 3:

[1812] The user uses the terminal to input the content of their inquiry. For example, they might input, "I'd like to know about mobile phone rate plans."

[1813] Step 4:

[1814] The terminal transmits the user's input to the server, and at the same time, the emotion engine collects the user's emotion data (voice tone, input speed, phrase selection, etc.) and transmits it to the server.

[1815] Step 5:

[1816] The server analyzes the received user input and uses natural language processing techniques to extract important keywords and phrases.

[1817] Step 6:

[1818] The emotion engine analyzes the received emotion data and identifies the user's emotional state (e.g., joy, anxiety, anger, etc.). Based on this information, the server tentatively organizes the user's consultation details.

[1819] Step 7:

[1820] The server organizes the provisionally organized information based on the extracted keywords and emotion data and displays it on the terminal. For example, the information might be "Consultation topic: Price plan," "Specific content: Please tell me," and "Emotional state: Anxiety."

[1821] Step 8:

[1822] The server can ask additional questions or make additional considerations based on the user's emotional state, for example, "If you would like to know more about our pricing plans, we can explain them in detail. Please let us know if you have any concerns."

[1823] Step 9:

[1824] The user enters additional details, for example, "I want to know how to change from a smartphone to a tablet."

[1825] Step 10:

[1826] The terminal sends additional user input to the server, and the emotion engine collects new emotion data and sends it to the server.

[1827] Step 11:

[1828] The server analyzes the received additional information and formats it into the final consultation content. The emotion engine analyzes the user's emotional state again and includes the emotion data in the final sheet.

[1829] Step 12:

[1830] The server generates a final consultation content sheet and sends it to the terminal for the user to confirm. The sheet contains, for example, the following information:

[1831] Consultation topic: Pricing plan

[1832] Specific content: How to change from smartphone to tablet

[1833] Emotional state: Anxiety

[1834] Step 13:

[1835] The user checks the sheet contents and makes any necessary changes. For example, they might enter, "I want to change the Wi-Fi in addition to the tablet."

[1836] Step 14:

[1837] The terminal transmits the corrections to the server, and the emotion engine collects the user's emotion data again.

[1838] Step 15:

[1839] The server analyzes the modifications and regenerates the sheet, including the final emotion data from the emotion engine.

[1840] Step 16:

[1841] The server saves the finalized sheet in the database and notifies the person in charge that the new consultation content and emotional data have been registered.

[1842] Step 17:

[1843] The responder uses a dedicated terminal to check the pre-generated sheet and emotion data, and prepares to respond efficiently and appropriately when the user visits.

[1844] Example 2

[1845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1846] In recent years, there has been a demand for stores, government offices, call centers, and other facilities to make effective use of users' waiting time and efficiently handle inquiries. However, current systems have difficulty grasping users' emotions and the details of their inquiries, which can lead to a decline in user satisfaction. Furthermore, they are unable to respond in a way that takes users' emotions into appropriate consideration, which can result in problems and dissatisfaction. It is necessary to solve these problems and provide a system that provides high satisfaction to both users and staff.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1848] In this invention, the server includes: means for a user to use a terminal to start a session with an interactive artificial intelligence; means for the server to receive the user's consultation content, analyze it using natural language processing technology, and extract important keywords and phrases; means for the server to analyze the user's emotional data using an emotion recognition engine and acquire emotional information; means for the server to create provisional organized information based on the extracted keywords and emotional information, organize it according to a format, and present it to the terminal; means for the server to ask the user additional questions, organize the final information, and generate a consultation content sheet; means for the server to store the final generated sheet in a database and notify the person handling the consultation; and means for the person handling the consultation to respond efficiently while taking the user's emotional information into consideration. This makes it possible to understand the user's emotions and detailed consultation content, and respond accurately and efficiently.

[1849] A "terminal" is an electronic device used by a user, which is a means for initiating a session with an interactive artificial intelligence and inputting and displaying consultation details.

[1850] "Conversational artificial intelligence" refers to an algorithm or system that allows a user to interact with it in natural language, analyzing input from the user and generating appropriate responses.

[1851] A "server" is a computer system that receives, processes, and analyzes data sent from terminals via a network.

[1852] "Natural language processing technology" is a general term for technology that analyzes, understands, and generates human language, and is used to extract important keywords and phrases.

[1853] An "emotion recognition engine" is software or an algorithm that analyzes emotions from user input data and obtains that information.

[1854] "Temporarily organized information" is information that has been temporarily organized based on keywords and emotional information extracted from the contents of the user's consultation.

[1855] A "format" is a standardized structure or form for organizing and presenting information.

[1856] A "consultation sheet" is a written or electronic form containing the final organized consultation and emotional information.

[1857] A "database" is a system for efficiently storing, managing, and searching accumulated data.

[1858] The "responder" is a person in charge of responding to the content of the inquiry from the user, and is a person who receives a notification from the server and takes an appropriate action.

[1859] This invention is a system in which a user inputs the details of a consultation using a terminal, a server analyzes and organizes the input, and generates a final consultation details sheet to notify the person handling the consultation. Specific embodiments for implementing this system are described below.

[1860] Overall system configuration

[1861] The system consists of a device used by the user, a server, a database, and an emotion recognition engine. The user starts a session with the conversational AI using a device such as a smartphone or PC. The server receives the user's consultation content and emotion data sent from the device, analyzes them, and organizes the user's consultation content.

[1862] System hardware and software examples

[1863] Device: A user device such as a smartphone, tablet, or PC.

[1864] Server: A high-performance computer system (e.g., a cloud server such as AWS or Azure).

[1865] Database: A SQL or NoSQL database (e.g. MySQL, MongoDB).

[1866] Emotion recognition engine: Software that analyzes emotions (e.g., Microsoft Azure Cognitive Services, IBM Watson, etc.).

[1867] Natural Language Processing (NLP): Technology for analyzing text (e.g., Google Natural Language API, spaCy).

[1868] Explanation of program processing

[1869] 1. Starting an interactive session

[1870] The user starts a session with the conversational AI using a terminal. The terminal sends a session start request to the server, and the server starts the session. The server sends the initial message "Please tell us your consultation details" to the terminal and displays it to the user.

[1871] 2. Hearing Phase

[1872] The user inputs the content of their inquiry through the device. For example, if they input "I'd like to know about my mobile phone plan," the device sends this input to the server. At the same time, the emotion recognition engine analyzes the user's emotional data.

[1873] 3. Preliminary organization of information

[1874] The server analyzes the received consultation content using natural language processing (NLP) technology to extract important keywords and phrases. It also obtains emotional data analyzed by an emotion recognition engine and creates provisional sorting information.

[1875] 4. Confirmation and additional hearings

[1876] The server then verifies the user based on the provisionally organized information and asks additional questions, such as, "Which part specifically would you like to know about?" The server then reorganizes the information based on the user's response.

[1877] 5. Emotion-Based Regulation

[1878] The server takes into account the user's emotional data and adjusts the content and order of follow-up questions, such as "What are you worried about?"

[1879] 6. Final information organization

[1880] The server then organizes the consultation details based on the information it finally obtains and writes the final information in a format, such as "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1881] 7. Creating a sheet

[1882] The server then generates a sheet based on the final organized information, which also includes the user's emotional data.

[1883] 8. Check and correct

[1884] The sheet is sent to the terminal, where the user can check the contents and make corrections as necessary. The corrections are then sent back to the server, and the sheet is updated.

[1885] 9. Data Retention and Notification

[1886] The confirmed sheet is saved in the database, and the person in charge is notified of the new consultation content. The person in charge checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[1887] Examples and prompts

[1888] Scenario: Consultation about changing mobile phone plan

[1889] 1. Starting an interactive session

[1890] A user launches the app on their smartphone and starts chatting with a conversational AI. The server then sends a message saying, "Please tell us what you would like to discuss."

[1891] 2. Hearing Phase

[1892] The user types "I'd like to know about mobile phone plans," and the device sends this information to the server. At the same time, the emotion recognition engine analyzes the user's emotions.

[1893] 3. Preliminary organization of information

[1894] The server extracts keywords such as "price plan" and "please tell me" and creates provisional sorting information along with emotional data.

[1895] 4. Confirmation and additional hearings

[1896] Based on the provisionally organized information, the server asks the user, "Which part specifically would you like to know?" The user enters, "I would like to know how to change from a smartphone to a tablet," and the device resends the request to the server.

[1897] 5. Emotion-Based Regulation

[1898] If the emotion recognition engine detects the user's anxiety, the server asks a follow-up question: "What are you worried about?"

[1899] 6. Final information organization

[1900] Based on the additional information and emotional data, the server organizes the final information into "Consultation topic: Pricing plan" and "Specific content: How to switch from a smartphone to a tablet."

[1901] Example of input prompt for generative AI model

[1902] The functionality of this system can be simulated concretely by inputting the following prompt sentences into the generative AI model:

[1903] A user opens a smartphone app and types, "I'd like to know about my mobile phone plan." The system sends the initial message to the user, and the emotion engine analyzes the user's emotions. What action will the system take next?

[1904] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1905] Program processing flow

[1906] Step 1:

[1907] A user starts an application on a device such as a smartphone or PC and starts a session with the conversational AI. The device sends a session start request to the server. The server receives the request, prepares to start the session, and generates an initial message, "Please tell us your consultation details," and sends it to the device. The device receives this message and displays it to the user.

[1908] Step 2:

[1909] The user inputs the content of their inquiry into an input field on the device. For example, they might input "Please tell me about my mobile phone plan." The device then sends the input content as text data to the server. The server receives this text data and transfers it to an emotion recognition engine. The server then begins to apply natural language processing (NLP) to the user's input content. The input data includes the user's text data and emotion data analyzed by the emotion recognition engine.

[1910] Step 3:

[1911] The server obtains emotional data (e.g., "anxiety," "interest") obtained by an emotion recognition engine, and important keywords and phrases (e.g., "price plan," "please tell me") extracted using natural language processing technology. Through this process, the server understands the user's basic inquiry content and emotional state regarding "price plan." For example, if the input data is "please tell me about mobile phone rate plans," and the emotional data is "anxiety," the keywords "price plan" and "anxiety" are extracted.

[1912] Step 4:

[1913] The server creates provisionally organized information based on the keywords and emotion information extracted by the server, and organizes it according to a format. This provisionally organized information is sent to the terminal for confirmation by the user, for example, as "You're asking about a pricing plan, right?" The terminal receives this information and displays it to the user. The user can confirm the content and enter more specific questions and answers.

[1914] Step 5:

[1915] The server receives the user's additional input and analyzes it again using natural language processing technology and an emotion recognition engine. For example, if the user inputs, "I want to know how to switch from a smartphone to a tablet," the server re-extracts the keywords "how to switch" and "I want to know" as well as the user's emotion data. Based on this additional information, the server creates more detailed provisionally organized information and sends it to the device.

[1916] Step 6:

[1917] The server adjusts the content of the answers and follow-up questions based on the user's emotional data. For example, if the user enters "I want to know how to switch from a smartphone to a tablet" and the emotional data indicates "anxiety," the server generates a follow-up message asking "What are you worried about?" and sends it to the device. The user then enters an additional answer and sends it again to the server.

[1918] Step 7:

[1919] The server organizes the final consultation content based on the additional information and emotional data. For example, it organizes the information according to a format such as "Consultation topic: Price plan," "Specific content: How to switch from a smartphone to a tablet," and "Emotional state: Anxiety." The server generates a sheet based on this information and sends it to the device.

[1920] Step 8:

[1921] The user can check the generated sheet on the terminal and make corrections as necessary. Once the corrections are sent to the server, the server updates the sheet again and sends it to the terminal.

[1922] Step 9:

[1923] The server saves the confirmed consultation content sheet in a database. It also notifies the person handling the consultation that a new consultation content has been registered. The person handling the consultation checks the sheet on a dedicated terminal and prepares to respond efficiently when the user visits.

[1924] (Application example 2)

[1925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1926] Conventional consultation systems systematically organize consultation content without considering the user's emotions, which results in an inability to fully grasp the user's true needs and concerns. Furthermore, in face-to-face consultations, it is difficult for staff to obtain emotional information in real time, which can prevent them from providing an appropriate response. In such situations, user satisfaction declines and efficient response becomes difficult.

[1927] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input consultation details using a terminal; a means for the server to analyze the user's consultation details using natural language processing technology and extract important keywords and phrases; a means for the server to organize provisional consultation details information according to a format and present it to the terminal; a means for the server to ask additional questions, organize the final information, and generate a consultation details sheet; a means for the server to save the final sheet in a database and notify the person handling the consultation; a means for the terminal to start an interaction session and analyze the user's emotional data from the voice input in real time; a means for the smart glasses to recognize the customer's emotional state in real time and generate an appropriate response; a means for providing consultation response information to staff based on the final information; and a means for sharing information, such as emotional data, in real time so that staff can respond efficiently. This makes it possible to recognize emotions in real time through interaction with the user and provide appropriate responses.

[1928] The "means for a user to input the content of a consultation using a terminal" refers to an interface that allows a user to input the content of a consultation by text or voice using a mobile terminal, personal computer, or other device.

[1929] "Means for the server to use natural language processing technology to analyze the content of the user's inquiry and extract important keywords and phrases" refers to a technology in which the server uses natural language processing technology, a type of artificial intelligence technology, ...

Claims

1. A means for a user to input the content of a consultation using a terminal; The server uses natural language processing technology to analyze the user's inquiry and extract important keywords and phrases. A means for the server to organize provisionally organized information on consultation contents according to a format and present it on the terminal; A means for the server to ask additional questions, organize the final information, and generate a consultation content sheet; The system includes a means for the server to store the final sheet in a database and notify the responder.

2. 2. The system according to claim 1, further comprising means for providing a user interface including input and confirmation of consultation contents.

3. 2. The system according to claim 1, further comprising means for using the accumulated consultation content data to learn so that the artificial intelligence technology will automatically present future problem-solving proposals.

Images