System

Abstract

A system is provided.SOLUTION: A system comprising: voice recognition means for receiving a customer's voice and converting the voice data into text data; natural language processing means for analyzing the text data and classifying an inquiry content; processing means for determining an appropriate response based on the inquiry content; means for generating an answer according to the appropriate response or means for transferring a call to an appropriate person in charge; and voice synthesis means for converting the answer into voice data and providing it to the customer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, efficient and prompt responses to telephone inquiries are required for customer support in a variety of industries. However, conventional IVR (Interactive Voice Response) systems require customers to select the appropriate option from multiple options, which can be cumbersome for customers. Furthermore, even when an operator responds, it can take time to understand the content of the initial inquiry, making it difficult to respond quickly enough to provide a satisfactory result to the customer. The present invention aims to solve these problems and improve the efficiency and satisfaction of customer inquiries. [Means for solving the problem]

[0005] The present invention provides a system including a speech recognition unit that receives customer voice and converts the voice data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry content, a processing unit that determines an appropriate response based on the inquiry content, a unit that generates an answer according to the appropriate response or a unit that transfers the call to an appropriate representative, and a speech synthesis unit that converts the answer into voice data and provides it to the customer. This system allows customers to make inquiries smoothly without going through a cumbersome IVR, which is expected to improve customer satisfaction. Furthermore, by quickly transferring calls to the appropriate representative, the burden on operators can be reduced and inquiry response efficiency can be improved.

[0006] "Customer" means any person or entity making an inquiry using the System.

[0007] "Voice recognition means" refers to a function or technology that converts voice data input by a customer into text data.

[0008] "Voice data" refers to data that represents the customer's speech in digital form.

[0009] "Text data" refers to data expressed in characters that has been converted by a speech recognition means.

[0010] "Natural language processing means" refers to technology or functions that analyze text data, understand its content, and classify it.

[0011] "Inquiry content" refers to questions or requests made by customers to the system.

[0012] "Processing means" refers to the function or algorithm that determines the appropriate response based on the input data or inquiry.

[0013] "Means for generating an answer" refers to a technology or function that automatically generates an answer to the inquiry.

[0014] "Staff member" refers to the operator or expert who responds to customer inquiries through the system.

[0015] "Call routing means" means the technology or functionality that routes a customer's call to the appropriate agent.

[0016] "Speech synthesis means" refers to a technology or function that converts text data into voice data and provides it to customers in voice form.

[0017] The term "system" refers to a combination of a series of components and functions implemented in the present invention. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system of the present invention receives customer voice in real time, converts the voice into text data, and analyzes the text data to determine an appropriate response. Specific embodiments will be described in detail below from the perspectives of the server, terminal, and user.

[0040] Voice reception and recognition

[0041] User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0042] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0043] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0044] Content analysis

[0045] Server: The converted text data is passed to a natural language processing means for analysis. Based on the analysis results, this means classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0046] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "Usage Guide."

[0047] Determining the appropriate response

[0048] Server: Based on the analysis results, it determines the appropriate response. For example, it automatically generates an answer for simple FAQs, and transfers the call to the appropriate person for complex inquiries.

[0049] Example: When the question is classified into the category "How to use guide," a pre-prepared answer is searched for and the answer is given directly or in voice using a voice synthesis means.

[0050] Generate and provide answers

[0051] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[0052] Example: Generate text such as "To use product A, first turn it on, then..." and convert it into voice data using a voice synthesis means. The text then replies to the customer, "To use product A, first turn it on..."

[0053] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, etc.) is displayed on the representative's terminal and the call with the customer continues.

[0054] Example: The information "This is a question about how to use product A. There have been no previous inquiries" is displayed on the representative's device, and the call is connected to the representative.

[0055] Termination processing

[0056] Server: After the call ends, all inquiries and responses are recorded, allowing for prompt responses to subsequent inquiries.

[0057] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0058] This system allows for quick and accurate responses to customer inquiries and avoids the need for cumbersome IVR, which is expected to improve customer satisfaction and increase operator work efficiency.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0062] Step 2:

[0063] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[0064] Step 3:

[0065] Server: The speech recognition engine converts the received voice data into text data and returns the converted text data to the server.

[0066] Step 4:

[0067] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[0068] Step 5:

[0069] Server: Based on the analysis results of the NLP engine, classify the inquiry into a specific category. For example, determine that the inquiry is related to "How to use guides."

[0070] Step 6:

[0071] Server: Determines the appropriate response based on the category of the inquiry. In this case, since the usage guide is a simple FAQ, it decides to generate an automatic answer.

[0072] Step 7:

[0073] Server: Using a chat generation AI model, it generates an automatic response to the inquiry. For example, it generates a text response such as "To use product A, first turn it on, then..."

[0074] Step 8:

[0075] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice and returns the voice data to the server.

[0076] Step 9:

[0077] Server: Sends voice data to the customer and provides a response over the phone. The server responds to the customer by voice, saying, "To use product A, first turn it on..."

[0078] Step 10:

[0079] Server: After the call ends, the inquiry and response results are recorded in the database. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided via voice synthesis" is saved.

[0080] Step 11:

[0081] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] In today's world, there is a demand for fast and accurate responses to customer inquiries, but most call centers rely on manual response methods, which can be inefficient and result in low customer satisfaction. Additionally, cumbersome IVR (Interactive Voice Response) responses can be stressful for customers. Furthermore, transferring calls to the appropriate person based on the content of the inquiry and managing inquiry history can be cumbersome.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes a speech recognition unit that receives the customer's voice and converts the speech data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This allows the inquiry to be automatically analyzed and an appropriate response to be provided quickly. Furthermore, by adding a recording unit that records the inquiry and the response result after the call ends, subsequent inquiries can also be responded to quickly.

[0087] The "voice recognition means" is a device or program that converts voice data received from a customer into text data in real time.

[0088] The "natural language processing means" is a device or program that analyzes text data and classifies the inquiry content into specific categories based on the content.

[0089] The "processing means" is a device or program that determines an appropriate response based on the content of the inquiry.

[0090] The "answer generating means" is a device or program that automatically generates an answer according to the content of the inquiry.

[0091] A "call forwarding means" is a device or program that forwards a customer's call to the appropriate person.

[0092] The "voice synthesis means" is a device or program that converts the generated text data into voice data and provides it to the customer.

[0093] "Recording means" refers to a device or program that records and saves the content of the inquiry and the response results after the call ends.

[0094] The present invention is a system for responding to customer inquiries quickly and accurately, which receives customer voice in real time, converts the voice into text data, and then analyzes the text data to determine an appropriate response.

[0095] Voice reception and recognition

[0096] 1. User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0097] 2. Server: The server receives the customer's voice data via the telephone system and passes it to a speech recognition API (e.g., Google® Cloud Speech-to-Text), which converts the voice data into text data.

[0098] Example: A customer says, "Please tell me how to use product A," and a speech recognition API converts this into text: "Please tell me how to use product A."

[0099] Content analysis

[0100] 1. Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0101] Example: Confirm that the text data is "Please tell me how to use product A" and analyze the text using a natural language processing API.

[0102] 2. Server: Receives the analysis results of the natural language processing API and classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0103] Example: Based on the analysis results, "Inquiries about how to use product A" are classified into the "Usage Guide" category.

[0104] Determining the appropriate response

[0105] 1. Server: Determines the appropriate response based on the identified category, for example automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[0106] Example: Since the category is "How to use guide," the appropriate response is to determine whether to search the FAQ or use speech synthesis.

[0107] Generate and provide answers

[0108] 1. Server: When generating answers automatically, generate answer text using a chat generation AI model (e.g., ChatGPT (registered trademark) from OpenAI (registered trademark)). The generated text is passed to a speech synthesis API (e.g., Amazon Polly) and converted into voice data.

[0109] Example: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0110] 2. Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0111] Example: The generated voice data is played back to the customer via the telephone line. Also, when transferring the call to a representative, the following information is displayed: "This is a question about how to use product A. There have been no previous inquiries."

[0112] Termination processing

[0113] 1. Server: After the call ends, record all inquiries and the response results.

[0114] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0115] Examples of prompt statements

[0116] When you receive a customer query such as "How do I use product A?", generate appropriate FAQs and relevant answers.

[0117] Example of generated text: Prompt text to be passed to ChatGPT: "A customer asked me, 'How do I use product A?' Please briefly explain the specific steps to take."

[0118] As described above, the system of the present invention aims to efficiently process customer inquiries and improve customer satisfaction.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] User: A customer calls and verbally replies with their inquiry. For example, they say, "Please tell me how to use product A." This action generates voice data.

[0122] Step 2:

[0123] Server: The server receives the customer's voice data via the telephone system. It sends the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[0124] Input: Customer voice data.

[0125] Data processing: Converts voice data into text data in real time using a speech recognition API.

[0126] Output: The converted text data.

[0127] What it does: It receives a customer's speech saying "Please tell me how to use product A" and passes it to a speech recognition API, which converts the speech into text saying "Please tell me how to use product A."

[0128] Step 3:

[0129] Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0130] Input: Text data from the speech recognition API.

[0131] Data processing: Analyze text data using natural language processing APIs and classify the content into specific categories.

[0132] Output: Categorized analysis results.

[0133] Specific operation: The system confirms that the text data is "Please tell me how to use product A," analyzes the text using a natural language processing API, and classifies the inquiry into the "Usage Guide" category based on the analysis results.

[0134] Step 4:

[0135] Server: Determines the appropriate response based on the identified category, for example, automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[0136] Input: Parsed results from a natural language processing API.

[0137] Data processing: Applying rule-based algorithms or machine learning models to determine appropriate responses based on the analysis results.

[0138] Output: The appropriate action (auto-generated answer or call transfer instructions).

[0139] Specific operation: Based on the category "How to use guide", it determines whether to search FAQs or use speech synthesis as the appropriate response. If it is a simple FAQ, it searches for a pre-prepared answer and outputs it.

[0140] Step 5:

[0141] Server: When generating answers automatically, generate answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). Pass the generated text to a speech synthesis API (e.g., Amazon Polly) and convert it into voice data.

[0142] Input: FAQ search results or inquiry content from natural language processing API.

[0143] Data processing: Response text is generated using a chat generation AI model and converted into voice data using a speech synthesis API.

[0144] Output: The generated audio data.

[0145] Specific operation: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0146] Step 6:

[0147] Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0148] Input: Generated voice data or call transfer instructions.

[0149] Data processing: The generated voice data is played back over the telephone line, or the information is displayed on the person in charge's terminal.

[0150] Output: Response to customer or information provided to agent.

[0151] Specific operation: The generated voice data is played to the customer via the telephone line. Also, when transferring the call to a representative, the information "This is a question about how to use product A. There have been no previous inquiries" is displayed.

[0152] Step 7:

[0153] Server: After the call ends, record all inquiries and the response results.

[0154] Input: Inquiry details and response results after the call ends.

[0155] Data processing: Record the inquiry details and response results in a database.

[0156] Output: Recorded data.

[0157] Specific operation: Save the record "X month, X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" in the database.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] When customers inquire about products in a virtual store, they need a fast and accurate response. However, conventional systems do not process customer voice interactions in real time, which can lead to lower customer satisfaction. In addition, it is difficult to provide an immediate and appropriate response to complex inquiries, so an effective response method is needed.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes a speech recognition unit that receives a customer's voice and converts the voice data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This enables a system that includes a unit including smart glasses that performs real-time voice interaction with customers in a virtual store, a unit that performs the above processing on a cloud server, and a unit that generates answers using a generative AI model.

[0163] "Customer voice" is voice data uttered by customers in the virtual store.

[0164] "Speech recognition means" refers to a technical means for converting a customer's voice data into text data.

[0165] "Text data" is a character string that represents the customer's voice converted by a voice recognition means.

[0166] "Natural language processing means" is a processing technology for analyzing text data and classifying the content of inquiries.

[0167] The "processing means for determining an appropriate response" is a technical means for determining a response method based on the analyzed inquiry content.

[0168] The "means for generating an answer" refers to a technical means for generating an answer based on the content of the inquiry.

[0169] "Means for routing calls to the right person" means the technical means for routing calls to the right person for complex inquiries.

[0170] "Speech synthesis means" refers to a technical means for converting the generated response into voice data and providing it to the customer.

[0171] A "virtual store" is a virtual sales venue operated on the Internet.

[0172] "Voice interaction" refers to two-way communication between a customer and a company via voice.

[0173] "Smart glasses" are wearable devices with the ability to receive and play audio.

[0174] A "cloud server" is a server system for performing processing in a distributed computing environment.

[0175] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[0176] The system of the present invention is a voice interaction system for supporting customer support in a virtual store. This system is composed of a voice recognition unit, a natural language processing unit, a processing unit for determining an appropriate response, a response generation unit, a voice synthesis unit, smart glasses, a cloud server, and a generative AI model.

[0177] Voice reception and recognition

[0178] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0179] Smart glasses: Receives the customer's voice and sends the voice data to the cloud.

[0180] Cloud server: Using a speech recognition tool (e.g., Google Speech-to-Text API), converts the voice data into text data. For example, the voice "Please tell me the features of this product" is converted into the text "Please tell me the features of this product."

[0181] Content analysis

[0182] Cloud server: The converted text data is passed to a natural language processing means (e.g., OpenAI GPT-3 (registered trademark)) and its contents are analyzed. Through the analysis, the inquiry content is classified into categories such as "product information" and "usage guide." For example, the text "Please tell me the features of this product" is classified as "product information."

[0183] Determine appropriate responses and generate answers

[0184] Cloud server: Determines the appropriate response based on the analysis results and generates an answer corresponding to the inquiry using a generative AI model (e.g., GPT-3). The generated answer text might be, for example, "The features of this product are as follows..."

[0185] Providing answers

[0186] Cloud server: The generated answer is converted into voice data using a voice synthesis means (e.g., Amazon Polly). The converted voice data is sent to the smart glasses and provided to the customer.

[0187] Smart glasses: Plays back audio data and provides answers to customers, such as "The features of this product are as follows..."

[0188] Examples and Prompts

[0189] As a concrete example, if a customer asks, "What are the features of this product?", the above steps will return a spoken response saying, "The features of this product are as follows..." An example of a prompt sentence when generating an answer using a generative AI model is as follows:

[0190] "You have a customer who wants to know the features of your product. Please answer the following question: What are the features of this product?"

[0191] This system allows customers in virtual stores to obtain quick and accurate information through voice interaction, improving the user experience.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0195] Input: Customer voice data

[0196] Output: Input of audio data to smart glasses

[0197] Specific operation: The customer's voice is collected through the microphone in the smart glasses.

[0198] Step 2:

[0199] Smart glasses: Collected customer voice data is sent to the cloud.

[0200] Input: Customer voice data

[0201] Output: Sending audio data to the cloud

[0202] Specific operation: The smart glasses upload the audio data to the cloud server.

[0203] Step 3:

[0204] Server: Converts the voice data into text data using a voice recognition means (e.g., Google Speech-to-Text API).

[0205] Input: Audio data

[0206] Output: Text data

[0207] Specific operation: Calls the Google Speech-to-Text API and converts the voice data into text. For example, "What are the features of this product?" becomes the text "What are the features of this product?"

[0208] Step 4:

[0209] Server: The converted text data is passed to a natural language processing means (e.g., OpenAI GPT-3) and its content is analyzed.

[0210] Input: Text data

[0211] Output: Category as analysis result (e.g. "Product Information", "Usage Guide", etc.)

[0212] How it works: The GPT-3 model inputs text data, performs content analysis, and classifies the query into the appropriate category. For example, the text "What are the features of this product?" is classified as "Product Information."

[0213] Step 5:

[0214] Server: Based on the analysis results, it generates answers using a generative AI model.

[0215] Input: Parsed categories and original text data

[0216] Output: Generated answer text

[0217] How it works: Using the GPT-3 model, it generates answers based on the analyzed categories. For example, based on a prompt such as "A customer wants to know the features of a product. Please answer the following question: What are the features of this product?", it generates an answer such as "The features of this product are as follows..."

[0218] Step 6:

[0219] Server: The generated answer text is converted into voice data using a speech synthesis means (e.g., Amazon Polly) and sent to the smart glasses.

[0220] Input: Generated answer text

[0221] Output: Audio data

[0222] Specific operation: Using Amazon Polly, the answer text is converted into speech data and sent to the smart glasses. For example, the text "The features of this product are as follows..." is converted into speech data.

[0223] Step 7:

[0224] Smart glasses: Plays back audio data and provides answers to customers.

[0225] Input: Audio data

[0226] Output: Audio output to the customer

[0227] Specific operation: Play a voice through the speaker of the smart glasses and reply to the customer, for example, "The features of this product are as follows..."

[0228] Through these steps, customers can access real-time product information in the virtual store, improving the user experience.

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, enabling it to respond according to the customer's emotional state. A specific embodiment will be described in detail below from the perspectives of the server, terminal, and user.

[0231] Voice reception and recognition

[0232] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0233] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0234] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0235] emotion recognition

[0236] Server: Passes the customer's voice data to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state (e.g., anger, anxiety, satisfaction).

[0237] Example: If a customer excitedly says, "I don't know how to use product A!", the emotion engine will recognize that the customer is in an emotional state of "anxious" or "angry."

[0238] Content analysis

[0239] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, this engine classifies the inquiry into a specific category (e.g., "How to use guide" or "Troubleshooting"). The analysis results of the emotion engine are also taken into account.

[0240] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "How to use guide" while taking into account the customer's emotion of "anxiety."

[0241] Determining the appropriate response

[0242] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ, or routing the call to the right agent for a complex, emotional inquiry.

[0243] Example: If the call falls into the category "How to Use Guide" and the customer's emotion is "Anxiety," it is determined that a more flexible and gentle response is required, and the call is transferred to a representative.

[0244] Generate and provide answers

[0245] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[0246] Example: A text response such as "To use product A, first turn it on, then..." is generated and converted into voice data using a voice synthesis means. The response to the customer is "To use product A, first turn it on...".

[0247] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, emotional state, etc.) is displayed on the representative's terminal and the call with the customer continues.

[0248] Example: The information "This is a question about how to use product A. The customer is currently in a state of anxiety" is displayed on the agent's device, and the call is connected to the agent.

[0249] Termination processing

[0250] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[0251] Example: Save a record that reads, "X month, X day, 2023, inquiry content: How to use product A. Response result: Response by a staff member. Customer emotion: Anxiety."

[0252] This system allows for quick and accurate responses to customer inquiries, and by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and increase the work efficiency of operators.

[0253] The processing flow will be explained below.

[0254] Step 1:

[0255] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0256] Step 2:

[0257] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[0258] Step 3:

[0259] Server: The speech recognition engine converts the received voice data into text data in real time and returns the converted text data to the server.

[0260] Step 4:

[0261] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[0262] Step 5:

[0263] Server: Passes the voice data from the customer to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if the customer is angry, their emotional state is recognized as "anger."

[0264] Step 6:

[0265] Server: Integrates the text analysis results from the NLP engine and the emotion recognition results from the emotion engine, and classifies the inquiry content into specific categories (e.g., "How to use guide" or "Troubleshooting").

[0266] Step 7:

[0267] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ, or routing the call to the right agent for a complex, emotional inquiry.

[0268] Step 8:

[0269] Server: When generating answers automatically, the server uses a chat generation AI model to generate answer text. For example, it generates text like, "To use product A, first turn it on, then..."

[0270] Step 9:

[0271] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice data and returns the voice data to the server.

[0272] Step 10:

[0273] Server: Provides voice data to the customer and responds over the phone. For example, it responds to the customer by voice, saying, "To use product A, first turn it on..."

[0274] Step 11:

[0275] Server: When transferring a call to a representative, appropriate information (such as the inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device so that the call with the customer can continue.

[0276] Step 12:

[0277] Terminal: The agent interacts with the customer based on the information displayed on the terminal. For example, the terminal may display, "This is a question about how to use product A. The customer is currently in a state of anxiety."

[0278] Step 13:

[0279] Server: After the call ends, record the inquiry details and the response results in the database. For example, record "Xth month, 2023, inquiry details: How to use product A, response results: Response by the person in charge, customer emotion: Anxiety."

[0280] Step 14:

[0281] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Conventional customer support systems simply convert customer voice into text, but are unable to take into account the customer's emotional state. This makes it difficult to provide appropriate responses, particularly to emotional inquiries, resulting in lower customer satisfaction. There is a need for a system that can solve this problem and provide quick and appropriate answers.

[0285] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means that receives the customer's voice and converts the voice data into text data, an emotion recognition means that recognizes the emotional state based on the tone, speed, and pitch of the voice, and a natural language processing means that analyzes the text data and classifies the inquiry content. This enables a quick and appropriate response taking into account the emotional state of the customer.

[0286] The "voice recognition means" is a means for receiving voice data from a customer and converting it into text data.

[0287] An "emotion recognition means" is a means for analyzing the tone, speed, and pitch of a voice to recognize the emotional state of a customer.

[0288] "Natural language processing means" refers to means for analyzing text data and classifying its contents.

[0289] The "processing means" is a means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer.

[0290] The "means for generating an answer" is a means for automatically generating an answer based on the content of the inquiry.

[0291] A "means for transferring a call" is a means for transferring a call to an appropriate person.

[0292] The "voice synthesis means" is a means for converting the generated response into voice data and providing it to the customer.

[0293] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, making it possible to respond according to the customer's emotional state. Specific embodiments are described in detail below.

[0294] Voice reception and recognition

[0295] When a user calls and speaks their inquiry, the server receives the customer's voice data. For example, if a customer says, "Please tell me how to use product A," services such as Twilio and VoIP are used to receive this voice data in real time. The received voice data is passed to the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[0296] emotion recognition

[0297] The server passes the voice data to an emotion recognition means (e.g., IBM Watson (registered trademark) Tone Analyzer) and recognizes the customer's emotional state by analyzing the tone, speed, pitch, etc. of the voice. For example, if a customer says excitedly, "I don't know how to use product A!", the emotion engine will recognize the emotional state as "anxiety" or "anger."

[0298] Content analysis

[0299] The server passes the converted text data to a natural language processing means (e.g., a custom engine using the BERT model) and analyzes its contents. Based on the analysis results, this means classifies the inquiry content into a specific category (e.g., "usage guide" or "troubleshooting"). Furthermore, the analysis results of the emotion engine are also taken into consideration. For example, the text "Please tell me how to use product A" is analyzed and classified as a "usage guide," while also taking into account the customer's emotion of "anxiety."

[0300] Determining the appropriate response

[0301] The server determines the appropriate response based on the inquiry category and the customer's emotional state. For example, it automatically generates an answer for simple FAQs, but transfers the call to the appropriate agent for complex, emotional inquiries. For example, if the inquiry is classified as a "How to Use Guide" and the customer's emotion is "Anxiety," the call will be transferred to an agent.

[0302] Generate and provide answers

[0303] When the server automatically generates an answer, it uses a generative AI model (e.g., OpenAI's GPT-3) to generate the answer text. It then converts it into audio data using a speech synthesis API (e.g., Google Cloud Text-to-Speech) and provides it to the customer. For example, it generates text such as "To use product A, first turn it on...", converts it into audio data, and responds to the customer.

[0304] On the other hand, when transferring a call to a representative, appropriate information (e.g., inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device, and the call with the customer continues. For example, information such as "This is an inquiry about how to use product A. The customer is currently in a state of anxiety" is displayed on the representative's device, and the call is connected to the representative.

[0305] Termination processing

[0306] After the call ends, the server records the inquiry and the response result in a database. This allows subsequent inquiries to be handled promptly. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Response by the person in charge, customer's emotion: Anxiety" is saved.

[0307] Prompt Sentence Examples

[0308] By using the following prompt statements, you can execute each processing step of the system in detail.

[0309] Convert the customer's speech into text. For example, if they say "How do I use product A?", return that text.

[0310] Analyze customer sentiment from this audio data to determine whether they are in a state of anxiety.

[0311] Pass this text data to a natural language processing engine and classify the inquiry content into the "How to Use Guide" category.

[0312] This system is an effective means of responding to customer inquiries quickly and accurately, and by taking into account the emotional state of the customer, it can provide an individualized response, which is expected to result in improved customer satisfaction and improved operator efficiency.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1:

[0315] A user makes a call and speaks their inquiry. For example, the user might say, "Please tell me how to use product A." This voice data is sent to the server.

[0316] Input: User's voice data

[0317] Output: Raw audio data sent to the server

[0318] Step 2:

[0319] The server stores the voice data received via Twilio, etc. This stored voice data is then sent to the Google Cloud Speech-to-Text API to be converted into text data.

[0320] Input: Audio data received by the server

[0321] Output: Text data generated by the Google Cloud Speech-to-Text API (e.g., "How do I use Product A?")

[0322] Step 3:

[0323] The server sends the converted text data and raw voice data to an emotion recognition tool (an API such as IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state.

[0324] Input: Text data and raw audio data

[0325] Output: Emotional state (e.g., "anxious" or "angry")

[0326] Step 4:

[0327] The server sends the text data to a natural language processing engine (such as a custom engine using the BERT model) to analyze the query, which then categorizes the query into categories such as "How to use guide" or "Troubleshooting."

[0328] Input: Text data

[0329] Output: Category label (e.g. "How to use guide")

[0330] Step 5:

[0331] The server determines the appropriate response based on the category of the inquiry and the customer's emotional state: simple inquiries are answered by an automated system, while complex or emotional inquiries are transferred to a human agent.

[0332] Input: category label, emotional state

[0333] Output: Decision on appropriate response (e.g., generating an auto-attendant or deciding on call forwarding)

[0334] Step 6:

[0335] If an automated response is selected, the server generates a text response using a generative AI model (such as OpenAI's GPT-3), then sends this text to a speech synthesis API (such as Google Cloud Text-to-Speech) to convert it into audio data.

[0336] Input: Inquiry content based on category labels, prompt sentences to input to the generative AI model

[0337] Output: Generated answer text and audio data (e.g., "To use product A, first turn it on...")

[0338] Step 7:

[0339] The server provides the generated voice data to the user. On the other hand, when transferring a call, the server displays appropriate information (such as the inquiry content, past inquiry history, and emotional state) on the agent's terminal and connects the call to the agent.

[0340] Input: Generated voice data or information to be conveyed to the agent

[0341] Output: A spoken response to the user or information displayed on the agent's terminal (e.g., "This is a question about how to use product A. The customer is currently in a state of anxiety.")

[0342] Step 8:

[0343] After the call ends, the server records the inquiry and the response results in a database, allowing future inquiries to be handled promptly and appropriately.

[0344] Input: Call termination information, inquiry details, response results, customer emotional state

[0345] Output: Inquiry and response history stored in the database (e.g., "X month, X day, 2023, inquiry content: How to use product A, response result: Response by a staff member, customer emotion: Anxiety")

[0346] By implementing the above processing steps, the system of the present invention can respond to customer inquiries quickly and accurately. Furthermore, by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and operator efficiency.

[0347] (Application example 2)

[0348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Conventional speech recognition systems have difficulty accurately recognizing and responding to a customer's emotional state, and lack a means to intuitively and visually guide the appropriate response. As a result, it has been difficult to improve customer satisfaction, and the burden on operators has increased. This invention aims to solve these problems by simultaneously converting customer speech into text and recognizing emotions, and visually guiding the appropriate response.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0351] In this invention, the server includes a speech recognition means for receiving a customer's voice and converting the speech data into text data, a natural language processing means for analyzing the text data and classifying the inquiry content, an emotion recognition means for analyzing the text data and recognizing the emotional state of the customer, a processing means for determining an appropriate response based on the inquiry content and the emotional state of the customer, and a smart device display means for visually guiding the appropriate response. This enables a quick and appropriate response that takes the customer's emotions into consideration, thereby improving customer satisfaction.

[0352] The "voice recognition means" is a device that receives the customer's voice and converts the voice data into text data.

[0353] The "natural language processing means" is a device that analyzes the text data and classifies the inquiry content.

[0354] The "emotion recognition means" is a device that analyzes the text data and recognizes the emotional state of the customer.

[0355] The "processing means" is a device that determines an appropriate response based on the content of the inquiry and the emotional state of the customer.

[0356] The "voice synthesis means" is a device that converts the generated response into voice data and provides it to the customer.

[0357] The "smart device display means" is a device with a display function for visually guiding the appropriate response.

[0358] MODE FOR CARRYING OUT THE INVENTION

[0359] Voice reception and recognition

[0360] User: A customer speaks their inquiry at the store. For example, they say, "Do you have this product in stock?"

[0361] Server: Receives the customer's voice data through the microphone attached to the smart glasses and passes it to a voice recognition means (for example, Google voice recognition API). This voice recognition means converts the voice data into text data.

[0362] emotion recognition

[0363] Server: Passes the text data to an emotion recognition means (for example, the Distilroberta emotion analysis model), analyzes the tone, speed, and pitch of the voice, and recognizes the customer's emotional state (for example, joy, anger, anxiety).

[0364] Content analysis

[0365] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, the inquiry is classified into a specific category (e.g., "inventory check" or "product specifications"). Emotion recognition results are also taken into account.

[0366] Determining the appropriate response

[0367] Server: Determines the appropriate response based on the category of inquiry and the customer's emotional state. For example, if a customer is anxious about checking inventory, the server will guide the salesperson to respond quickly and courteously.

[0368] Providing a visual guide

[0369] Device: The smart glasses display will display appropriate guidance, such as a message like "Checking inventory. Please wait a moment."

[0370] Providing answers

[0371] Server: If an automatic reply is possible, a generative AI model is used to generate a reply text, which is then converted into voice data using a voice synthesis means and provided.

[0372] Termination processing

[0373] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[0374] Hardware and software used

[0375] Hardware used: Smart glasses, server

[0376] Software used: Google speech recognition API, NLP engine, distilroberta sentiment analysis model, generative AI model

[0377] Adding specific examples

[0378] Examples:

[0379] A store associate wears smart glasses and a customer asks, "Do you have this item in stock?"

[0380] Customer speech is transcribed in real time: "Do you have this item in stock?"

[0381] The sentiment analysis engine detects "interests."

[0382] The smart glasses display a guide that reads, "Please check stock and let us know immediately."

[0383] Example prompt for a generative AI model:

[0384] If a customer excitedly asks, "Do you have this item in stock?", what advice should the sales associate give them?

[0385] In this way, the present invention provides a system that can analyze customer voices in real time, recognize their emotional state, and provide appropriate responses. This system is expected to improve customer satisfaction and streamline store operations.

[0386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0387] Step 1:

[0388] The user makes a voice inquiry through a microphone in the smart glasses worn by the store clerk, and the input is captured as voice data.

[0389] Step 2:

[0390] The server receives voice data from the smart glasses. The input is voice data, which is passed to a voice recognition means (Google voice recognition API) and converted into text data. The output is text data.

[0391] Step 3:

[0392] The server passes the converted text data to an emotion recognition means (distilroberta emotion analysis model). The input is text data, and the data is processed to analyze the emotional state. The output is the customer's emotional state data (e.g., joy, anger, anxiety).

[0393] Step 4:

[0394] The server passes the text data to a natural language processing engine (NLP engine), which analyzes the content and classifies the inquiry into specific categories. The input is text data, and content analysis and category classification are performed as data operations. The output is category data (for example, "inventory check" or "product specifications").

[0395] Step 5:

[0396] The server uses a processing means to determine an appropriate response based on the category data of the inquiry content and the emotional state data of the customer. The input is the category data and the emotional state data, and the response is determined as a data calculation. The output is an appropriate response guideline.

[0397] Step 6:

[0398] The terminal (smart glasses) visually displays the determined correspondence guidelines. The input is the correspondence guidelines, which are converted into a display format as data processing. The output is a guide message displayed on the smart glasses display.

[0399] Step 7:

[0400] The server automatically generates answer text using a generative AI model as needed, and converts it into voice data using a voice synthesis means. The input is answer text data, which is converted into voice data as data processing. The output is voice data.

[0401] Step 8:

[0402] The user responds appropriately to the customer by following the guide message on the display of the smart glasses. The input is the guide message displayed on the display of the smart glasses, and the output is the actual customer response action.

[0403] Step 9:

[0404] After the response is completed, the server records the inquiry details and the response results in a database. The input is the response result data, which is converted into a recording format as data processing. The output is the record data saved in the database.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0406] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0408] [Second embodiment]

[0409] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0410] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0411] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0413] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0415] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0416] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0417] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0420] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0421] The system of the present invention receives customer voice in real time, converts the voice into text data, and analyzes the text data to determine an appropriate response. Specific embodiments will be described in detail below from the perspectives of the server, terminal, and user.

[0422] Voice reception and recognition

[0423] User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0424] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0425] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0426] Content analysis

[0427] Server: The converted text data is passed to a natural language processing means for analysis. Based on the analysis results, this means classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0428] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "Usage Guide."

[0429] Determining the appropriate response

[0430] Server: Based on the analysis results, it determines the appropriate response. For example, it automatically generates an answer for simple FAQs, and transfers the call to the appropriate person for complex inquiries.

[0431] Example: When the question is classified into the category "How to use guide," a pre-prepared answer is searched for and the answer is given directly or in voice using a voice synthesis means.

[0432] Generate and provide answers

[0433] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[0434] Example: Generate text such as "To use product A, first turn it on, then..." and convert it into voice data using a voice synthesis means. The text then replies to the customer, "To use product A, first turn it on..."

[0435] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, etc.) is displayed on the representative's terminal and the call with the customer continues.

[0436] Example: The information "This is a question about how to use product A. There have been no previous inquiries" is displayed on the representative's device, and the call is connected to the representative.

[0437] Termination processing

[0438] Server: After the call ends, all inquiries and responses are recorded, allowing for prompt responses to subsequent inquiries.

[0439] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0440] This system allows for quick and accurate responses to customer inquiries and avoids the need for cumbersome IVR, which is expected to improve customer satisfaction and increase operator work efficiency.

[0441] The processing flow will be explained below.

[0442] Step 1:

[0443] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0444] Step 2:

[0445] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[0446] Step 3:

[0447] Server: The speech recognition engine converts the received voice data into text data and returns the converted text data to the server.

[0448] Step 4:

[0449] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[0450] Step 5:

[0451] Server: Based on the analysis results of the NLP engine, classify the inquiry into a specific category. For example, determine that the inquiry is related to "How to use guides."

[0452] Step 6:

[0453] Server: Determines the appropriate response based on the category of the inquiry. In this case, since the usage guide is a simple FAQ, it decides to generate an automatic answer.

[0454] Step 7:

[0455] Server: Using a chat generation AI model, it generates an automatic response to the inquiry. For example, it generates a text response such as "To use product A, first turn it on, then..."

[0456] Step 8:

[0457] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice and returns the voice data to the server.

[0458] Step 9:

[0459] Server: Sends voice data to the customer and provides a response over the phone. The server responds to the customer by voice, saying, "To use product A, first turn it on..."

[0460] Step 10:

[0461] Server: After the call ends, the inquiry and response results are recorded in the database. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided via voice synthesis" is saved.

[0462] Step 11:

[0463] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[0464] Example 1

[0465] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] In today's world, there is a demand for fast and accurate responses to customer inquiries, but most call centers rely on manual response methods, which can be inefficient and result in low customer satisfaction. Additionally, cumbersome IVR (Interactive Voice Response) responses can be stressful for customers. Furthermore, transferring calls to the appropriate person based on the content of the inquiry and managing inquiry history can be cumbersome.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes a speech recognition unit that receives the customer's voice and converts the speech data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This allows the inquiry to be automatically analyzed and an appropriate response to be provided quickly. Furthermore, by adding a recording unit that records the inquiry and the response result after the call ends, subsequent inquiries can also be responded to quickly.

[0469] The "voice recognition means" is a device or program that converts voice data received from a customer into text data in real time.

[0470] The "natural language processing means" is a device or program that analyzes text data and classifies the inquiry content into specific categories based on the content.

[0471] The "processing means" is a device or program that determines an appropriate response based on the content of the inquiry.

[0472] The "answer generating means" is a device or program that automatically generates an answer according to the content of the inquiry.

[0473] A "call forwarding means" is a device or program that forwards a customer's call to the appropriate person.

[0474] The "voice synthesis means" is a device or program that converts the generated text data into voice data and provides it to the customer.

[0475] "Recording means" refers to a device or program that records and saves the content of the inquiry and the response results after the call ends.

[0476] The present invention is a system for responding to customer inquiries quickly and accurately, which receives customer voice in real time, converts the voice into text data, and then analyzes the text data to determine an appropriate response.

[0477] Voice reception and recognition

[0478] 1. User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0479] 2. Server: The server receives the customer's voice data via the telephone system and passes it to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[0480] Example: A customer says, "Please tell me how to use product A," and a speech recognition API converts this into text: "Please tell me how to use product A."

[0481] Content analysis

[0482] 1. Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0483] Example: Confirm that the text data is "Please tell me how to use product A" and analyze the text using a natural language processing API.

[0484] 2. Server: Receives the analysis results of the natural language processing API and classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0485] Example: Based on the analysis results, "Inquiries about how to use product A" are classified into the "Usage Guide" category.

[0486] Determining the appropriate response

[0487] 1. Server: Determines the appropriate response based on the identified category, for example automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[0488] Example: Since the category is "How to use guide," the appropriate response is to determine whether to search the FAQ or use speech synthesis.

[0489] Generate and provide answers

[0490] 1. Server: To generate an answer automatically, generate the answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). The generated text is passed to a speech synthesis API (e.g., Amazon Polly) and converted into voice data.

[0491] Example: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0492] 2. Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0493] Example: The generated voice data is played back to the customer via the telephone line. Also, when transferring the call to a representative, the following information is displayed: "This is a question about how to use product A. There have been no previous inquiries."

[0494] Termination processing

[0495] 1. Server: After the call ends, record all inquiries and the response results.

[0496] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0497] Examples of prompt statements

[0498] When you receive a customer query such as "How do I use product A?", generate appropriate FAQs and relevant answers.

[0499] Example of generated text: Prompt text to be passed to ChatGPT: "A customer asked me, 'How do I use product A?' Please briefly explain the specific steps to take."

[0500] As described above, the system of the present invention aims to efficiently process customer inquiries and improve customer satisfaction.

[0501] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0502] Step 1:

[0503] User: A customer calls and verbally replies with their inquiry. For example, they say, "Please tell me how to use product A." This action generates voice data.

[0504] Step 2:

[0505] Server: The server receives the customer's voice data via the telephone system. It sends the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[0506] Input: Customer voice data.

[0507] Data processing: Converts voice data into text data in real time using a speech recognition API.

[0508] Output: The converted text data.

[0509] What it does: It receives a customer's speech saying "Please tell me how to use product A" and passes it to a speech recognition API, which converts the speech into text saying "Please tell me how to use product A."

[0510] Step 3:

[0511] Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0512] Input: Text data from the speech recognition API.

[0513] Data processing: Analyze text data using natural language processing APIs and classify the content into specific categories.

[0514] Output: Categorized analysis results.

[0515] Specific operation: The system confirms that the text data is "Please tell me how to use product A," analyzes the text using a natural language processing API, and classifies the inquiry into the "Usage Guide" category based on the analysis results.

[0516] Step 4:

[0517] Server: Determines the appropriate response based on the identified category, for example, automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[0518] Input: Parsed results from a natural language processing API.

[0519] Data processing: Applying rule-based algorithms or machine learning models to determine appropriate responses based on the analysis results.

[0520] Output: The appropriate action (auto-generated answer or call transfer instructions).

[0521] Specific operation: Based on the category "How to use guide", it determines whether to search FAQs or use speech synthesis as the appropriate response. If it is a simple FAQ, it searches for a pre-prepared answer and outputs it.

[0522] Step 5:

[0523] Server: When generating answers automatically, generate answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). Pass the generated text to a speech synthesis API (e.g., Amazon Polly) and convert it into voice data.

[0524] Input: FAQ search results or inquiry content from natural language processing API.

[0525] Data processing: Response text is generated using a chat generation AI model and converted into voice data using a speech synthesis API.

[0526] Output: The generated audio data.

[0527] Specific operation: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0528] Step 6:

[0529] Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0530] Input: Generated voice data or call transfer instructions.

[0531] Data processing: The generated voice data is played back over the telephone line, or the information is displayed on the person in charge's terminal.

[0532] Output: Response to customer or information provided to agent.

[0533] Specific operation: The generated voice data is played to the customer via the telephone line. Also, when transferring the call to a representative, the information "This is a question about how to use product A. There have been no previous inquiries" is displayed.

[0534] Step 7:

[0535] Server: After the call ends, record all inquiries and the response results.

[0536] Input: Inquiry details and response results after the call ends.

[0537] Data processing: Record the inquiry details and response results in a database.

[0538] Output: Recorded data.

[0539] Specific operation: Save the record "X month, X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" in the database.

[0540] (Application example 1)

[0541] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0542] When customers inquire about products in a virtual store, they need a fast and accurate response. However, conventional systems do not process customer voice interactions in real time, which can lead to lower customer satisfaction. In addition, it is difficult to provide an immediate and appropriate response to complex inquiries, so an effective response method is needed.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0544] In this invention, the server includes a speech recognition unit that receives a customer's voice and converts the voice data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This enables a system that includes a unit including smart glasses that performs real-time voice interaction with customers in a virtual store, a unit that performs the above processing on a cloud server, and a unit that generates answers using a generative AI model.

[0545] "Customer voice" is voice data uttered by customers in the virtual store.

[0546] "Speech recognition means" refers to a technical means for converting a customer's voice data into text data.

[0547] "Text data" is a character string that represents the customer's voice converted by a voice recognition means.

[0548] "Natural language processing means" is a processing technology for analyzing text data and classifying the content of inquiries.

[0549] The "processing means for determining an appropriate response" is a technical means for determining a response method based on the analyzed inquiry content.

[0550] The "means for generating an answer" refers to a technical means for generating an answer based on the content of the inquiry.

[0551] "Means for routing calls to the right person" means the technical means for routing calls to the right person for complex inquiries.

[0552] "Speech synthesis means" refers to a technical means for converting the generated response into voice data and providing it to the customer.

[0553] A "virtual store" is a virtual sales venue operated on the Internet.

[0554] "Voice interaction" refers to two-way communication between a customer and a company via voice.

[0555] "Smart glasses" are wearable devices with the ability to receive and play audio.

[0556] A "cloud server" is a server system for performing processing in a distributed computing environment.

[0557] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[0558] The system of the present invention is a voice interaction system for supporting customer support in a virtual store. This system is composed of a voice recognition unit, a natural language processing unit, a processing unit for determining an appropriate response, a response generation unit, a voice synthesis unit, smart glasses, a cloud server, and a generative AI model.

[0559] Voice reception and recognition

[0560] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0561] Smart glasses: Receives the customer's voice and sends the voice data to the cloud.

[0562] Cloud server: Using a speech recognition tool (e.g., Google Speech-to-Text API), converts the voice data into text data. For example, the voice "Please tell me the features of this product" is converted into the text "Please tell me the features of this product."

[0563] Content analysis

[0564] Cloud server: The converted text data is passed to a natural language processing tool (e.g., OpenAI GPT-3) and its contents are analyzed. Through the analysis, the inquiry content is classified into categories such as "product information" and "usage guide." For example, the text "Please tell me the features of this product" is classified as "product information."

[0565] Determine appropriate responses and generate answers

[0566] Cloud server: Determines the appropriate response based on the analysis results and generates an answer corresponding to the inquiry using a generative AI model (e.g., GPT-3). The generated answer text might be, for example, "The features of this product are as follows..."

[0567] Providing answers

[0568] Cloud server: The generated answer is converted into voice data using a voice synthesis means (e.g., Amazon Polly). The converted voice data is sent to the smart glasses and provided to the customer.

[0569] Smart glasses: Plays back audio data and provides answers to customers, such as "The features of this product are as follows..."

[0570] Examples and Prompts

[0571] As a concrete example, if a customer asks, "What are the features of this product?", the above steps will return a spoken response saying, "The features of this product are as follows..." An example of a prompt sentence when generating an answer using a generative AI model is as follows:

[0572] "You have a customer who wants to know the features of your product. Please answer the following question: What are the features of this product?"

[0573] This system allows customers in virtual stores to obtain quick and accurate information through voice interaction, improving the user experience.

[0574] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0575] Step 1:

[0576] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0577] Input: Customer voice data

[0578] Output: Input of audio data to smart glasses

[0579] Specific operation: The customer's voice is collected through the microphone in the smart glasses.

[0580] Step 2:

[0581] Smart glasses: Collected customer voice data is sent to the cloud.

[0582] Input: Customer voice data

[0583] Output: Sending audio data to the cloud

[0584] Specific operation: The smart glasses upload the audio data to the cloud server.

[0585] Step 3:

[0586] Server: Converts the voice data into text data using a voice recognition means (e.g., Google Speech-to-Text API).

[0587] Input: Audio data

[0588] Output: Text data

[0589] Specific operation: Calls the Google Speech-to-Text API and converts the voice data into text. For example, "What are the features of this product?" becomes the text "What are the features of this product?"

[0590] Step 4:

[0591] Server: The converted text data is passed to a natural language processing means (e.g., OpenAI GPT-3) and its content is analyzed.

[0592] Input: Text data

[0593] Output: Category as analysis result (e.g. "Product Information", "Usage Guide", etc.)

[0594] How it works: The GPT-3 model inputs text data, performs content analysis, and classifies the query into the appropriate category. For example, the text "What are the features of this product?" is classified as "Product Information."

[0595] Step 5:

[0596] Server: Based on the analysis results, it generates answers using a generative AI model.

[0597] Input: Parsed categories and original text data

[0598] Output: Generated answer text

[0599] How it works: Using the GPT-3 model, it generates answers based on the analyzed categories. For example, based on a prompt such as "A customer wants to know the features of a product. Please answer the following question: What are the features of this product?", it generates an answer such as "The features of this product are as follows..."

[0600] Step 6:

[0601] Server: The generated answer text is converted into voice data using a speech synthesis means (e.g., Amazon Polly) and sent to the smart glasses.

[0602] Input: Generated answer text

[0603] Output: Audio data

[0604] Specific operation: Using Amazon Polly, the answer text is converted into speech data and sent to the smart glasses. For example, the text "The features of this product are as follows..." is converted into speech data.

[0605] Step 7:

[0606] Smart glasses: Plays back audio data and provides answers to customers.

[0607] Input: Audio data

[0608] Output: Audio output to the customer

[0609] Specific operation: Play a voice through the speaker of the smart glasses and reply to the customer, for example, "The features of this product are as follows..."

[0610] Through these steps, customers can access real-time product information in the virtual store, improving the user experience.

[0611] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0612] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, enabling it to respond according to the customer's emotional state. A specific embodiment will be described in detail below from the perspectives of the server, terminal, and user.

[0613] Voice reception and recognition

[0614] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0615] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0616] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0617] emotion recognition

[0618] Server: Passes the customer's voice data to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state (e.g., anger, anxiety, satisfaction).

[0619] Example: If a customer excitedly says, "I don't know how to use product A!", the emotion engine will recognize that the customer is in an emotional state of "anxious" or "angry."

[0620] Content analysis

[0621] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, this engine classifies the inquiry into a specific category (e.g., "How to use guide" or "Troubleshooting"). The analysis results of the emotion engine are also taken into account.

[0622] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "How to use guide" while taking into account the customer's emotion of "anxiety."

[0623] Determining the appropriate response

[0624] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ, or routing the call to the right agent for a complex, emotional inquiry.

[0625] Example: If the call falls into the category "How to Use Guide" and the customer's emotion is "Anxiety," it is determined that a more flexible and gentle response is required, and the call is transferred to a representative.

[0626] Generate and provide answers

[0627] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[0628] Example: A text response such as "To use product A, first turn it on, then..." is generated and converted into voice data using a voice synthesis means. The response to the customer is "To use product A, first turn it on...".

[0629] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, emotional state, etc.) is displayed on the representative's terminal and the call with the customer continues.

[0630] Example: The information "This is a question about how to use product A. The customer is currently in a state of anxiety" is displayed on the agent's device, and the call is connected to the agent.

[0631] Termination processing

[0632] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[0633] Example: Save a record that reads, "X month, X day, 2023, inquiry content: How to use product A. Response result: Response by a staff member. Customer emotion: Anxiety."

[0634] This system allows for quick and accurate responses to customer inquiries, and by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and increase the work efficiency of operators.

[0635] The processing flow will be explained below.

[0636] Step 1:

[0637] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0638] Step 2:

[0639] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[0640] Step 3:

[0641] Server: The speech recognition engine converts the received voice data into text data in real time and returns the converted text data to the server.

[0642] Step 4:

[0643] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[0644] Step 5:

[0645] Server: Passes the voice data from the customer to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if the customer is angry, their emotional state is recognized as "anger."

[0646] Step 6:

[0647] Server: Integrates the text analysis results from the NLP engine and the emotion recognition results from the emotion engine, and classifies the inquiry content into specific categories (e.g., "How to use guide" or "Troubleshooting").

[0648] Step 7:

[0649] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ, or routing the call to the right agent for a complex, emotional inquiry.

[0650] Step 8:

[0651] Server: When generating answers automatically, the server uses a chat generation AI model to generate answer text. For example, it generates text like, "To use product A, first turn it on, then..."

[0652] Step 9:

[0653] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice data and returns the voice data to the server.

[0654] Step 10:

[0655] Server: Provides voice data to the customer and responds over the phone. For example, it responds to the customer by voice, saying, "To use product A, first turn it on..."

[0656] Step 11:

[0657] Server: When transferring a call to a representative, appropriate information (such as the inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device so that the call with the customer can continue.

[0658] Step 12:

[0659] Terminal: The agent interacts with the customer based on the information displayed on the terminal. For example, the terminal may display, "This is a question about how to use product A. The customer is currently in a state of anxiety."

[0660] Step 13:

[0661] Server: After the call ends, record the inquiry details and the response results in the database. For example, record "Xth month, 2023, inquiry details: How to use product A, response results: Response by the person in charge, customer emotion: Anxiety."

[0662] Step 14:

[0663] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[0664] Example 2

[0665] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] Conventional customer support systems simply convert customer voice into text, but are unable to take into account the customer's emotional state. This makes it difficult to provide appropriate responses, particularly to emotional inquiries, resulting in lower customer satisfaction. There is a need for a system that can solve this problem and provide quick and appropriate answers.

[0667] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means that receives the customer's voice and converts the voice data into text data, an emotion recognition means that recognizes the emotional state based on the tone, speed, and pitch of the voice, and a natural language processing means that analyzes the text data and classifies the inquiry content. This enables a quick and appropriate response taking into account the emotional state of the customer.

[0668] The "voice recognition means" is a means for receiving voice data from a customer and converting it into text data.

[0669] An "emotion recognition means" is a means for analyzing the tone, speed, and pitch of a voice to recognize the emotional state of a customer.

[0670] "Natural language processing means" refers to means for analyzing text data and classifying its contents.

[0671] The "processing means" is a means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer.

[0672] The "means for generating an answer" is a means for automatically generating an answer based on the content of the inquiry.

[0673] A "means for transferring a call" is a means for transferring a call to an appropriate person.

[0674] The "voice synthesis means" is a means for converting the generated response into voice data and providing it to the customer.

[0675] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, making it possible to respond according to the customer's emotional state. Specific embodiments are described in detail below.

[0676] Voice reception and recognition

[0677] When a user calls and speaks their inquiry, the server receives the customer's voice data. For example, if a customer says, "Please tell me how to use product A," services such as Twilio and VoIP are used to receive this voice data in real time. The received voice data is passed to the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[0678] emotion recognition

[0679] The server passes the voice data to an emotion recognition means (e.g., IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if a customer says excitedly, "I don't know how to use product A!", the emotion engine will recognize the emotional state as "anxiety" or "anger."

[0680] Content analysis

[0681] The server passes the converted text data to a natural language processing means (e.g., a custom engine using the BERT model) and analyzes its contents. Based on the analysis results, this means classifies the inquiry content into a specific category (e.g., "usage guide" or "troubleshooting"). Furthermore, the analysis results of the emotion engine are also taken into consideration. For example, the text "Please tell me how to use product A" is analyzed and classified as a "usage guide," while also taking into account the customer's emotion of "anxiety."

[0682] Determining the appropriate response

[0683] The server determines the appropriate response based on the inquiry category and the customer's emotional state. For example, it automatically generates an answer for simple FAQs, but transfers the call to the appropriate agent for complex, emotional inquiries. For example, if the inquiry is classified as a "How to Use Guide" and the customer's emotion is "Anxiety," the call will be transferred to an agent.

[0684] Generate and provide answers

[0685] When the server automatically generates an answer, it uses a generative AI model (e.g., OpenAI's GPT-3) to generate the answer text. It then converts it into audio data using a speech synthesis API (e.g., Google Cloud Text-to-Speech) and provides it to the customer. For example, it generates text such as "To use product A, first turn it on...", converts it into audio data, and responds to the customer.

[0686] On the other hand, when transferring a call to a representative, appropriate information (e.g., inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device, and the call with the customer continues. For example, information such as "This is an inquiry about how to use product A. The customer is currently in a state of anxiety" is displayed on the representative's device, and the call is connected to the representative.

[0687] Termination processing

[0688] After the call ends, the server records the inquiry and the response result in a database. This allows subsequent inquiries to be handled promptly. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Response by the person in charge, customer's emotion: Anxiety" is saved.

[0689] Prompt Sentence Examples

[0690] By using the following prompt statements, you can execute each processing step of the system in detail.

[0691] Convert the customer's speech into text. For example, if they say "How do I use product A?", return that text.

[0692] Analyze customer sentiment from this audio data to determine whether they are in a state of anxiety.

[0693] Pass this text data to a natural language processing engine and classify the inquiry content into the "How to Use Guide" category.

[0694] This system is an effective means of responding to customer inquiries quickly and accurately, and by taking into account the emotional state of the customer, it can provide an individualized response, which is expected to result in improved customer satisfaction and improved operator efficiency.

[0695] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0696] Step 1:

[0697] A user makes a call and speaks their inquiry. For example, the user might say, "Please tell me how to use product A." This voice data is sent to the server.

[0698] Input: User's voice data

[0699] Output: Raw audio data sent to the server

[0700] Step 2:

[0701] The server stores the voice data received via Twilio, etc. This stored voice data is then sent to the Google Cloud Speech-to-Text API to be converted into text data.

[0702] Input: Audio data received by the server

[0703] Output: Text data generated by the Google Cloud Speech-to-Text API (e.g., "How do I use Product A?")

[0704] Step 3:

[0705] The server sends the converted text data and raw voice data to an emotion recognition tool (an API such as IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state.

[0706] Input: Text data and raw audio data

[0707] Output: Emotional state (e.g., "anxious" or "angry")

[0708] Step 4:

[0709] The server sends the text data to a natural language processing engine (such as a custom engine using the BERT model) to analyze the query, which then categorizes the query into categories such as "How to use guide" or "Troubleshooting."

[0710] Input: Text data

[0711] Output: Category label (e.g. "How to use guide")

[0712] Step 5:

[0713] The server determines the appropriate response based on the category of the inquiry and the customer's emotional state: simple inquiries are answered by an automated system, while complex or emotional inquiries are transferred to a human agent.

[0714] Input: category label, emotional state

[0715] Output: Decision on appropriate response (e.g., generating an auto-attendant or deciding on call forwarding)

[0716] Step 6:

[0717] If an automated response is selected, the server generates a text response using a generative AI model (such as OpenAI's GPT-3), then sends this text to a speech synthesis API (such as Google Cloud Text-to-Speech) to convert it into audio data.

[0718] Input: Inquiry content based on category labels, prompt sentences to input to the generative AI model

[0719] Output: Generated answer text and audio data (e.g., "To use product A, first turn it on...")

[0720] Step 7:

[0721] The server provides the generated voice data to the user. On the other hand, when transferring a call, the server displays appropriate information (such as the inquiry content, past inquiry history, and emotional state) on the agent's terminal and connects the call to the agent.

[0722] Input: Generated voice data or information to be conveyed to the agent

[0723] Output: A spoken response to the user or information displayed on the agent's terminal (e.g., "This is a question about how to use product A. The customer is currently in a state of anxiety.")

[0724] Step 8:

[0725] After the call ends, the server records the inquiry and the response results in a database, allowing future inquiries to be handled promptly and appropriately.

[0726] Input: Call termination information, inquiry details, response results, customer emotional state

[0727] Output: Inquiry and response history stored in the database (e.g., "X month, X day, 2023, inquiry content: How to use product A, response result: Response by a staff member, customer emotion: Anxiety")

[0728] By implementing the above processing steps, the system of the present invention can respond to customer inquiries quickly and accurately. Furthermore, by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and operator efficiency.

[0729] (Application example 2)

[0730] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] Conventional speech recognition systems have difficulty accurately recognizing and responding to a customer's emotional state, and lack a means to intuitively and visually guide the appropriate response. As a result, it has been difficult to improve customer satisfaction, and the burden on operators has increased. This invention aims to solve these problems by simultaneously converting customer speech into text and recognizing emotions, and visually guiding the appropriate response.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0733] In this invention, the server includes a speech recognition means for receiving a customer's voice and converting the speech data into text data, a natural language processing means for analyzing the text data and classifying the inquiry content, an emotion recognition means for analyzing the text data and recognizing the emotional state of the customer, a processing means for determining an appropriate response based on the inquiry content and the emotional state of the customer, and a smart device display means for visually guiding the appropriate response. This enables a quick and appropriate response that takes the customer's emotions into consideration, thereby improving customer satisfaction.

[0734] The "voice recognition means" is a device that receives the customer's voice and converts the voice data into text data.

[0735] The "natural language processing means" is a device that analyzes the text data and classifies the inquiry content.

[0736] The "emotion recognition means" is a device that analyzes the text data and recognizes the emotional state of the customer.

[0737] The "processing means" is a device that determines an appropriate response based on the content of the inquiry and the emotional state of the customer.

[0738] The "voice synthesis means" is a device that converts the generated response into voice data and provides it to the customer.

[0739] The "smart device display means" is a device with a display function for visually guiding the appropriate response.

[0740] MODE FOR CARRYING OUT THE INVENTION

[0741] Voice reception and recognition

[0742] User: A customer speaks their inquiry at the store. For example, they say, "Do you have this product in stock?"

[0743] Server: Receives the customer's voice data through the microphone attached to the smart glasses and passes it to a voice recognition means (for example, Google voice recognition API). This voice recognition means converts the voice data into text data.

[0744] emotion recognition

[0745] Server: Passes the text data to an emotion recognition means (for example, the Distilroberta emotion analysis model), analyzes the tone, speed, and pitch of the voice, and recognizes the customer's emotional state (for example, joy, anger, anxiety).

[0746] Content analysis

[0747] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, the inquiry is classified into a specific category (e.g., "inventory check" or "product specifications"). Emotion recognition results are also taken into account.

[0748] Determining the appropriate response

[0749] Server: Determines the appropriate response based on the category of inquiry and the customer's emotional state. For example, if a customer is anxious about checking inventory, the server will guide the salesperson to respond quickly and courteously.

[0750] Providing a visual guide

[0751] Device: The smart glasses display will display appropriate guidance, such as a message like "Checking inventory. Please wait a moment."

[0752] Providing answers

[0753] Server: If an automatic reply is possible, a generative AI model is used to generate a reply text, which is then converted into voice data using a voice synthesis means and provided.

[0754] Termination processing

[0755] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[0756] Hardware and software used

[0757] Hardware used: Smart glasses, server

[0758] Software used: Google speech recognition API, NLP engine, distilroberta sentiment analysis model, generative AI model

[0759] Adding specific examples

[0760] Examples:

[0761] A store associate wears smart glasses and a customer asks, "Do you have this item in stock?"

[0762] Customer speech is transcribed in real time: "Do you have this item in stock?"

[0763] The sentiment analysis engine detects "interests."

[0764] The smart glasses display a guide that reads, "Please check stock and let us know immediately."

[0765] Example prompt for a generative AI model:

[0766] If a customer excitedly asks, "Do you have this item in stock?", what advice should the sales associate give them?

[0767] In this way, the present invention provides a system that can analyze customer voices in real time, recognize their emotional state, and provide appropriate responses. This system is expected to improve customer satisfaction and streamline store operations.

[0768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0769] Step 1:

[0770] The user makes a voice inquiry through a microphone in the smart glasses worn by the store clerk, and the input is captured as voice data.

[0771] Step 2:

[0772] The server receives voice data from the smart glasses. The input is voice data, which is passed to a voice recognition means (Google voice recognition API) and converted into text data. The output is text data.

[0773] Step 3:

[0774] The server passes the converted text data to an emotion recognition means (distilroberta emotion analysis model). The input is text data, and the data is processed to analyze the emotional state. The output is the customer's emotional state data (e.g., joy, anger, anxiety).

[0775] Step 4:

[0776] The server passes the text data to a natural language processing engine (NLP engine), which analyzes the content and classifies the inquiry into specific categories. The input is text data, and content analysis and category classification are performed as data operations. The output is category data (for example, "inventory check" or "product specifications").

[0777] Step 5:

[0778] The server uses a processing means to determine an appropriate response based on the category data of the inquiry content and the emotional state data of the customer. The input is the category data and the emotional state data, and the response is determined as a data calculation. The output is an appropriate response guideline.

[0779] Step 6:

[0780] The terminal (smart glasses) visually displays the determined correspondence guidelines. The input is the correspondence guidelines, which are converted into a display format as data processing. The output is a guide message displayed on the smart glasses display.

[0781] Step 7:

[0782] The server automatically generates answer text using a generative AI model as needed, and converts it into voice data using a voice synthesis means. The input is answer text data, which is converted into voice data as data processing. The output is voice data.

[0783] Step 8:

[0784] The user responds appropriately to the customer by following the guide message on the display of the smart glasses. The input is the guide message displayed on the display of the smart glasses, and the output is the actual customer response action.

[0785] Step 9:

[0786] After the response is completed, the server records the inquiry details and the response results in a database. The input is the response result data, which is converted into a recording format as data processing. The output is the record data saved in the database.

[0787] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0788] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0789] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0790] [Third embodiment]

[0791] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0792] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0793] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0794] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0795] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0796] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0797] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0798] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0799] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0800] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0801] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0802] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0803] The system of the present invention receives customer voice in real time, converts the voice into text data, and analyzes the text data to determine an appropriate response. Specific embodiments will be described in detail below from the perspectives of the server, terminal, and user.

[0804] Voice reception and recognition

[0805] User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0806] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0807] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0808] Content analysis

[0809] Server: The converted text data is passed to a natural language processing means for analysis. Based on the analysis results, this means classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0810] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "Usage Guide."

[0811] Determining the appropriate response

[0812] Server: Based on the analysis results, it determines the appropriate response. For example, it automatically generates an answer for simple FAQs, and transfers the call to the appropriate person for complex inquiries.

[0813] Example: When the question is classified into the category "How to use guide," a pre-prepared answer is searched for and the answer is given directly or in voice using a voice synthesis means.

[0814] Generate and provide answers

[0815] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[0816] Example: Generate text such as "To use product A, first turn it on, then..." and convert it into voice data using a voice synthesis means. The text then replies to the customer, "To use product A, first turn it on..."

[0817] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, etc.) is displayed on the representative's terminal and the call with the customer continues.

[0818] Example: The information "This is a question about how to use product A. There have been no previous inquiries" is displayed on the representative's device, and the call is connected to the representative.

[0819] Termination processing

[0820] Server: After the call ends, all inquiries and responses are recorded, allowing for prompt responses to subsequent inquiries.

[0821] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0822] This system allows for quick and accurate responses to customer inquiries and avoids the need for cumbersome IVR, which is expected to improve customer satisfaction and increase operator work efficiency.

[0823] The processing flow will be explained below.

[0824] Step 1:

[0825] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0826] Step 2:

[0827] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[0828] Step 3:

[0829] Server: The speech recognition engine converts the received voice data into text data and returns the converted text data to the server.

[0830] Step 4:

[0831] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[0832] Step 5:

[0833] Server: Based on the analysis results of the NLP engine, classify the inquiry into a specific category. For example, determine that the inquiry is related to "How to use guides."

[0834] Step 6:

[0835] Server: Determines the appropriate response based on the category of the inquiry. In this case, since the usage guide is a simple FAQ, it decides to generate an automatic answer.

[0836] Step 7:

[0837] Server: Using a chat generation AI model, it generates an automatic response to the inquiry. For example, it generates a text response such as "To use product A, first turn it on, then..."

[0838] Step 8:

[0839] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice and returns the voice data to the server.

[0840] Step 9:

[0841] Server: Sends voice data to the customer and provides a response over the phone. The server responds to the customer by voice, saying, "To use product A, first turn it on..."

[0842] Step 10:

[0843] Server: After the call ends, the inquiry and response results are recorded in the database. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided via voice synthesis" is saved.

[0844] Step 11:

[0845] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[0846] Example 1

[0847] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0848] In today's world, there is a demand for fast and accurate responses to customer inquiries, but most call centers rely on manual response methods, which can be inefficient and result in low customer satisfaction. Additionally, cumbersome IVR (Interactive Voice Response) responses can be stressful for customers. Furthermore, transferring calls to the appropriate person based on the content of the inquiry and managing inquiry history can be cumbersome.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0850] In this invention, the server includes a speech recognition unit that receives the customer's voice and converts the speech data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This allows the inquiry to be automatically analyzed and an appropriate response to be provided quickly. Furthermore, by adding a recording unit that records the inquiry and the response result after the call ends, subsequent inquiries can also be responded to quickly.

[0851] The "voice recognition means" is a device or program that converts voice data received from a customer into text data in real time.

[0852] The "natural language processing means" is a device or program that analyzes text data and classifies the inquiry content into specific categories based on the content.

[0853] The "processing means" is a device or program that determines an appropriate response based on the content of the inquiry.

[0854] The "answer generating means" is a device or program that automatically generates an answer according to the content of the inquiry.

[0855] A "call forwarding means" is a device or program that forwards a customer's call to the appropriate person.

[0856] The "voice synthesis means" is a device or program that converts the generated text data into voice data and provides it to the customer.

[0857] "Recording means" refers to a device or program that records and saves the content of the inquiry and the response results after the call ends.

[0858] The present invention is a system for responding to customer inquiries quickly and accurately, which receives customer voice in real time, converts the voice into text data, and then analyzes the text data to determine an appropriate response.

[0859] Voice reception and recognition

[0860] 1. User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[0861] 2. Server: The server receives the customer's voice data via the telephone system and passes it to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[0862] Example: A customer says, "Please tell me how to use product A," and a speech recognition API converts this into text: "Please tell me how to use product A."

[0863] Content analysis

[0864] 1. Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0865] Example: Confirm that the text data is "Please tell me how to use product A" and analyze the text using a natural language processing API.

[0866] 2. Server: Receives the analysis results of the natural language processing API and classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[0867] Example: Based on the analysis results, "Inquiries about how to use product A" are classified into the "Usage Guide" category.

[0868] Determining the appropriate response

[0869] 1. Server: Determines the appropriate response based on the identified category, for example automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[0870] Example: Since the category is "How to use guide," the appropriate response is to determine whether to search the FAQ or use speech synthesis.

[0871] Generate and provide answers

[0872] 1. Server: To generate an answer automatically, generate the answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). The generated text is passed to a speech synthesis API (e.g., Amazon Polly) and converted into voice data.

[0873] Example: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0874] 2. Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0875] Example: The generated voice data is played back to the customer via the telephone line. Also, when transferring the call to a representative, the following information is displayed: "This is a question about how to use product A. There have been no previous inquiries."

[0876] Termination processing

[0877] 1. Server: After the call ends, record all inquiries and the response results.

[0878] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[0879] Examples of prompt statements

[0880] When you receive a customer query such as "How do I use product A?", generate appropriate FAQs and relevant answers.

[0881] Example of generated text: Prompt text to be passed to ChatGPT: "A customer asked me, 'How do I use product A?' Please briefly explain the specific steps to take."

[0882] As described above, the system of the present invention aims to efficiently process customer inquiries and improve customer satisfaction.

[0883] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0884] Step 1:

[0885] User: A customer calls and verbally replies with their inquiry. For example, they say, "Please tell me how to use product A." This action generates voice data.

[0886] Step 2:

[0887] Server: The server receives the customer's voice data via the telephone system. It sends the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[0888] Input: Customer voice data.

[0889] Data processing: Converts voice data into text data in real time using a speech recognition API.

[0890] Output: The converted text data.

[0891] What it does: It receives a customer's speech saying "Please tell me how to use product A" and passes it to a speech recognition API, which converts the speech into text saying "Please tell me how to use product A."

[0892] Step 3:

[0893] Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[0894] Input: Text data from the speech recognition API.

[0895] Data processing: Analyze text data using natural language processing APIs and classify the content into specific categories.

[0896] Output: Categorized analysis results.

[0897] Specific operation: The system confirms that the text data is "Please tell me how to use product A," analyzes the text using a natural language processing API, and classifies the inquiry into the "Usage Guide" category based on the analysis results.

[0898] Step 4:

[0899] Server: Determines the appropriate response based on the identified category, for example, automatically generating an answer for a simple FAQ, or routing the call to the appropriate agent for a complex inquiry.

[0900] Input: Parsed results from a natural language processing API.

[0901] Data processing: Applying rule-based algorithms or machine learning models to determine appropriate responses based on the analysis results.

[0902] Output: The appropriate action (auto-generated answer or call transfer instructions).

[0903] Specific operation: Based on the category "How to use guide", it determines whether to search FAQs or use speech synthesis as the appropriate response. If it is a simple FAQ, it searches for a pre-prepared answer and outputs it.

[0904] Step 5:

[0905] Server: When generating answers automatically, generate answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). Pass the generated text to a speech synthesis API (e.g., Amazon Polly) and convert it into voice data.

[0906] Input: FAQ search results or inquiry content from natural language processing API.

[0907] Data processing: Response text is generated using a chat generation AI model and converted into voice data using a speech synthesis API.

[0908] Output: The generated audio data.

[0909] Specific operation: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[0910] Step 6:

[0911] Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[0912] Input: Generated voice data or call transfer instructions.

[0913] Data processing: The generated voice data is played back over the telephone line, or the information is displayed on the person in charge's terminal.

[0914] Output: Response to customer or information provided to agent.

[0915] Specific operation: The generated voice data is played to the customer via the telephone line. Also, when transferring the call to a representative, the information "This is a question about how to use product A. There have been no previous inquiries" is displayed.

[0916] Step 7:

[0917] Server: After the call ends, record all inquiries and the response results.

[0918] Input: Inquiry details and response results after the call ends.

[0919] Data processing: Record the inquiry details and response results in a database.

[0920] Output: Recorded data.

[0921] Specific operation: Save the record "X month, X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" in the database.

[0922] (Application example 1)

[0923] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0924] When customers inquire about products in a virtual store, they need a fast and accurate response. However, conventional systems do not process customer voice interactions in real time, which can lead to lower customer satisfaction. In addition, it is difficult to provide an immediate and appropriate response to complex inquiries, so an effective response method is needed.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0926] In this invention, the server includes a speech recognition unit that receives a customer's voice and converts the voice data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This enables a system that includes a unit including smart glasses that performs real-time voice interaction with customers in a virtual store, a unit that performs the above processing on a cloud server, and a unit that generates answers using a generative AI model.

[0927] "Customer voice" is voice data uttered by customers in the virtual store.

[0928] "Speech recognition means" refers to a technical means for converting a customer's voice data into text data.

[0929] "Text data" is a character string that represents the customer's voice converted by a voice recognition means.

[0930] "Natural language processing means" is a processing technology for analyzing text data and classifying the content of inquiries.

[0931] The "processing means for determining an appropriate response" is a technical means for determining a response method based on the analyzed inquiry content.

[0932] The "means for generating an answer" refers to a technical means for generating an answer based on the content of the inquiry.

[0933] "Means for routing calls to the right person" means the technical means for routing calls to the right person for complex inquiries.

[0934] "Speech synthesis means" refers to a technical means for converting the generated response into voice data and providing it to the customer.

[0935] A "virtual store" is a virtual sales venue operated on the Internet.

[0936] "Voice interaction" refers to two-way communication between a customer and a company via voice.

[0937] "Smart glasses" are wearable devices with the ability to receive and play audio.

[0938] A "cloud server" is a server system for performing processing in a distributed computing environment.

[0939] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[0940] The system of the present invention is a voice interaction system for supporting customer support in a virtual store. This system is composed of a voice recognition unit, a natural language processing unit, a processing unit for determining an appropriate response, a response generation unit, a voice synthesis unit, smart glasses, a cloud server, and a generative AI model.

[0941] Voice reception and recognition

[0942] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0943] Smart glasses: Receives the customer's voice and sends the voice data to the cloud.

[0944] Cloud server: Using a speech recognition tool (e.g., Google Speech-to-Text API), converts the voice data into text data. For example, the voice "Please tell me the features of this product" is converted into the text "Please tell me the features of this product."

[0945] Content analysis

[0946] Cloud server: The converted text data is passed to a natural language processing tool (e.g., OpenAI GPT-3) and its contents are analyzed. Through the analysis, the inquiry content is classified into categories such as "product information" and "usage guide." For example, the text "Please tell me the features of this product" is classified as "product information."

[0947] Determine appropriate responses and generate answers

[0948] Cloud server: Determines the appropriate response based on the analysis results and generates an answer corresponding to the inquiry using a generative AI model (e.g., GPT-3). The generated answer text might be, for example, "The features of this product are as follows..."

[0949] Providing answers

[0950] Cloud server: The generated answer is converted into voice data using a voice synthesis means (e.g., Amazon Polly). The converted voice data is sent to the smart glasses and provided to the customer.

[0951] Smart glasses: Plays back audio data and provides answers to customers, such as "The features of this product are as follows..."

[0952] Examples and Prompts

[0953] As a concrete example, if a customer asks, "What are the features of this product?", the above steps will return a spoken response saying, "The features of this product are as follows..." An example of a prompt sentence when generating an answer using a generative AI model is as follows:

[0954] "You have a customer who wants to know the features of your product. Please answer the following question: What are the features of this product?"

[0955] This system allows customers in virtual stores to obtain quick and accurate information through voice interaction, improving the user experience.

[0956] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0957] Step 1:

[0958] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[0959] Input: Customer voice data

[0960] Output: Input of audio data to smart glasses

[0961] Specific operation: The customer's voice is collected through the microphone in the smart glasses.

[0962] Step 2:

[0963] Smart glasses: Collected customer voice data is sent to the cloud.

[0964] Input: Customer voice data

[0965] Output: Sending audio data to the cloud

[0966] Specific operation: The smart glasses upload the audio data to the cloud server.

[0967] Step 3:

[0968] Server: Converts the voice data into text data using a voice recognition means (e.g., Google Speech-to-Text API).

[0969] Input: Audio data

[0970] Output: Text data

[0971] Specific operation: Calls the Google Speech-to-Text API and converts the voice data into text. For example, "What are the features of this product?" becomes the text "What are the features of this product?"

[0972] Step 4:

[0973] Server: The converted text data is passed to a natural language processing means (e.g., OpenAI GPT-3) and its content is analyzed.

[0974] Input: Text data

[0975] Output: Category as analysis result (e.g. "Product Information", "Usage Guide", etc.)

[0976] How it works: The GPT-3 model inputs text data, performs content analysis, and classifies the query into the appropriate category. For example, the text "What are the features of this product?" is classified as "Product Information."

[0977] Step 5:

[0978] Server: Based on the analysis results, it generates answers using a generative AI model.

[0979] Input: Parsed categories and original text data

[0980] Output: Generated answer text

[0981] How it works: Using the GPT-3 model, it generates answers based on the analyzed categories. For example, based on a prompt such as "A customer wants to know the features of a product. Please answer the following question: What are the features of this product?", it generates an answer such as "The features of this product are as follows..."

[0982] Step 6:

[0983] Server: The generated answer text is converted into voice data using a speech synthesis means (e.g., Amazon Polly) and sent to the smart glasses.

[0984] Input: Generated answer text

[0985] Output: Audio data

[0986] Specific operation: Using Amazon Polly, the answer text is converted into speech data and sent to the smart glasses. For example, the text "The features of this product are as follows..." is converted into speech data.

[0987] Step 7:

[0988] Smart glasses: Plays back audio data and provides answers to customers.

[0989] Input: Audio data

[0990] Output: Audio output to the customer

[0991] Specific operation: Play a voice through the speaker of the smart glasses and reply to the customer, for example, "The features of this product are as follows..."

[0992] Through these steps, customers can access real-time product information in the virtual store, improving the user experience.

[0993] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0994] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, enabling it to respond according to the customer's emotional state. A specific embodiment will be described below in detail from the perspectives of the server, terminal, and user.

[0995] Voice reception and recognition

[0996] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[0997] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[0998] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[0999] emotion recognition

[1000] Server: Passes the customer's voice data to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state (e.g., anger, anxiety, satisfaction).

[1001] Example: If a customer excitedly says, "I don't know how to use product A!", the emotion engine will recognize that the customer is in an emotional state of "anxious" or "angry."

[1002] Content analysis

[1003] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, this engine classifies the inquiry into a specific category (e.g., "How to use guide" or "Troubleshooting"). The analysis results of the emotion engine are also taken into account.

[1004] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "How to use guide" while taking into account the customer's emotion of "anxiety."

[1005] Determining the appropriate response

[1006] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ or routing the call to the right agent for a complex, emotional inquiry.

[1007] Example: If the call falls into the category "How to Use Guide" and the customer's emotion is "Anxiety," it is determined that a more flexible and gentle response is required, and the call is transferred to a representative.

[1008] Generate and provide answers

[1009] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[1010] Example: A text response such as "To use product A, first turn it on, then..." is generated and converted into voice data using a voice synthesis means. The response to the customer is "To use product A, first turn it on...".

[1011] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, emotional state, etc.) is displayed on the representative's terminal and the call with the customer continues.

[1012] Example: The information "This is a question about how to use product A. The customer is currently in a state of anxiety" is displayed on the agent's device, and the call is connected to the agent.

[1013] Termination processing

[1014] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[1015] Example: Save a record that reads, "X month, X day, 2023, inquiry content: How to use product A. Response result: Response by a staff member. Customer emotion: Anxiety."

[1016] This system allows for quick and accurate responses to customer inquiries, and by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and increase the work efficiency of operators.

[1017] The processing flow will be explained below.

[1018] Step 1:

[1019] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[1020] Step 2:

[1021] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[1022] Step 3:

[1023] Server: The speech recognition engine converts the received voice data into text data in real time and returns the converted text data to the server.

[1024] Step 4:

[1025] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[1026] Step 5:

[1027] Server: Passes the voice data from the customer to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if the customer is angry, their emotional state is recognized as "anger."

[1028] Step 6:

[1029] Server: Integrates the text analysis results from the NLP engine and the emotion recognition results from the emotion engine, and classifies the inquiry content into specific categories (e.g., "How to use guide" or "Troubleshooting").

[1030] Step 7:

[1031] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ or routing the call to the right agent for a complex, emotional inquiry.

[1032] Step 8:

[1033] Server: When generating answers automatically, the server uses a chat generation AI model to generate answer text. For example, it generates text like, "To use product A, first turn it on, then..."

[1034] Step 9:

[1035] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice data and returns the voice data to the server.

[1036] Step 10:

[1037] Server: Provides voice data to the customer and responds over the phone. For example, it responds to the customer by voice, saying, "To use product A, first turn it on..."

[1038] Step 11:

[1039] Server: When transferring a call to a representative, appropriate information (such as the inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device so that the call with the customer can continue.

[1040] Step 12:

[1041] Terminal: The agent interacts with the customer based on the information displayed on the terminal. For example, the terminal may display, "This is a question about how to use product A. The customer is currently in a state of anxiety."

[1042] Step 13:

[1043] Server: After the call ends, record the inquiry details and the response results in the database. For example, record "Xth month, 2023, inquiry details: How to use product A, response results: Response by the person in charge, customer emotion: Anxiety."

[1044] Step 14:

[1045] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[1046] Example 2

[1047] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] Conventional customer support systems simply convert customer voice into text, but are unable to take into account the customer's emotional state. This makes it difficult to provide appropriate responses, particularly to emotional inquiries, resulting in lower customer satisfaction. There is a need for a system that can solve this problem and provide quick and appropriate answers.

[1049] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means that receives the customer's voice and converts the voice data into text data, an emotion recognition means that recognizes the emotional state based on the tone, speed, and pitch of the voice, and a natural language processing means that analyzes the text data and classifies the inquiry content. This enables a quick and appropriate response taking into account the emotional state of the customer.

[1050] The "voice recognition means" is a means for receiving voice data from a customer and converting it into text data.

[1051] An "emotion recognition means" is a means for analyzing the tone, speed, and pitch of a voice to recognize the emotional state of a customer.

[1052] "Natural language processing means" refers to means for analyzing text data and classifying its contents.

[1053] The "processing means" is a means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer.

[1054] The "means for generating an answer" is a means for automatically generating an answer based on the content of the inquiry.

[1055] A "means for transferring a call" is a means for transferring a call to an appropriate person.

[1056] The "voice synthesis means" is a means for converting the generated response into voice data and providing it to the customer.

[1057] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, making it possible to respond according to the customer's emotional state. Specific embodiments are described in detail below.

[1058] Voice reception and recognition

[1059] When a user calls and speaks their inquiry, the server receives the customer's voice data. For example, if a customer says, "Please tell me how to use product A," services such as Twilio and VoIP are used to receive this voice data in real time. The received voice data is passed to the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[1060] emotion recognition

[1061] The server passes the voice data to an emotion recognition means (e.g., IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if a customer says excitedly, "I don't know how to use product A!", the emotion engine will recognize the emotional state as "anxiety" or "anger."

[1062] Content analysis

[1063] The server passes the converted text data to a natural language processing means (e.g., a custom engine using the BERT model) and analyzes its contents. Based on the analysis results, this means classifies the inquiry content into a specific category (e.g., "usage guide" or "troubleshooting"). Furthermore, the analysis results of the emotion engine are also taken into consideration. For example, the text "Please tell me how to use product A" is analyzed and classified as a "usage guide," while also taking into account the customer's emotion of "anxiety."

[1064] Determining the appropriate response

[1065] The server determines the appropriate response based on the inquiry category and the customer's emotional state. For example, it automatically generates an answer for simple FAQs, but transfers the call to the appropriate agent for complex, emotional inquiries. For example, if the inquiry is classified as a "How to Use Guide" and the customer's emotion is "Anxiety," the call will be transferred to an agent.

[1066] Generate and provide answers

[1067] When the server automatically generates an answer, it uses a generative AI model (e.g., OpenAI's GPT-3) to generate the answer text. It then converts it into audio data using a speech synthesis API (e.g., Google Cloud Text-to-Speech) and provides it to the customer. For example, it generates text such as "To use product A, first turn it on...", converts it into audio data, and responds to the customer.

[1068] On the other hand, when transferring a call to a representative, appropriate information (e.g., inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device, and the call with the customer continues. For example, information such as "This is an inquiry about how to use product A. The customer is currently in a state of anxiety" is displayed on the representative's device, and the call is connected to the representative.

[1069] Termination processing

[1070] After the call ends, the server records the inquiry and the response result in a database. This allows subsequent inquiries to be handled promptly. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Response by the person in charge, customer's emotion: Anxiety" is saved.

[1071] Prompt Sentence Examples

[1072] By using the following prompt statements, you can execute each processing step of the system in detail.

[1073] Convert the customer's speech into text. For example, if they say "How do I use product A?", return that text.

[1074] Analyze customer sentiment from this audio data to determine whether they are in a state of anxiety.

[1075] Pass this text data to a natural language processing engine and classify the inquiry content into the "How to Use Guide" category.

[1076] This system is an effective means of responding to customer inquiries quickly and accurately, and by taking into account the emotional state of the customer, it can provide an individualized response, which is expected to result in improved customer satisfaction and improved operator efficiency.

[1077] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1078] Step 1:

[1079] A user makes a call and speaks their inquiry. For example, the user might say, "Please tell me how to use product A." This voice data is sent to the server.

[1080] Input: User's voice data

[1081] Output: Raw audio data sent to the server

[1082] Step 2:

[1083] The server stores the voice data received via Twilio, etc. This stored voice data is then sent to the Google Cloud Speech-to-Text API to be converted into text data.

[1084] Input: Audio data received by the server

[1085] Output: Text data generated by the Google Cloud Speech-to-Text API (e.g., "How do I use Product A?")

[1086] Step 3:

[1087] The server sends the converted text data and raw voice data to an emotion recognition tool (an API such as IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state.

[1088] Input: Text data and raw audio data

[1089] Output: Emotional state (e.g., "anxious" or "angry")

[1090] Step 4:

[1091] The server sends the text data to a natural language processing engine (such as a custom engine using the BERT model) to analyze the query, which then categorizes the query into categories such as "How to use guide" or "Troubleshooting."

[1092] Input: Text data

[1093] Output: Category label (e.g. "How to use guide")

[1094] Step 5:

[1095] The server determines the appropriate response based on the category of the inquiry and the customer's emotional state: simple inquiries are answered by an automated system, while complex or emotional inquiries are transferred to a human agent.

[1096] Input: category label, emotional state

[1097] Output: Decision on appropriate response (e.g., generating an auto-attendant or deciding on call forwarding)

[1098] Step 6:

[1099] If an automated response is selected, the server generates a text response using a generative AI model (such as OpenAI's GPT-3), then sends this text to a speech synthesis API (such as Google Cloud Text-to-Speech) to convert it into audio data.

[1100] Input: Inquiry content based on category labels, prompt sentences to input to the generative AI model

[1101] Output: Generated answer text and audio data (e.g., "To use product A, first turn it on...")

[1102] Step 7:

[1103] The server provides the generated voice data to the user. On the other hand, when transferring a call, the server displays appropriate information (such as the inquiry content, past inquiry history, and emotional state) on the agent's terminal and connects the call to the agent.

[1104] Input: Generated voice data or information to be conveyed to the agent

[1105] Output: A spoken response to the user or information displayed on the agent's terminal (e.g., "This is a question about how to use product A. The customer is currently in a state of anxiety.")

[1106] Step 8:

[1107] After the call ends, the server records the inquiry and the response results in a database, allowing future inquiries to be handled promptly and appropriately.

[1108] Input: Call termination information, inquiry details, response results, customer emotional state

[1109] Output: Inquiry and response history stored in the database (e.g., "X month, X day, 2023, inquiry content: How to use product A, response result: Response by a staff member, customer emotion: Anxiety")

[1110] By implementing the above processing steps, the system of the present invention can respond to customer inquiries quickly and accurately. Furthermore, by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and operator efficiency.

[1111] (Application example 2)

[1112] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1113] Conventional speech recognition systems have difficulty accurately recognizing and responding to a customer's emotional state, and lack a means to intuitively and visually guide the appropriate response. As a result, it has been difficult to improve customer satisfaction, and the burden on operators has increased. This invention aims to solve these problems by simultaneously converting customer speech into text and recognizing emotions, and visually guiding the appropriate response.

[1114] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1115] In this invention, the server includes a speech recognition means for receiving a customer's voice and converting the speech data into text data, a natural language processing means for analyzing the text data and classifying the inquiry content, an emotion recognition means for analyzing the text data and recognizing the emotional state of the customer, a processing means for determining an appropriate response based on the inquiry content and the emotional state of the customer, and a smart device display means for visually guiding the appropriate response. This enables a quick and appropriate response that takes the customer's emotions into consideration, thereby improving customer satisfaction.

[1116] The "voice recognition means" is a device that receives the customer's voice and converts the voice data into text data.

[1117] The "natural language processing means" is a device that analyzes the text data and classifies the inquiry content.

[1118] The "emotion recognition means" is a device that analyzes the text data and recognizes the emotional state of the customer.

[1119] The "processing means" is a device that determines an appropriate response based on the content of the inquiry and the emotional state of the customer.

[1120] The "voice synthesis means" is a device that converts the generated response into voice data and provides it to the customer.

[1121] The "smart device display means" is a device with a display function for visually guiding the appropriate response.

[1122] MODE FOR CARRYING OUT THE INVENTION

[1123] Voice reception and recognition

[1124] User: A customer speaks their inquiry at the store. For example, they say, "Do you have this product in stock?"

[1125] Server: Receives the customer's voice data through the microphone attached to the smart glasses and passes it to a voice recognition means (for example, Google voice recognition API). This voice recognition means converts the voice data into text data.

[1126] emotion recognition

[1127] Server: Passes the text data to an emotion recognition means (for example, the Distilroberta emotion analysis model), analyzes the tone, speed, and pitch of the voice, and recognizes the customer's emotional state (for example, joy, anger, anxiety).

[1128] Content analysis

[1129] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, the inquiry is classified into a specific category (e.g., "inventory check" or "product specifications"). Emotion recognition results are also taken into consideration.

[1130] Determining the appropriate response

[1131] Server: Determines the appropriate response based on the category of inquiry and the customer's emotional state. For example, if a customer is anxious about checking inventory, the server will guide the salesperson to respond quickly and courteously.

[1132] Providing a visual guide

[1133] Device: The smart glasses display will display appropriate guidance, such as a message like "Checking inventory. Please wait a moment."

[1134] Providing answers

[1135] Server: If an automatic reply is possible, a generative AI model is used to generate a reply text, which is then converted into voice data using a voice synthesis means and provided.

[1136] Termination processing

[1137] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[1138] Hardware and software used

[1139] Hardware used: Smart glasses, server

[1140] Software used: Google speech recognition API, NLP engine, distilroberta sentiment analysis model, generative AI model

[1141] Adding specific examples

[1142] Examples:

[1143] A store associate wears smart glasses and a customer asks, "Do you have this item in stock?"

[1144] Customer speech is transcribed in real time: "Do you have this item in stock?"

[1145] The sentiment analysis engine detects "interests."

[1146] The smart glasses display a guide that reads, "Please check stock and let us know immediately."

[1147] Example prompt for a generative AI model:

[1148] If a customer excitedly asks, "Do you have this item in stock?", what advice should the sales associate give them?

[1149] In this way, the present invention provides a system that can analyze customer voices in real time, recognize their emotional state, and provide appropriate responses. This system is expected to improve customer satisfaction and streamline store operations.

[1150] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1151] Step 1:

[1152] The user makes a voice inquiry through a microphone in the smart glasses worn by the store clerk, and the input is captured as voice data.

[1153] Step 2:

[1154] The server receives voice data from the smart glasses. The input is voice data, which is passed to a voice recognition means (Google voice recognition API) and converted into text data. The output is text data.

[1155] Step 3:

[1156] The server passes the converted text data to an emotion recognition means (distilroberta emotion analysis model). The input is text data, and the data is processed to analyze the emotional state. The output is the customer's emotional state data (e.g., joy, anger, anxiety).

[1157] Step 4:

[1158] The server passes the text data to a natural language processing engine (NLP engine), which analyzes the content and classifies the inquiry into specific categories. The input is text data, and content analysis and category classification are performed as data operations. The output is category data (for example, "inventory check" or "product specifications").

[1159] Step 5:

[1160] The server uses a processing means to determine an appropriate response based on the category data of the inquiry content and the emotional state data of the customer. The input is the category data and the emotional state data, and the response is determined as a data calculation. The output is an appropriate response guideline.

[1161] Step 6:

[1162] The terminal (smart glasses) visually displays the determined correspondence guidelines. The input is the correspondence guidelines, which are converted into a display format as data processing. The output is a guide message displayed on the smart glasses display.

[1163] Step 7:

[1164] The server automatically generates answer text using a generative AI model as needed, and converts it into voice data using a voice synthesis means. The input is answer text data, which is converted into voice data as data processing. The output is voice data.

[1165] Step 8:

[1166] The user responds appropriately to the customer by following the guide message on the display of the smart glasses. The input is the guide message displayed on the display of the smart glasses, and the output is the actual customer response action.

[1167] Step 9:

[1168] After the response is completed, the server records the inquiry details and the response results in a database. The input is the response result data, which is converted into a recording format as data processing. The output is the record data saved in the database.

[1169] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1170] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1171] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1172] [Fourth embodiment]

[1173] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1174] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1175] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1176] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1177] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1178] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1179] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1180] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1181] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1182] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1183] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1184] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1185] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1186] The system of the present invention receives customer voice in real time, converts the voice into text data, and analyzes the text data to determine an appropriate response. Specific embodiments will be described in detail below from the perspectives of the server, terminal, and user.

[1187] Voice reception and recognition

[1188] User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[1189] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[1190] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[1191] Content analysis

[1192] Server: The converted text data is passed to a natural language processing means for analysis. Based on the analysis results, this means classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[1193] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "Usage Guide."

[1194] Determining the appropriate response

[1195] Server: Based on the analysis results, it determines the appropriate response. For example, it automatically generates an answer for simple FAQs, and transfers the call to the appropriate person for complex inquiries.

[1196] Example: When the question is classified into the category "How to use guide," a pre-prepared answer is searched for and the answer is given directly or in voice using a voice synthesis means.

[1197] Generate and provide answers

[1198] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[1199] Example: Generate text such as "To use product A, first turn it on, then..." and convert it into voice data using a voice synthesis means. The text then replies to the customer, "To use product A, first turn it on..."

[1200] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, etc.) is displayed on the representative's terminal and the call with the customer continues.

[1201] Example: The information "This is a question about how to use product A. There have been no previous inquiries" is displayed on the representative's device, and the call is connected to the representative.

[1202] Termination processing

[1203] Server: After the call ends, all inquiries and responses are recorded, allowing for prompt responses to subsequent inquiries.

[1204] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[1205] This system allows for quick and accurate responses to customer inquiries and avoids the need for cumbersome IVR, which is expected to improve customer satisfaction and increase operator work efficiency.

[1206] The processing flow will be explained below.

[1207] Step 1:

[1208] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[1209] Step 2:

[1210] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[1211] Step 3:

[1212] Server: The speech recognition engine converts the received voice data into text data and returns the converted text data to the server.

[1213] Step 4:

[1214] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[1215] Step 5:

[1216] Server: Based on the analysis results of the NLP engine, classify the inquiry into a specific category. For example, determine that the inquiry is related to "How to use guides."

[1217] Step 6:

[1218] Server: Determines the appropriate response based on the category of the inquiry. In this case, since the usage guide is a simple FAQ, it decides to generate an automatic answer.

[1219] Step 7:

[1220] Server: Using a chat generation AI model, it generates an automatic response to the inquiry. For example, it generates a text response such as "To use product A, first turn it on, then..."

[1221] Step 8:

[1222] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice and returns the voice data to the server.

[1223] Step 9:

[1224] Server: Sends voice data to the customer and provides a response over the phone. The server responds to the customer by voice, saying, "To use product A, first turn it on..."

[1225] Step 10:

[1226] Server: After the call ends, the inquiry and response results are recorded in the database. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided via voice synthesis" is saved.

[1227] Step 11:

[1228] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[1229] Example 1

[1230] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] In today's world, there is a demand for fast and accurate responses to customer inquiries, but most call centers rely on manual response methods, which can be inefficient and result in low customer satisfaction. Additionally, cumbersome IVR (Interactive Voice Response) responses can be stressful for customers. Furthermore, transferring calls to the appropriate person based on the content of the inquiry and managing inquiry history can be cumbersome.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1233] In this invention, the server includes a speech recognition unit that receives the customer's voice and converts the speech data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This allows the inquiry to be automatically analyzed and an appropriate response to be provided quickly. Furthermore, by adding a recording unit that records the inquiry and the response result after the call ends, subsequent inquiries can also be responded to quickly.

[1234] The "voice recognition means" is a device or program that converts voice data received from a customer into text data in real time.

[1235] The "natural language processing means" is a device or program that analyzes text data and classifies the inquiry content into specific categories based on the content.

[1236] The "processing means" is a device or program that determines an appropriate response based on the content of the inquiry.

[1237] The "answer generating means" is a device or program that automatically generates an answer according to the content of the inquiry.

[1238] A "call forwarding means" is a device or program that forwards a customer's call to the appropriate person.

[1239] The "voice synthesis means" is a device or program that converts the generated text data into voice data and provides it to the customer.

[1240] "Recording means" refers to a device or program that records and saves the content of the inquiry and the response results after the call ends.

[1241] The present invention is a system for responding to customer inquiries quickly and accurately, which receives customer voice in real time, converts the voice into text data, and then analyzes the text data to determine an appropriate response.

[1242] Voice reception and recognition

[1243] 1. User: A customer calls and verbally communicates their inquiry. For example, "How do I use product A?"

[1244] 2. Server: The server receives the customer's voice data via the telephone system and passes it to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[1245] Example: A customer says, "Please tell me how to use product A," and a speech recognition API converts this into text: "Please tell me how to use product A."

[1246] Content analysis

[1247] 1. Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[1248] Example: Confirm that the text data is "Please tell me how to use product A" and analyze the text using a natural language processing API.

[1249] 2. Server: Receives the analysis results of the natural language processing API and classifies the inquiry content into specific categories (e.g., "usage guide" or "troubleshooting").

[1250] Example: Based on the analysis results, "Inquiries about how to use product A" are classified into the "Usage Guide" category.

[1251] Determining the appropriate response

[1252] 1. Server: Determines the appropriate response based on the identified category, for example automatically generating an answer for a simple FAQ or routing the call to the appropriate agent for a complex inquiry.

[1253] Example: Since the category is "How to use guide," the appropriate response is to determine whether to search the FAQ or use speech synthesis.

[1254] Generate and provide answers

[1255] 1. Server: To generate an answer automatically, generate the answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). The generated text is passed to a speech synthesis API (e.g., Amazon Polly) and converted into voice data.

[1256] Example: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[1257] 2. Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[1258] Example: The generated voice data is played back to the customer via the telephone line. Also, when transferring the call to a representative, the following information is displayed: "This is a question about how to use product A. There have been no previous inquiries."

[1259] Termination processing

[1260] 1. Server: After the call ends, record all inquiries and the response results.

[1261] Example: A record such as "X month X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" is saved in the database.

[1262] Examples of prompt statements

[1263] When you receive a customer query such as "How do I use product A?", generate appropriate FAQs and relevant answers.

[1264] Example of generated text: Prompt text to be passed to ChatGPT: "A customer asked me, 'How do I use product A?' Please briefly explain the specific steps to take."

[1265] As described above, the system of the present invention aims to efficiently process customer inquiries and improve customer satisfaction.

[1266] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1267] Step 1:

[1268] User: A customer calls and verbally replies with their inquiry. For example, they say, "Please tell me how to use product A." This action generates voice data.

[1269] Step 2:

[1270] Server: The server receives the customer's voice data via the telephone system. It sends the received voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the voice data into text data.

[1271] Input: Customer voice data.

[1272] Data processing: Converts voice data into text data in real time using a speech recognition API.

[1273] Output: The converted text data.

[1274] What it does: It receives a customer's speech saying "Please tell me how to use product A" and passes it to a speech recognition API, which converts the speech into text saying "Please tell me how to use product A."

[1275] Step 3:

[1276] Server: The text data obtained by speech recognition is sent to a natural language processing API (e.g., Google Cloud Natural Language API) and its content is analyzed.

[1277] Input: Text data from the speech recognition API.

[1278] Data processing: Analyze text data using natural language processing APIs and classify the content into specific categories.

[1279] Output: Categorized analysis results.

[1280] Specific operation: The system confirms that the text data is "Please tell me how to use product A," analyzes the text using a natural language processing API, and classifies the inquiry into the "Usage Guide" category based on the analysis results.

[1281] Step 4:

[1282] Server: Determines the appropriate response based on the identified category, for example, automatically generating an answer for a simple FAQ, or routing the call to the appropriate agent for a complex inquiry.

[1283] Input: Parsed results from a natural language processing API.

[1284] Data processing: Applying rule-based algorithms or machine learning models to determine appropriate responses based on the analysis results.

[1285] Output: The appropriate action (auto-generated answer or call transfer instructions).

[1286] Specific operation: Based on the category "How to use guide", it determines whether to search FAQs or use speech synthesis as the appropriate response. If it is a simple FAQ, it searches for a pre-prepared answer and outputs it.

[1287] Step 5:

[1288] Server: When generating answers automatically, generate answer text using a chat generation AI model (e.g., OpenAI's ChatGPT). Pass the generated text to a speech synthesis API (e.g., Amazon Polly) and convert it into voice data.

[1289] Input: FAQ search results or inquiry content from natural language processing API.

[1290] Data processing: Response text is generated using a chat generation AI model and converted into voice data using a speech synthesis API.

[1291] Output: The generated audio data.

[1292] Specific operation: Enter the prompt "Please tell me how to use product A" into ChatGPT, generate a response text, pass it to the speech synthesis API, and convert it into voice data such as "To use product A, first turn it on..."

[1293] Step 6:

[1294] Terminal: Plays the generated voice data as a response to the customer, or if transferring the call to a representative, displays the necessary information (inquiry details, past call history, etc.) on the representative's terminal.

[1295] Input: Generated voice data or call transfer instructions.

[1296] Data processing: The generated voice data is played back over the telephone line, or the information is displayed on the person in charge's terminal.

[1297] Output: Response to customer or information provided to agent.

[1298] Specific operation: The generated voice data is played to the customer via the telephone line. Also, when transferring the call to a representative, the information "This is a question about how to use product A. There have been no previous inquiries" is displayed.

[1299] Step 7:

[1300] Server: After the call ends, record all inquiries and the response results.

[1301] Input: Inquiry details and response results after the call ends.

[1302] Data processing: Record the inquiry details and response results in a database.

[1303] Output: Recorded data.

[1304] Specific operation: Save the record "X month, X day, 2023, inquiry: How to use product A, response result: Answer provided by voice synthesis" in the database.

[1305] (Application example 1)

[1306] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1307] When customers inquire about products in a virtual store, they need a fast and accurate response. However, conventional systems do not process customer voice interactions in real time, which can lead to lower customer satisfaction. In addition, it is difficult to provide an immediate and appropriate response to complex inquiries, so an effective response method is needed.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1309] In this invention, the server includes a speech recognition unit that receives a customer's voice and converts the voice data into text data, a natural language processing unit that analyzes the text data and classifies the inquiry, and a processing unit that determines an appropriate response based on the inquiry. This enables a system that includes a unit including smart glasses that performs real-time voice interaction with customers in a virtual store, a unit that performs the above processing on a cloud server, and a unit that generates answers using a generative AI model.

[1310] "Customer voice" is voice data uttered by customers in the virtual store.

[1311] "Speech recognition means" refers to a technical means for converting a customer's voice data into text data.

[1312] "Text data" is a character string that represents the customer's voice converted by a voice recognition means.

[1313] "Natural language processing means" is a processing technology for analyzing text data and classifying the content of inquiries.

[1314] The "processing means for determining an appropriate response" is a technical means for determining a response method based on the analyzed inquiry content.

[1315] The "means for generating an answer" refers to a technical means for generating an answer based on the content of the inquiry.

[1316] "Means for routing calls to the right person" means the technical means for routing calls to the right person for complex inquiries.

[1317] "Speech synthesis means" refers to a technical means for converting the generated response into voice data and providing it to the customer.

[1318] A "virtual store" is a virtual sales venue operated on the Internet.

[1319] "Voice interaction" refers to two-way communication between a customer and a company via voice.

[1320] "Smart glasses" are wearable devices with the ability to receive and play audio.

[1321] A "cloud server" is a server system for performing processing in a distributed computing environment.

[1322] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[1323] The system of the present invention is a voice interaction system for supporting customer support in a virtual store. This system is composed of a voice recognition unit, a natural language processing unit, a processing unit for determining an appropriate response, a response generation unit, a voice synthesis unit, smart glasses, a cloud server, and a generative AI model.

[1324] Voice reception and recognition

[1325] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[1326] Smart glasses: Receives the customer's voice and sends the voice data to the cloud.

[1327] Cloud server: Using a speech recognition tool (e.g., Google Speech-to-Text API), converts the voice data into text data. For example, the voice "Please tell me the features of this product" is converted into the text "Please tell me the features of this product."

[1328] Content analysis

[1329] Cloud server: The converted text data is passed to a natural language processing tool (e.g., OpenAI GPT-3) and its contents are analyzed. Through the analysis, the inquiry content is classified into categories such as "product information" and "usage guide." For example, the text "Please tell me the features of this product" is classified as "product information."

[1330] Determine appropriate responses and generate answers

[1331] Cloud server: Determines the appropriate response based on the analysis results and generates an answer corresponding to the inquiry using a generative AI model (e.g., GPT-3). The generated answer text might be, for example, "The features of this product are as follows..."

[1332] Providing answers

[1333] Cloud server: The generated answer is converted into voice data using a voice synthesis means (e.g., Amazon Polly). The converted voice data is sent to the smart glasses and provided to the customer.

[1334] Smart glasses: Plays back audio data and provides answers to customers, such as "The features of this product are as follows..."

[1335] Examples and Prompts

[1336] As a concrete example, if a customer asks, "What are the features of this product?", the above steps will return a spoken response saying, "The features of this product are as follows..." An example of a prompt sentence when generating an answer using a generative AI model is as follows:

[1337] "You have a customer who wants to know the features of your product. Please answer the following question: What are the features of this product?"

[1338] This system allows customers in virtual stores to obtain quick and accurate information through voice interaction, improving the user experience.

[1339] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1340] Step 1:

[1341] User: A customer asks a question to the smart glasses, for example, "What are the features of this product?"

[1342] Input: Customer voice data

[1343] Output: Input of audio data to smart glasses

[1344] Specific operation: The customer's voice is collected through the microphone in the smart glasses.

[1345] Step 2:

[1346] Smart glasses: Collected customer voice data is sent to the cloud.

[1347] Input: Customer voice data

[1348] Output: Sending audio data to the cloud

[1349] Specific operation: The smart glasses upload the audio data to the cloud server.

[1350] Step 3:

[1351] Server: Converts the voice data into text data using a voice recognition means (e.g., Google Speech-to-Text API).

[1352] Input: Audio data

[1353] Output: Text data

[1354] Specific operation: Calls the Google Speech-to-Text API and converts the voice data into text. For example, "What are the features of this product?" becomes the text "What are the features of this product?"

[1355] Step 4:

[1356] Server: The converted text data is passed to a natural language processing means (e.g., OpenAI GPT-3) and its content is analyzed.

[1357] Input: Text data

[1358] Output: Category as analysis result (e.g. "Product Information", "Usage Guide", etc.)

[1359] How it works: The GPT-3 model inputs text data, performs content analysis, and classifies the query into the appropriate category. For example, the text "What are the features of this product?" is classified as "Product Information."

[1360] Step 5:

[1361] Server: Based on the analysis results, it generates answers using a generative AI model.

[1362] Input: Parsed categories and original text data

[1363] Output: Generated answer text

[1364] How it works: Using the GPT-3 model, it generates answers based on the analyzed categories. For example, based on a prompt such as "A customer wants to know the features of a product. Please answer the following question: What are the features of this product?", it generates an answer such as "The features of this product are as follows..."

[1365] Step 6:

[1366] Server: The generated answer text is converted into voice data using a speech synthesis means (e.g., Amazon Polly) and sent to the smart glasses.

[1367] Input: Generated answer text

[1368] Output: Audio data

[1369] Specific operation: Using Amazon Polly, the answer text is converted into speech data and sent to the smart glasses. For example, the text "The features of this product are as follows..." is converted into speech data.

[1370] Step 7:

[1371] Smart glasses: Plays back audio data and provides answers to customers.

[1372] Input: Audio data

[1373] Output: Audio output to the customer

[1374] Specific operation: Play a voice through the speaker of the smart glasses and reply to the customer, for example, "The features of this product are as follows..."

[1375] Through these steps, customers can access real-time product information in the virtual store, improving the user experience.

[1376] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1377] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, enabling it to respond according to the customer's emotional state. A specific embodiment will be described below in detail from the perspectives of the server, terminal, and user.

[1378] Voice reception and recognition

[1379] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[1380] Server: Receives the customer's voice data and passes it to a voice recognition means (e.g., a voice recognition API). This voice recognition means converts the voice data into text data.

[1381] Example: If a customer says, "Please tell me how to use product A," the speech recognition means converts this speech into "Please tell me how to use product A."

[1382] emotion recognition

[1383] Server: Passes the customer's voice data to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state (e.g., anger, anxiety, satisfaction).

[1384] Example: If a customer excitedly says, "I don't know how to use product A!", the emotion engine will recognize that the customer is in an emotional state of "anxious" or "angry."

[1385] Content analysis

[1386] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, this engine classifies the inquiry into a specific category (e.g., "How to use guide" or "Troubleshooting"). The analysis results of the emotion engine are also taken into account.

[1387] Example: Analyze the converted text "Please tell me how to use product A" and classify it into the category "How to use guide" while taking into account the customer's emotion of "anxiety."

[1388] Determining the appropriate response

[1389] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ or routing the call to the right agent for a complex, emotional inquiry.

[1390] Example: If the call falls into the category "How to Use Guide" and the customer's emotion is "Anxiety," it is determined that a more flexible and gentle response is required, and the call is transferred to a representative.

[1391] Generate and provide answers

[1392] Server: When generating a response automatically, the chat generation AI model is used to generate the response text, which is then converted into voice data using a voice synthesis means.

[1393] Example: A text response such as "To use product A, first turn it on, then..." is generated and converted into voice data using a voice synthesis means. The response to the customer is "To use product A, first turn it on...".

[1394] Terminal: When transferring a call to a representative, the appropriate information (enquiry details, past enquiry history, emotional state, etc.) is displayed on the representative's terminal and the call with the customer continues.

[1395] Example: The information "This is a question about how to use product A. The customer is currently in a state of anxiety" is displayed on the agent's device, and the call is connected to the agent.

[1396] Termination processing

[1397] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[1398] Example: Save a record that reads, "X month, X day, 2023, inquiry content: How to use product A. Response result: Response by a staff member. Customer emotion: Anxiety."

[1399] This system allows for quick and accurate responses to customer inquiries, and by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and increase the work efficiency of operators.

[1400] The processing flow will be explained below.

[1401] Step 1:

[1402] User: A customer calls and speaks about their inquiry. For example, they might say, "Please tell me how to use product A."

[1403] Step 2:

[1404] Server: Receives voice from the customer and saves it as voice data. The server sends this voice data to the voice recognition engine.

[1405] Step 3:

[1406] Server: The speech recognition engine converts the received voice data into text data in real time and returns the converted text data to the server.

[1407] Step 4:

[1408] Server: Obtains text data and passes it to a natural language processing engine (NLP engine), which tokenizes the text data and analyzes the query.

[1409] Step 5:

[1410] Server: Passes the voice data from the customer to the emotion engine, which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if the customer is angry, their emotional state is recognized as "anger."

[1411] Step 6:

[1412] Server: Integrates the text analysis results from the NLP engine and the emotion recognition results from the emotion engine, and classifies the inquiry content into specific categories (e.g., "How to use guide" or "Troubleshooting").

[1413] Step 7:

[1414] Server: Determines the appropriate response based on the category of the inquiry and the customer's emotional state, for example automatically generating an answer for a simple FAQ or routing the call to the right agent for a complex, emotional inquiry.

[1415] Step 8:

[1416] Server: When generating answers automatically, the server uses a chat generation AI model to generate answer text. For example, it generates text like, "To use product A, first turn it on, then..."

[1417] Step 9:

[1418] Server: The generated text response is passed to a speech synthesis engine and converted into voice data. The speech synthesis engine converts the text into voice data and returns the voice data to the server.

[1419] Step 10:

[1420] Server: Provides voice data to the customer and responds over the phone. For example, it responds to the customer by voice, saying, "To use product A, first turn it on..."

[1421] Step 11:

[1422] Server: When transferring a call to a representative, appropriate information (such as the inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device so that the call with the customer can continue.

[1423] Step 12:

[1424] Terminal: The agent interacts with the customer based on the information displayed on the terminal. For example, the terminal may display, "This is a question about how to use product A. The customer is currently in a state of anxiety."

[1425] Step 13:

[1426] Server: After the call ends, record the inquiry details and the response results in the database. For example, record "Xth month, 2023, inquiry details: How to use product A, response results: Response by the person in charge, customer emotion: Anxiety."

[1427] Step 14:

[1428] Terminals: This information can also be viewed on the terminals of customer service representatives as needed, allowing for a quicker response the next time an inquiry is made.

[1429] Example 2

[1430] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] Conventional customer support systems simply convert customer voice into text, but are unable to take into account the customer's emotional state. This makes it difficult to provide appropriate responses, particularly to emotional inquiries, resulting in lower customer satisfaction. There is a need for a system that can solve this problem and provide quick and appropriate answers.

[1432] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means that receives the customer's voice and converts the voice data into text data, an emotion recognition means that recognizes the emotional state based on the tone, speed, and pitch of the voice, and a natural language processing means that analyzes the text data and classifies the inquiry content. This enables a quick and appropriate response taking into account the emotional state of the customer.

[1433] The "voice recognition means" is a means for receiving voice data from a customer and converting it into text data.

[1434] An "emotion recognition means" is a means for analyzing the tone, speed, and pitch of a voice to recognize the emotional state of a customer.

[1435] "Natural language processing means" refers to means for analyzing text data and classifying its contents.

[1436] The "processing means" is a means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer.

[1437] The "means for generating an answer" is a means for automatically generating an answer based on the content of the inquiry.

[1438] A "means for transferring a call" is a means for transferring a call to an appropriate person.

[1439] The "voice synthesis means" is a means for converting the generated response into voice data and providing it to the customer.

[1440] The system of the present invention receives customer voice in real time, converts the voice into text data, analyzes it, and responds appropriately. Furthermore, the system incorporates an emotion engine that recognizes the user's emotions, making it possible to respond according to the customer's emotional state. Specific embodiments are described in detail below.

[1441] Voice reception and recognition

[1442] When a user calls and speaks their inquiry, the server receives the customer's voice data. For example, if a customer says, "Please tell me how to use product A," services such as Twilio and VoIP are used to receive this voice data in real time. The received voice data is passed to the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[1443] emotion recognition

[1444] The server passes the voice data to an emotion recognition means (e.g., IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state. For example, if a customer says excitedly, "I don't know how to use product A!", the emotion engine will recognize the emotional state as "anxiety" or "anger."

[1445] Content analysis

[1446] The server passes the converted text data to a natural language processing means (e.g., a custom engine using the BERT model) and analyzes its contents. Based on the analysis results, this means classifies the inquiry content into a specific category (e.g., "usage guide" or "troubleshooting"). Furthermore, the analysis results of the emotion engine are also taken into consideration. For example, the text "Please tell me how to use product A" is analyzed and classified as a "usage guide," while also taking into account the customer's emotion of "anxiety."

[1447] Determining the appropriate response

[1448] The server determines the appropriate response based on the inquiry category and the customer's emotional state. For example, it automatically generates an answer for simple FAQs, but transfers the call to the appropriate agent for complex, emotional inquiries. For example, if the inquiry is classified as a "How to Use Guide" and the customer's emotion is "Anxiety," the call will be transferred to an agent.

[1449] Generate and provide answers

[1450] When the server automatically generates an answer, it uses a generative AI model (e.g., OpenAI's GPT-3) to generate the answer text. It then converts it into audio data using a speech synthesis API (e.g., Google Cloud Text-to-Speech) and provides it to the customer. For example, it generates text such as "To use product A, first turn it on...", converts it into audio data, and responds to the customer.

[1451] On the other hand, when transferring a call to a representative, appropriate information (e.g., inquiry content, past inquiry history, emotional state, etc.) is displayed on the representative's device, and the call with the customer continues. For example, information such as "This is an inquiry about how to use product A. The customer is currently in a state of anxiety" is displayed on the representative's device, and the call is connected to the representative.

[1452] Termination processing

[1453] After the call ends, the server records the inquiry and the response result in a database. This allows subsequent inquiries to be handled promptly. For example, a record such as "X month X day, 2023, inquiry: How to use product A, response result: Response by the person in charge, customer's emotion: Anxiety" is saved.

[1454] Prompt Sentence Examples

[1455] By using the following prompt statements, you can execute each processing step of the system in detail.

[1456] Convert the customer's speech into text. For example, if they say "How do I use product A?", return that text.

[1457] Analyze customer sentiment from this audio data to determine whether they are in a state of anxiety.

[1458] Pass this text data to a natural language processing engine and classify the inquiry content into the "How to Use Guide" category.

[1459] This system is an effective means of responding to customer inquiries quickly and accurately, and by taking into account the emotional state of the customer, it can provide an individualized response, which is expected to result in improved customer satisfaction and improved operator efficiency.

[1460] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1461] Step 1:

[1462] A user makes a call and speaks their inquiry. For example, the user might say, "Please tell me how to use product A." This voice data is sent to the server.

[1463] Input: User's voice data

[1464] Output: Raw audio data sent to the server

[1465] Step 2:

[1466] The server stores the voice data received via Twilio, etc. This stored voice data is then sent to the Google Cloud Speech-to-Text API to be converted into text data.

[1467] Input: Audio data received by the server

[1468] Output: Text data generated by the Google Cloud Speech-to-Text API (e.g., "How do I use Product A?")

[1469] Step 3:

[1470] The server sends the converted text data and raw voice data to an emotion recognition tool (an API such as IBM Watson Tone Analyzer), which analyzes the tone, speed, and pitch of the voice to recognize the customer's emotional state.

[1471] Input: Text data and raw audio data

[1472] Output: Emotional state (e.g., "anxious" or "angry")

[1473] Step 4:

[1474] The server sends the text data to a natural language processing engine (such as a custom engine using the BERT model) to analyze the query, which then categorizes the query into categories such as "How to use guide" or "Troubleshooting."

[1475] Input: Text data

[1476] Output: Category label (e.g. "How to use guide")

[1477] Step 5:

[1478] The server determines the appropriate response based on the category of the inquiry and the customer's emotional state: simple inquiries are answered by an automated system, while complex or emotional inquiries are transferred to a human agent.

[1479] Input: category label, emotional state

[1480] Output: Decision on appropriate response (e.g., generating an auto-attendant or deciding on call forwarding)

[1481] Step 6:

[1482] If an automated response is selected, the server generates a text response using a generative AI model (such as OpenAI's GPT-3), then sends this text to a speech synthesis API (such as Google Cloud Text-to-Speech) to convert it into audio data.

[1483] Input: Inquiry content based on category labels, prompt sentences to input to the generative AI model

[1484] Output: Generated answer text and audio data (e.g., "To use product A, first turn it on...")

[1485] Step 7:

[1486] The server provides the generated voice data to the user. On the other hand, when transferring a call, the server displays appropriate information (such as the inquiry content, past inquiry history, and emotional state) on the agent's terminal and connects the call to the agent.

[1487] Input: Generated voice data or information to be conveyed to the agent

[1488] Output: A spoken response to the user or information displayed on the agent's terminal (e.g., "This is a question about how to use product A. The customer is currently in a state of anxiety.")

[1489] Step 8:

[1490] After the call ends, the server records the inquiry and the response results in a database, allowing future inquiries to be handled promptly and appropriately.

[1491] Input: Call termination information, inquiry details, response results, customer emotional state

[1492] Output: Inquiry and response history stored in the database (e.g., "X month, X day, 2023, inquiry content: How to use product A, response result: Response by a staff member, customer emotion: Anxiety")

[1493] By implementing the above processing steps, the system of the present invention can respond to customer inquiries quickly and accurately. Furthermore, by taking into account the emotional state of the customer, it is possible to provide an appropriate response to each individual customer, which is expected to improve customer satisfaction and operator efficiency.

[1494] (Application example 2)

[1495] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] Conventional speech recognition systems have difficulty accurately recognizing and responding to a customer's emotional state, and lack a means to intuitively and visually guide the appropriate response. As a result, it has been difficult to improve customer satisfaction, and the burden on operators has increased. This invention aims to solve these problems by simultaneously converting customer speech into text and recognizing emotions, and visually guiding the appropriate response.

[1497] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1498] In this invention, the server includes a speech recognition means for receiving a customer's voice and converting the speech data into text data, a natural language processing means for analyzing the text data and classifying the inquiry content, an emotion recognition means for analyzing the text data and recognizing the emotional state of the customer, a processing means for determining an appropriate response based on the inquiry content and the emotional state of the customer, and a smart device display means for visually guiding the appropriate response. This enables a quick and appropriate response that takes the customer's emotions into consideration, thereby improving customer satisfaction.

[1499] The "voice recognition means" is a device that receives the customer's voice and converts the voice data into text data.

[1500] The "natural language processing means" is a device that analyzes the text data and classifies the inquiry content.

[1501] The "emotion recognition means" is a device that analyzes the text data and recognizes the emotional state of the customer.

[1502] The "processing means" is a device that determines an appropriate response based on the content of the inquiry and the emotional state of the customer.

[1503] The "voice synthesis means" is a device that converts the generated response into voice data and provides it to the customer.

[1504] The "smart device display means" is a device with a display function for visually guiding the appropriate response.

[1505] MODE FOR CARRYING OUT THE INVENTION

[1506] Voice reception and recognition

[1507] User: A customer speaks their inquiry at the store. For example, they say, "Do you have this product in stock?"

[1508] Server: Receives the customer's voice data through the microphone attached to the smart glasses and passes it to a voice recognition means (for example, Google voice recognition API). This voice recognition means converts the voice data into text data.

[1509] emotion recognition

[1510] Server: Passes the text data to an emotion recognition means (for example, the Distilroberta emotion analysis model), analyzes the tone, speed, and pitch of the voice, and recognizes the customer's emotional state (for example, joy, anger, anxiety).

[1511] Content analysis

[1512] Server: The converted text data is passed to a natural language processing engine (NLP engine) for analysis. Based on the analysis results, the inquiry is classified into a specific category (e.g., "inventory check" or "product specifications"). Emotion recognition results are also taken into consideration.

[1513] Determining the appropriate response

[1514] Server: Determines the appropriate response based on the category of inquiry and the customer's emotional state. For example, if a customer is anxious about checking inventory, the server will guide the salesperson to respond quickly and courteously.

[1515] Providing a visual guide

[1516] Device: The smart glasses display will display appropriate guidance, such as a message like "Checking inventory. Please wait a moment."

[1517] Providing answers

[1518] Server: If an automatic reply is possible, a generative AI model is used to generate a reply text, which is then converted into voice data using a voice synthesis means and provided.

[1519] Termination processing

[1520] Server: After the call ends, the inquiry and response results are recorded in a database. This allows subsequent inquiries to be handled promptly.

[1521] Hardware and software used

[1522] Hardware used: Smart glasses, server

[1523] Software used: Google speech recognition API, NLP engine, distilroberta sentiment analysis model, generative AI model

[1524] Adding specific examples

[1525] Examples:

[1526] A store associate wears smart glasses and a customer asks, "Do you have this item in stock?"

[1527] Customer speech is transcribed in real time: "Do you have this item in stock?"

[1528] The sentiment analysis engine detects "interests."

[1529] The smart glasses display a guide that reads, "Please check stock and let us know immediately."

[1530] Example prompt for a generative AI model:

[1531] If a customer excitedly asks, "Do you have this item in stock?", what advice should the sales associate give them?

[1532] In this way, the present invention provides a system that can analyze customer voices in real time, recognize their emotional state, and provide appropriate responses. This system is expected to improve customer satisfaction and streamline store operations.

[1533] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1534] Step 1:

[1535] The user makes a voice inquiry through a microphone in the smart glasses worn by the store clerk, and the input is captured as voice data.

[1536] Step 2:

[1537] The server receives voice data from the smart glasses. The input is voice data, which is passed to a voice recognition means (Google voice recognition API) and converted into text data. The output is text data.

[1538] Step 3:

[1539] The server passes the converted text data to an emotion recognition means (distilroberta emotion analysis model). The input is text data, and the data is processed to analyze the emotional state. The output is the customer's emotional state data (e.g., joy, anger, anxiety).

[1540] Step 4:

[1541] The server passes the text data to a natural language processing engine (NLP engine), which analyzes the content and classifies the inquiry into specific categories. The input is text data, and content analysis and category classification are performed as data operations. The output is category data (for example, "inventory check" or "product specifications").

[1542] Step 5:

[1543] The server uses a processing means to determine an appropriate response based on the category data of the inquiry content and the emotional state data of the customer. The input is the category data and the emotional state data, and the response is determined as a data calculation. The output is an appropriate response guideline.

[1544] Step 6:

[1545] The terminal (smart glasses) visually displays the determined correspondence guidelines. The input is the correspondence guidelines, which are converted into a display format as data processing. The output is a guide message displayed on the smart glasses display.

[1546] Step 7:

[1547] The server automatically generates answer text using a generative AI model as needed, and converts it into voice data using a voice synthesis means. The input is answer text data, which is converted into voice data as data processing. The output is voice data.

[1548] Step 8:

[1549] The user responds appropriately to the customer by following the guide message on the display of the smart glasses. The input is the guide message displayed on the display of the smart glasses, and the output is the actual customer response action.

[1550] Step 9:

[1551] After the response is completed, the server records the inquiry details and the response results in a database. The input is the response result data, which is converted into a recording format as data processing. The output is the record data saved in the database.

[1552] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1554] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1555] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1556] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1557] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1558] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1559] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1560] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1561] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1562] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1563] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1564] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1565] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1566] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1567] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1568] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1569] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1570] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1571] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1572] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1573] The following is further disclosed regarding the above embodiment.

[1574] (Claim 1)

[1575] a speech recognition means for receiving a customer's voice and converting the voice data into text data;

[1576] natural language processing means for analyzing the text data and classifying the inquiry content;

[1577] a processing means for determining an appropriate response based on the content of the inquiry;

[1578] means for generating a response or transferring the call to an appropriate person in accordance with the appropriate response;

[1579] The system includes a voice synthesis means for converting the response into voice data and providing it to the customer.

[1580] (Claim 2)

[1581] 2. The system according to claim 1, wherein the speech recognition means converts the customer's speech into text data in real time.

[1582] (Claim 3)

[1583] The system of claim 1 , wherein the processing means for determining an appropriate response uses a rule-based algorithm or a machine learning model.

[1584] "Example 1"

[1585] (Claim 1)

[1586] a speech recognition means for receiving a customer's voice and converting the voice data into text data;

[1587] natural language processing means for analyzing the text data and classifying the inquiry content;

[1588] a processing means for determining an appropriate response based on the content of the inquiry;

[1589] means for generating a response or transferring the call to an appropriate person in accordance with the appropriate response;

[1590] a voice synthesis means for converting the answer into voice data and providing it to the customer;

[1591] The system includes a recording means for recording the inquiry and the response results after the call is completed.

[1592] (Claim 2)

[1593] 2. The system according to claim 1, wherein the speech recognition means converts the customer's speech into text data in real time.

[1594] (Claim 3)

[1595] The system of claim 1 , wherein the processing means for determining an appropriate response uses a rule-based algorithm or a machine learning model.

[1596] "Application Example 1"

[1597] (Claim 1)

[1598] a speech recognition means for receiving a customer's voice and converting the voice data into text data;

[1599] natural language processing means for analyzing the text data and classifying the inquiry content;

[1600] a processing means for determining an appropriate response based on the content of the inquiry;

[1601] means for generating a response or transferring the call to an appropriate person in accordance with the appropriate response;

[1602] a voice synthesis means for converting the answer into voice data and providing it to the customer;

[1603] a means including smart glasses for real-time voice interaction with customers of the virtual store;

[1604] A means for performing the above processing on a cloud server;

[1605] A system including a means for generating an answer using a generative AI model.

[1606] (Claim 2)

[1607] 2. The system according to claim 1, wherein the speech recognition means converts the customer's speech into text data in real time.

[1608] (Claim 3)

[1609] The system of claim 1 , wherein the processing means for determining an appropriate response uses a rule-based algorithm or a machine learning model.

[1610] "Example 2: Combining Emotion Engines"

[1611] (Claim 1)

[1612] a speech recognition means for receiving a customer's voice and converting the voice data into text data;

[1613] emotion recognition means for analyzing the voice data and recognizing an emotional state based on the tone, rate, and pitch of the voice;

[1614] natural language processing means for analyzing the text data and classifying the inquiry content;

[1615] a processing means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer;

[1616] means for generating a response in accordance with the appropriate response or for transferring the call to an appropriate person;

[1617] The system includes a voice synthesis means for converting the response into voice data and providing it to the customer.

[1618] (Claim 2)

[1619] 2. The system according to claim 1, wherein the speech recognition means converts the customer's speech into text data in real time.

[1620] (Claim 3)

[1621] 10. The system of claim 1, wherein the processing means for determining an appropriate response uses a generative AI model.

[1622] "Application example 2 when combining emotion engines"

[1623] (Claim 1)

[1624] a speech recognition means for receiving a customer's voice and converting the voice data into text data;

[1625] natural language processing means for analyzing the text data and classifying the inquiry content;

[1626] emotion recognition means for analyzing the text data to recognize the emotional state of the customer;

[1627] a processing means for determining an appropriate response based on the content of the inquiry and the emotional state of the customer;

[1628] means for generating a response or transferring the call to an appropriate person in accordance with the appropriate response;

[1629] a voice synthesis means for converting the answer into voice data and providing it to the customer;

[1630] and a smart device display means for visually guiding the appropriate response.

[1631] (Claim 2)

[1632] 2. The system according to claim 1, wherein the speech recognition means converts the customer's speech into text data in real time.

[1633] (Claim 3)

[1634] The system of claim 1 , wherein the processing means for determining an appropriate response uses a rule-based algorithm or a machine learning model. [Explanation of symbols]

[1635] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a speech recognition means for receiving a customer's voice and converting the voice data into text data; natural language processing means for analyzing the text data and classifying the inquiry content; a processing means for determining an appropriate response based on the content of the inquiry; means for generating a response or transferring the call to an appropriate person in accordance with the appropriate response; The system includes a voice synthesis means for converting the response into voice data and providing it to the customer.

2. 2. The system according to claim 1, wherein said speech recognition means converts the customer's speech into text data in real time.

3. The system of claim 1 , wherein the processing means for determining an appropriate response uses a rule-based algorithm or a machine learning model.

Images