system

Abstract

A system is provided.SOLUTION: The system includes a means for converting voice data into text data, a means for extracting an intention or a keyword from the text data by using a natural language processing model, a means for searching for related FAQ data by using a similarity search algorithm, a means for generating an answer in a natural conversation form by using a generation AI, a means for collecting feedback of an operator and using the feedback for improving a system, a means for notifying a change in the FAQ database in real time, and a means for providing an answer to a customer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern customer support, it is important to respond to customer inquiries quickly and accurately. However, extracting relevant information from a massive FAQ database is not easy. This increases the burden on operators and can lead to reduced customer satisfaction. Furthermore, if changes to FAQ information are not notified in real time, there is no guarantee that the information provided to customers is up to date. The purpose of this invention is to solve these problems and provide a system that utilizes automatic speech recognition, natural language processing, and generative AI to provide effective FAQ search and response. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes the following means: a means for converting voice data into text data, a means for extracting intent and keywords from the text data using a natural language processing model, a means for searching related FAQ data using a similarity search algorithm, a means for generating answers in a natural dialogue format using generative AI, a means for collecting operator feedback and using it to improve the system, a means for notifying operators of changes to the FAQ database in real time, and a means for providing answers to customers. This system enables quick and accurate responses to customer inquiries, reduces the burden on operators, and improves customer satisfaction.

[0006] "Voice data" means data that is a recording or digital representation of a customer's voice.

[0007] "Text data" is data in the form of a character string that has been converted from voice data using an automatic voice recognition model.

[0008] A "natural language processing model" is a type of artificial intelligence technology used to extract customer intent and keywords from text data.

[0009] "Intent" refers to what the customer wants to know or what information they are seeking through their inquiry.

[0010] "Keywords" are important words or phrases in inquiries that are used to search the FAQ.

[0011] The "similarity search algorithm" is an algorithm that compares the degree of similarity between extracted keywords and phrases and entries in the FAQ database to identify highly relevant FAQs.

[0012] "FAQ data" are database entries containing frequently asked questions and their answers.

[0013] "Generative AI" is a type of artificial intelligence technology that generates natural, conversational responses based on input data.

[0014] "Feedback" refers to the evaluation or opinion provided by the operator about the system or the quality of the answers.

[0015] "Real-time notification" is a system that immediately notifies operators and related systems when there are changes to the FAQ database. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search related FAQs in response to customer inquiries and provide quick and accurate answers. This system works as follows.

[0038] The device receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0039] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0040] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0041] The server uses the extracted keywords to search the FAQ database, and uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, the FAQ about "how to record TV" is identified.

[0042] The identified FAQs are passed to a generative AI, which then generates a natural conversational response, such as "Press the record button and select the program you want to record."

[0043] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0044] The server also collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithm. Additionally, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. This ensures that the system always provides the latest information.

[0045] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0049] Step 2:

[0050] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0051] Step 3:

[0052] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0053] Step 4:

[0054] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0055] Step 5:

[0056] The server uses the extracted keywords to search the FAQ database, and then uses a similarity search algorithm to identify FAQs that closely match the keywords.

[0057] Step 6:

[0058] The server passes the identified FAQ to the generative AI, which then activates it. The generative AI then generates a natural dialogue-style answer based on the FAQ. For example, it generates an answer like, "Press the record button and select the program you want to record."

[0059] Step 7:

[0060] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0061] Step 8:

[0062] The terminal provides the generated answer to the customer, allowing them to get fast and accurate information.

[0063] Step 9:

[0064] The server collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithms, which are then used to retrain and tune the model.

[0065] Step 10:

[0066] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0067] Example 1

[0068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0069] In conventional customer support systems, it takes a lot of time and effort to manually search for information and generate answers each time a customer makes an inquiry, making it difficult to provide quick and accurate answers. This also increases the burden on operators, leading to problems such as reduced customer satisfaction. Therefore, the present invention aims to utilize automatic speech recognition and generative artificial intelligence to respond to customer inquiries quickly and accurately, thereby reducing the burden on operators and improving customer satisfaction.

[0070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0071] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative artificial intelligence, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for sending answers to a terminal and providing them to the customer by display or voice. This makes it possible to provide answers to customer inquiries quickly and accurately, reducing the burden on operators and improving customer satisfaction.

[0072] "User" refers to any individual or entity that queries the system.

[0073] A "terminal" is a device used by a user, and refers to an apparatus that includes hardware such as a microphone, speaker, and display.

[0074] "Voice data" refers to data that is a digital recording of a user's speech.

[0075] "Text data" refers to data in which voice data is expressed as text information.

[0076] A "natural language processing model" refers to a mathematical or machine learning model for analyzing and extracting meaning, intent, important keywords, etc. from text data.

[0077] A "similarity search algorithm" refers to an algorithm for searching for highly relevant data for a given query or keyword.

[0078] An "FAQ database" is a database that organizes and stores frequently asked questions and their answers.

[0079] "Generative AI" refers to artificial intelligence technology that generates new text or dialogue based on input information.

[0080] An "operator" refers to a person who manages and monitors the system and handles customer inquiries.

[0081] "Feedback" refers to evaluation information provided by operators or users regarding the performance of the system and the quality of the answers.

[0082] An "answer" is a response provided by the system to a user's inquiry.

[0083] "Real-time notification" refers to the system's ability to instantly notify relevant personnel of any changes or updates to the FAQ database.

[0084] The present invention is a system that utilizes automatic speech recognition and generative artificial intelligence to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system operates as follows.

[0085] A user makes a voice inquiry via a device. For example, the user might say, "Please tell me how to record this TV show." The device uses a built-in microphone to record the user's voice and sends the voice data to a server. The server converts the received voice data into text data using an automatic speech recognition (ASR) model (e.g., a voice recognition service). For example, the voice data "Please tell me how to record this TV show" is converted into text data "Please tell me how to record this TV show."

[0086] The server then uses a natural language processing (NLP) model (e.g., a natural language processing service) to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword. The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm (e.g., a search engine) to identify the most relevant FAQs from a large number of FAQs. As a concrete example, FAQs about "how to record TV" are identified.

[0087] The server then passes the identified FAQ to a generative AI (e.g., a generative AI model), which generates a natural, conversational answer. For example, the answer might be, "Press the record button and select the program you want to record." Here's an example prompt:

[0088] Example prompt sentence:

[0089] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0090] The generated answer is sent back to the terminal, which then displays the answer to the user in text or reads it out loud, allowing the user to quickly and accurately obtain an answer to their inquiry.

[0091] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. Furthermore, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. In this way, the system can always provide the latest information.

[0092] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0094] Step 1:

[0095] A user makes a voice inquiry via a terminal. For example, "Please tell me how to record this TV program." The input is the user's voice, and the output is the voice data picked up by the terminal's microphone.

[0096] Step 2:

[0097] The device uses a microphone to record the user's voice. The recorded voice data is saved as a digital audio file and sent to the server. The input is the voice data picked up by the device's microphone, and the output is the digital audio file sent to the server.

[0098] Step 3:

[0099] The server passes the received voice data to an automatic speech recognition (ASR) model and converts it into text data. For example, it is converted into text data such as "Please tell me how to record this TV." The input is the voice data received by the server, and the output is the text data converted by the ASR model.

[0100] Step 4:

[0101] The server uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, "TV recording methods" is recognized as an important keyword. The input is the text data output by the ASR model, and the output is the keywords extracted by the NLP model.

[0102] Step 5:

[0103] The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm to identify highly relevant FAQs. For example, it finds FAQs about "how to record TV." The input is the keywords extracted by the NLP model, and the output is the relevant FAQ data identified by the similarity search algorithm.

[0104] Step 6:

[0105] The server passes the identified FAQ to a generative AI system, which generates a natural conversational response. For example, a response such as "Press the record button and select the program you want to record" is generated. The following is an example of a prompt used for generation:

[0106] Example prompt sentence:

[0107] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0108] The input is FAQ data identified by the search algorithm, and the output is answers generated by generative artificial intelligence.

[0109] Step 7:

[0110] The generated answer is sent from the server to the device. The input is the answer generated by the generative AI, and the output is the answer data sent to the device.

[0111] Step 8:

[0112] The device displays the received answer to the user as text or reads it out loud. Specifically, in text display mode, the text is displayed on the screen, and in audio mode, the answer is played out loud from the speaker. The input is the answer data sent from the server, and the output is the answer information received by the user.

[0113] (Application example 1)

[0114] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional customer support systems have difficulty providing accurate and prompt answers to customer questions, and are particularly inadequate when it comes to responses to voice inquiries. Furthermore, they place a heavy burden on operators, making it difficult to improve the system in real time, and the answers often lack a natural feel. These issues often lead to a decline in customer satisfaction.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0117] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, means for providing answers to customers, voice input means, and means for receiving customer questions as voice input using a smart device. This makes it possible to provide quick and accurate answers to customer questions using voice input, thereby improving the efficiency of customer support and customer satisfaction.

[0118] "Audio data" refers to data obtained by converting an audio signal into digital format.

[0119] "Text data" refers to data obtained by converting voice data into text information.

[0120] A "natural language processing model" is a type of artificial intelligence used to extract intent and keywords from text data.

[0121] A "similarity search algorithm" is a method for searching for data that is most related to a specified keyword from among a large amount of data.

[0122] "Generative AI" refers to artificial intelligence technology that generates natural, conversational sentences based on given information.

[0123] "Operator" means a person assigned to assist users of the System.

[0124] "Feedback" refers to evaluations and opinions collected to improve the system.

[0125] "Smart devices" refer to highly functional electronic devices such as smartphones and head-mounted displays.

[0126] "Audio input means" refers to a microphone or related technology for collecting audio as digital data.

[0127] An "FAQ database" is a database that compiles frequently asked questions and their answers.

[0128] "Real-time notification" refers to the function of instantly notifying you of changes or updates to information.

[0129] "Customer" refers to a consumer who uses a service or product.

[0130] "Dialogue style" refers to a style of expression that involves interaction in a natural, conversational format.

[0131] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system is composed of elements such as "voice data," "text data," "natural language processing model," "similarity search algorithm," "generative AI," "operator," "feedback," "smart device," "voice input means," "FAQ database," "real-time notification," "customer," and "interactive format."

[0132] The operation of the system will now be described in detail.

[0133] The terminal (a smart device such as a smartphone or head-mounted display) receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The terminal records the customer's voice via a microphone and sends the voice data to the server.

[0134] The server converts the voice data into text data using an automatic speech recognition (ASR) model (e.g., Google® Cloud Speech-to-Text). The converted text data has the format "Please tell me how to record this TV."

[0135] The server then uses a natural language processing (NLP) model (e.g., spaCy) to extract intent and keywords from the text data. In this case, the NLP model identifies the key keyword, "how to record TV."

[0136] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (e.g., ElasticSearch®) to identify the most relevant FAQs. For example, FAQs about "how to record TV" are identified.

[0137] The identified FAQs are then passed to a generative AI (e.g., OpenAI® GPT-4®), which generates a natural, conversational answer, such as "Press the record button and select the program you want to record."

[0138] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0139] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. When changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and relevant parties as necessary.

[0140] As a concrete example, if a customer's question is "How do I record this TV?" the system will process it as follows:

[0141] 1. The customer's voice is collected by the microphone on their smart device.

[0142] 2. The audio data is sent to the server and converted to text data using Google Cloud Speech-to-Text.

[0143] 3. The text data "How to record TV" is extracted using spaCy.

[0144] 4. Related FAQs are searched in Elasticsearch.

[0145] 5. GPT-4 generates natural conversational answers based on the identified FAQs.

[0146] 6. The generated answer is sent to the device and displayed as text or provided as audio.

[0147] An example of an input prompt is as follows:

[0148] Q: A customer is asking how to record TV. They say, "How do I record this TV?" Generate an appropriate FAQ-based answer for this question.

[0149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0150] Step 1:

[0151] The terminal (smart device) receives a user (customer) inquiry via voice. Specifically, the user says, "Please tell me how to record this TV program." This voice is collected using the terminal's built-in microphone. Voice data is generated and sent to the server.

[0152] Step 2:

[0153] The server converts the received voice data into text data using an automatic speech recognition (ASR) model (Google Cloud Speech-to-Text). Specifically, the server passes the voice data to the ASR model, which outputs the text data "Please tell me how to record this TV." This converts the voice data into text information.

[0154] Step 3:

[0155] The server passes the converted text data to a natural language processing (NLP) model (spaCy) to extract intent and keywords. Specifically, the keyword "TV recording method" is extracted from the text data. The NLP model analyzes the input text and identifies important information.

[0156] Step 4:

[0157] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (Elasticsearch). Specifically, the server passes the keyword "TV recording method" to Elasticsearch to search for related FAQs. The most relevant FAQ is identified.

[0158] Step 5:

[0159] The server passes the identified FAQs to a generative AI (OpenAI GPT-4), which generates natural dialogue-style answers. Specifically, the FAQ data is input as a prompt into GPT-4, which outputs the answer "Press the record button and select the program you want to record." The generative AI generates appropriate dialogue-style sentences based on the input data.

[0160] Step 6:

[0161] The server sends the generated answer to the terminal, which then displays the answer to the customer in text or reads it aloud. Specifically, in the case of text display, the answer is displayed on the terminal screen, and in the case of audio, the answer is read aloud from the terminal's speaker.

[0162] Step 7:

[0163] The server collects feedback from operators and uses it to improve the system. Specifically, it collects feedback data and uses it to improve the accuracy of natural language processing models and similarity search algorithms. This allows the system's performance to continuously improve.

[0164] Step 8:

[0165] The server monitors changes to the FAQ database in real time and notifies operators and other relevant parties as necessary. Specifically, a process that monitors database updates is constantly running, and if a change is detected, a notification is automatically sent. This allows the system to always maintain the latest information.

[0166] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0167] This invention is a system that combines automatic speech recognition, natural language processing, generative AI, and an emotion engine that recognizes user emotions to provide more accurate responses to customer inquiries. This system operates as follows.

[0168] The device receives customer inquiries via voice. For example, consider a case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0169] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0170] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0171] Furthermore, the server activates an emotion engine to analyze the user's emotion from the voice data. The emotion engine analyzes the user's emotional state (e.g., joy, anger, sadness, etc.) from the voice data and obtains the result. This emotion analysis result is used to generate a response later.

[0172] The server uses the extracted keywords and the sentiment analysis results to search the FAQ database. It then uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, it identifies FAQs about "how to record TV."

[0173] The identified FAQs are passed to a generative AI, which then activates the AI. Based on the FAQs and the results of sentiment analysis, the generative AI generates a natural, conversational response that matches the user's emotions. For example, if it determines that the user is confused, it generates a response that includes reassuring language, such as, "Don't worry, just press the record button and select the program you want to record."

[0174] The generated answer is then sent back to the device, which displays it as text or reads it aloud.

[0175] The device provides customers with generated answers, which provide fast, accurate information and are emotionally sensitive, further increasing customer satisfaction.

[0176] The server also collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0177] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0178] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the customer's feelings, thereby reducing the burden on operators and significantly improving customer satisfaction.

[0179] The processing flow will be explained below.

[0180] Step 1:

[0181] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0182] Step 2:

[0183] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0184] Step 3:

[0185] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0186] Step 4:

[0187] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0188] Step 5:

[0189] The server activates an emotion engine to analyze the user's emotional state from the voice data. The emotion engine recognizes emotions such as "confused" or "angry" based on the user's tone and intonation of voice.

[0190] Step 6:

[0191] The server combines the results of the emotion engine and the NLP model, searches the FAQ database based on the extracted keywords and the sentiment analysis results, and then uses a similarity search algorithm to identify FAQs that have a high matching frequency with the keywords.

[0192] Step 7:

[0193] The server passes the identified FAQ to the generative AI, which then activates it. Based on the FAQ and the results of sentiment analysis, the generative AI generates a natural, conversational response that takes the user's emotions into consideration. For example, it might generate a response like, "Press the record button and select the program you want to record. Please also use our operation guide."

[0194] Step 8:

[0195] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0196] Step 9:

[0197] The terminal provides the customer with a generated answer, allowing them to obtain quick and accurate information. In addition, the response is sensitive to the user's feelings, improving customer satisfaction.

[0198] Step 10:

[0199] The server collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0200] Step 11:

[0201] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0202] Example 2

[0203] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0204] Conventional customer support systems use voice recognition and natural language processing, but lack the mechanisms to understand user emotions and respond appropriately. This has led to problems such as reduced user satisfaction and increased operator workload. Furthermore, FAQ databases are not updated in real time, which can result in delays in providing the latest information. The purpose of this invention is to solve these problems.

[0205] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0206] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for analyzing the user's emotional state from the voice data, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format that matches the user's emotions using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for providing answers to customers. This enables accurate and prompt information provision that takes user emotions into consideration, thereby improving customer satisfaction and reducing the burden on operators.

[0207] "Voice data" refers to data in which the voice spoken by the user via the terminal is recorded in digital format.

[0208] "Text data" refers to voice data converted into text format using an automatic speech recognition (ASR) model.

[0209] A "natural language processing model" is a general term for algorithms and software that analyze text data and extract intent and keywords.

[0210] "Intention" refers to specific actions and intentions extracted by a natural language processing model from what the user has said.

[0211] "Keywords" refer to important words or phrases identified from text data by a natural language processing model.

[0212] "Automatic speech recognition model" is a general term for algorithms and software for converting voice data into text data.

[0213] "Emotion engine" is a general term for algorithms and software for analyzing a user's emotional state (e.g., joy, anger, sadness, etc.) from voice or text data.

[0214] The "similarity search algorithm" is an algorithm for identifying highly relevant data from within the FAQ database based on extracted keywords.

[0215] The "FAQ database" is a database that compiles frequently asked questions and their answers.

[0216] "Generative AI" is a general term for algorithms and software that generate natural, conversational answers based on given data (e.g., FAQs or sentiment analysis results).

[0217] "Operator feedback" refers to the operator's evaluation of the system's answers, and is the data used to retrain and tune the model.

[0218] "System improvement" refers to the process of improving the accuracy of automatic speech recognition models, natural language processing models, emotion engines, and generative AI based on collected feedback.

[0219] "Means of real-time notification" refers to a system that immediately transmits information to the server and related terminals when there are changes in the FAQ database.

[0220] This invention is a system that combines automatic speech recognition (ASR) technology, natural language processing (NLP) technology, generative AI technology, and emotion recognition technology to provide customer service that is appropriate to the user's emotions. This system is mainly composed of a server and a terminal, and operates as follows.

[0221] The device receives the user's voice inquiry. For example, consider the case where the user says, "Please tell me how to record this TV program." The device records the user's voice using a built-in microphone and sends the voice data in digital format to a server. Voice data is typically sent using an HTTP request over the Internet.

[0222] The server uses an automatic speech recognition (ASR) model on the received voice data to convert it into text data. This process can be performed using a voice recognition service such as Google Cloud Speech-to-Text API. For example, the server can generate text data such as "Please tell me how to record this TV program" from the voice data.

[0223] The server then uses a natural language processing (NLP) model to extract user intent and key keywords from the text data. This process can be performed using natural language processing libraries such as spaCy or BERT. For example, the NLP model might extract the key keyword "how to record TV" from the text data.

[0224] The server then activates an emotion engine to analyze the user's emotional state from the voice and text data. This can be done using emotion analysis tools such as IBM Watson® Tone Analyzer. For example, the server can determine from the voice data that the user is confused.

[0225] The server uses the extracted keywords and the results of sentiment analysis to search the FAQ database. It uses a similarity search algorithm to identify the most relevant FAQs from among many. This process can be performed using a search engine such as Elasticsearch. For example, the server identifies FAQs about "how to record TV."

[0226] The identified FAQs are passed to a generative AI, which uses the FAQs and sentiment analysis results to generate natural, conversational answers that fit the user's emotions. This process can use a generative AI model such as OpenAI's GPT-3 (registered trademark). For example, a generated answer might be, "Don't worry. Just press the record button and select the program you want to record."

[0227] The generated answer is then sent back to the device, which can then display it as text or read it aloud using a speech synthesis engine. For example, the device may display it to the user as a text message and read it aloud using speech synthesis.

[0228] The server also collects feedback from operators and uses it to improve the accuracy of the automatic speech recognition model, natural language processing model, emotion engine, and generative AI. This feedback is used to retrain the models and tune parameters. Furthermore, the server monitors changes to the FAQ database in real time and notifies operators and related devices if any changes are detected.

[0229] As a concrete example, the following prompt sentences can be input to a generative AI model:

[0230] Example prompt: "How do I record this TV show?"

[0231] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the user's feelings, thereby improving customer satisfaction and reducing the burden on operators.

[0232] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0233] Step 1:

[0234] The device acquires the user's voice. When the user speaks into the microphone, such as "Please tell me how to record this TV program," the device records the voice and generates audio data. The input is the user's speech, and the output is digital audio data.

[0235] Step 2:

[0236] The device sends the recorded audio data to the server. This transmission is performed via an HTTP request over the Internet. The input is audio data, and the output is the transmission of audio data to the server.

[0237] Step 3:

[0238] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For this conversion, it uses, for example, the Google Cloud Speech-to-Text API. The input is voice data, and the output is text data such as "Please tell me how to record this TV."

[0239] Step 4:

[0240] The server then passes the converted text data to a natural language processing (NLP) model to extract key intent and keywords. Examples of NLP models used for this process include spaCy and BERT. The input is text data, and the output is extracted keywords, such as "how to record TV."

[0241] Step 5:

[0242] The server passes the voice data to an emotion engine to analyze the user's emotional state. For this processing, IBM Watson Tone Analyzer is used, for example. The input is the voice data, and the output is the user's emotional state (e.g., confusion or relief).

[0243] Step 6:

[0244] The server uses the extracted keywords and the results of the sentiment analysis to search the FAQ database. This search is performed using a search engine such as Elasticsearch. The input is the keywords and the results of the sentiment analysis, and the output is a list of relevant FAQs about "how to record TV."

[0245] Step 7:

[0246] The server passes the FAQs identified by the search to a generative AI, which generates a natural, conversational answer that matches the user's emotions. This process uses a generative AI model such as OpenAI's GPT-3. The input is the FAQ and the results of sentiment analysis, and the output is a response such as, "Don't worry. Just press the record button and select the program you want to record."

[0247] Step 8:

[0248] The server generates a response and sends it to the device, which displays it as text on the screen or reads it aloud using a speech synthesis engine. The input is the response, and the output is a response that the user can see or hear.

[0249] Step 9:

[0250] The server collects feedback from operators and uses it to improve the system. Operators evaluate the quality and appropriateness of the answers provided, and the results are used to retrain and tune the model. The input is operator feedback, and the output is an improved model.

[0251] Step 10:

[0252] The server monitors changes to the FAQ database in real time and notifies operators and terminals if any changes are detected. This allows the system to always provide the latest information. The input is new or changed FAQ data, and the output is real-time notifications.

[0253] (Application example 2)

[0254] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] Conventional customer support systems are capable of voice recognition and text conversion, but they do not adequately respond to the user's emotions, resulting in low customer satisfaction. Furthermore, the generated answers do not always take the user's emotions into consideration, resulting in inappropriate responses. Furthermore, they lack a mechanism for notifying users of changes to the FAQ database in real time, making it difficult to always provide the latest information to customers.

[0256] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, and means for recognizing the user's emotional state from the voice data using an emotion engine. This makes it possible to generate appropriate answers that take the user's emotions into consideration and to respond to customers quickly and accurately. In addition, by notifying changes to the FAQ database in real time, it is possible to always provide the latest information.

[0257] "Audio data" refers to data of an audio signal provided by a user through an audio input device such as a microphone.

[0258] "Text data" is data in a format in which voice data is converted into character information.

[0259] A "natural language processing model" is an algorithm or machine learning model for analyzing semantic information and context from text data.

[0260] A "similarity search algorithm" is an algorithm for searching a database for related information based on specific keywords or phrases.

[0261] "Generative AI" is an artificial intelligence model that generates natural, conversational text based on input data.

[0262] An "emotion engine" is an algorithm or system that analyzes and recognizes a user's emotional state from voice and text data.

[0263] "Operator feedback" refers to the operator's evaluation and comments regarding the system's accuracy and response content.

[0264] The "FAQ database" is a database that collects frequently asked questions and their answers.

[0265] "Means of real-time notification" refers to a function that instantly notifies relevant parties or systems of database update information, etc.

[0266] The "means for providing an answer to the customer" is a function for presenting the generated answer to the user in text or audio format.

[0267] "User's emotional state" refers to the feelings or mood of the user when they input voice or text.

[0268] "Intent and keywords" refer to verbs and nouns that are at the core of the context and are extracted from the text data entered by the user.

[0269] A system for realizing this application example operates with the following hardware and software configuration.

[0270] Hardware and software used

[0271] Hardware: Microphone (for voice input), server, terminal

[0272] Software: SpeechRecognition library, spaCy, Transformers (Hugging Face), TextBlob

[0273] System Operation Details

[0274] 1. Voice Recognition

[0275] The user speaks their query through the microphone. The device's microphone records the voice data, and the SpeechRecognition library is used to capture the data. For example, consider a user saying, "Please tell me how to record this TV." This voice is converted into text data in real time through Google's speech recognition API.

[0276] 2. Natural Language Processing

[0277] The server analyzes the acquired text data using a natural language processing model (spaCy). Here, intent (verbs) and keywords (nouns) are extracted from the text data. For example, keywords such as "TV recording method" are extracted.

[0278] 3. Sentiment analysis

[0279] We use TextBlob to recognize the user's emotional state from the audio data, which determines whether the emotional state is positive or negative, and this information is then reflected in the generated answer.

[0280] 4. Answer generation

[0281] The server combines the extracted intent, keywords, and sentiment analysis results and passes them to a generative AI (using the Transformers library) as a prompt. The AI ​​then adjusts the format and content of the response appropriately based on the user's emotional state. For example, if the server determines that the user is confused, it generates a response such as, "Don't worry, just press the record button and select the program you want to record."

[0282] 5. Providing answers

[0283] The generated answer is then sent back to the device, which displays the answer as text or reads it aloud, allowing the user to obtain quick and accurate information.

[0284] Specific prompt examples

[0285] Below are some examples of prompts passed to the generative AI:

[0286] text

[0287] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0288] This system is expected to improve customer satisfaction by providing appropriate answers to customers through a consistent process from voice input to emotion recognition, natural language processing, and answer generation.

[0289] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0290] Step 1:

[0291] The user makes a voice inquiry. The voice data is input to the device through the microphone. The device uses the SpeechRecognition library to record the voice data. The recorded voice data is processed as input data.

[0292] Step 2:

[0293] The device converts the recorded voice data into text data through Google's speech recognition API. The input data is voice data, and the output data is text data. This allows the voice content to be expressed in text format.

[0294] Step 3:

[0295] The server receives the text data and uses a natural language processing model (spaCy) to extract intent and keywords. At this stage, the text data is the input and the intent and keywords are the output. Specifically, the text data is grammatically analyzed to identify verbs and nouns.

[0296] Step 4:

[0297] The server uses TextBlob to analyze the user's emotional state from text data. The input data is text data, and the output data is the emotional state (e.g., positive, negative, etc.). The specific operation here is to determine the emotion from the context and tone of the text.

[0298] Step 5:

[0299] The server generates a prompt based on the extracted intent, keywords, and emotional state, and inputs it into a generative AI (Transformers). The input data is the prompt, and the output data is the answer text. The specific operation is for the generative AI to generate an appropriate answer for the input prompt.

[0300] Examples of prompts:

[0301] text

[0302] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0303] Step 6:

[0304] The generated answer text is sent from the server to the terminal. The input data is the answer text, and the output data is the answer that is displayed or read aloud. The terminal provides the answer to the user, providing a highly satisfactory response.

[0305] This system automates the entire process from receiving an inquiry to providing a response, while also providing a response that takes emotions into consideration.

[0306] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0307] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0308] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0309] [Second embodiment]

[0310] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0311] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0312] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0313] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0314] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0315] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0316] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0317] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0318] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0319] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0320] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0321] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0322] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search related FAQs in response to customer inquiries and provide quick and accurate answers. This system works as follows.

[0323] The device receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0324] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0325] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0326] The server uses the extracted keywords to search the FAQ database, and uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, the FAQ about "how to record TV" is identified.

[0327] The identified FAQs are passed to a generative AI, which then generates a natural conversational response, such as "Press the record button and select the program you want to record."

[0328] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0329] The server also collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithm. Additionally, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. This ensures that the system always provides the latest information.

[0330] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0331] The processing flow will be explained below.

[0332] Step 1:

[0333] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0334] Step 2:

[0335] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0336] Step 3:

[0337] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0338] Step 4:

[0339] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0340] Step 5:

[0341] The server uses the extracted keywords to search the FAQ database, and then uses a similarity search algorithm to identify FAQs that closely match the keywords.

[0342] Step 6:

[0343] The server passes the identified FAQ to the generative AI, which then activates it. The generative AI then generates a natural dialogue-style answer based on the FAQ. For example, it generates an answer like, "Press the record button and select the program you want to record."

[0344] Step 7:

[0345] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0346] Step 8:

[0347] The terminal provides the generated answer to the customer, allowing them to get fast and accurate information.

[0348] Step 9:

[0349] The server collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithms, which are then used to retrain and tune the model.

[0350] Step 10:

[0351] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0352] Example 1

[0353] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0354] In conventional customer support systems, it takes a lot of time and effort to manually search for information and generate answers each time a customer makes an inquiry, making it difficult to provide quick and accurate answers. This also increases the burden on operators, leading to problems such as reduced customer satisfaction. Therefore, the present invention aims to utilize automatic speech recognition and generative artificial intelligence to respond to customer inquiries quickly and accurately, thereby reducing the burden on operators and improving customer satisfaction.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0356] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative artificial intelligence, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for sending answers to a terminal and providing them to the customer by display or voice. This makes it possible to provide answers to customer inquiries quickly and accurately, reducing the burden on operators and improving customer satisfaction.

[0357] "User" refers to any individual or entity that queries the system.

[0358] A "terminal" is a device used by a user, and refers to an apparatus that includes hardware such as a microphone, speaker, and display.

[0359] "Voice data" refers to data that is a digital recording of a user's speech.

[0360] "Text data" refers to data in which voice data is expressed as text information.

[0361] A "natural language processing model" refers to a mathematical or machine learning model for analyzing and extracting meaning, intent, important keywords, etc. from text data.

[0362] A "similarity search algorithm" refers to an algorithm for searching for highly relevant data for a given query or keyword.

[0363] An "FAQ database" is a database that organizes and stores frequently asked questions and their answers.

[0364] "Generative AI" refers to artificial intelligence technology that generates new text or dialogue based on input information.

[0365] An "operator" refers to a person who manages and monitors the system and handles customer inquiries.

[0366] "Feedback" refers to evaluation information provided by operators or users regarding the performance of the system and the quality of the answers.

[0367] An "answer" is a response provided by the system to a user's inquiry.

[0368] "Real-time notification" refers to the system's ability to instantly notify relevant personnel of any changes or updates to the FAQ database.

[0369] The present invention is a system that utilizes automatic speech recognition and generative artificial intelligence to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system operates as follows.

[0370] A user makes a voice inquiry via a device. For example, the user might say, "Please tell me how to record this TV show." The device uses a built-in microphone to record the user's voice and sends the voice data to a server. The server converts the received voice data into text data using an automatic speech recognition (ASR) model (e.g., a voice recognition service). For example, the voice data "Please tell me how to record this TV show" is converted into text data "Please tell me how to record this TV show."

[0371] The server then uses a natural language processing (NLP) model (e.g., a natural language processing service) to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword. The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm (e.g., a search engine) to identify the most relevant FAQs from a large number of FAQs. As a concrete example, FAQs about "how to record TV" are identified.

[0372] The server then passes the identified FAQ to a generative AI (e.g., a generative AI model), which generates a natural, conversational answer. For example, the answer might be, "Press the record button and select the program you want to record." Here's an example prompt:

[0373] Example prompt sentence:

[0374] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0375] The generated answer is sent back to the terminal, which then displays the answer to the user in text or reads it out loud, allowing the user to quickly and accurately obtain an answer to their inquiry.

[0376] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. Furthermore, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. In this way, the system can always provide the latest information.

[0377] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0378] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0379] Step 1:

[0380] A user makes a voice inquiry via a terminal. For example, "Please tell me how to record this TV program." The input is the user's voice, and the output is the voice data picked up by the terminal's microphone.

[0381] Step 2:

[0382] The device uses a microphone to record the user's voice. The recorded voice data is saved as a digital audio file and sent to the server. The input is the voice data picked up by the device's microphone, and the output is the digital audio file sent to the server.

[0383] Step 3:

[0384] The server passes the received voice data to an automatic speech recognition (ASR) model and converts it into text data. For example, it is converted into text data such as "Please tell me how to record this TV." The input is the voice data received by the server, and the output is the text data converted by the ASR model.

[0385] Step 4:

[0386] The server uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, "TV recording methods" is recognized as an important keyword. The input is the text data output by the ASR model, and the output is the keywords extracted by the NLP model.

[0387] Step 5:

[0388] The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm to identify highly relevant FAQs. For example, it finds FAQs about "how to record TV." The input is the keywords extracted by the NLP model, and the output is the relevant FAQ data identified by the similarity search algorithm.

[0389] Step 6:

[0390] The server passes the identified FAQ to a generative AI system, which generates a natural conversational response. For example, a response such as "Press the record button and select the program you want to record" is generated. The following is an example of a prompt used for generation:

[0391] Example prompt sentence:

[0392] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0393] The input is FAQ data identified by the search algorithm, and the output is answers generated by generative artificial intelligence.

[0394] Step 7:

[0395] The generated answer is sent from the server to the device. The input is the answer generated by the generative AI, and the output is the answer data sent to the device.

[0396] Step 8:

[0397] The device displays the received answer to the user as text or reads it out loud. Specifically, in text display mode, the text is displayed on the screen, and in audio mode, the answer is played out loud from the speaker. The input is the answer data sent from the server, and the output is the answer information received by the user.

[0398] (Application example 1)

[0399] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0400] Conventional customer support systems have difficulty providing accurate and prompt answers to customer questions, and are particularly inadequate when it comes to responses to voice inquiries. Furthermore, they place a heavy burden on operators, making it difficult to improve the system in real time, and the answers often lack a natural feel. These issues often lead to a decline in customer satisfaction.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0402] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, means for providing answers to customers, voice input means, and means for receiving customer questions as voice input using a smart device. This makes it possible to provide quick and accurate answers to customer questions using voice input, thereby improving the efficiency of customer support and customer satisfaction.

[0403] "Audio data" refers to data obtained by converting an audio signal into digital format.

[0404] "Text data" refers to data obtained by converting voice data into text information.

[0405] A "natural language processing model" is a type of artificial intelligence used to extract intent and keywords from text data.

[0406] A "similarity search algorithm" is a method for searching for data that is most related to a specified keyword from among a large amount of data.

[0407] "Generative AI" refers to artificial intelligence technology that generates natural, conversational sentences based on given information.

[0408] "Operator" means a person assigned to assist users of the System.

[0409] "Feedback" refers to evaluations and opinions collected to improve the system.

[0410] "Smart devices" refer to highly functional electronic devices such as smartphones and head-mounted displays.

[0411] "Audio input means" refers to a microphone or related technology for collecting audio as digital data.

[0412] An "FAQ database" is a database that compiles frequently asked questions and their answers.

[0413] "Real-time notification" refers to the function of instantly notifying you of changes or updates to information.

[0414] "Customer" refers to a consumer who uses a service or product.

[0415] "Dialogue style" refers to a style of expression that involves interaction in a natural, conversational format.

[0416] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system is composed of elements such as "voice data," "text data," "natural language processing model," "similarity search algorithm," "generative AI," "operator," "feedback," "smart device," "voice input means," "FAQ database," "real-time notification," "customer," and "interactive format."

[0417] The operation of the system will now be described in detail.

[0418] The terminal (a smart device such as a smartphone or head-mounted display) receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The terminal records the customer's voice via a microphone and sends the voice data to the server.

[0419] The server converts the voice data into text data using an automatic speech recognition (ASR) model (e.g., Google Cloud Speech-to-Text). The converted text data has the format "Please tell me how to record this TV."

[0420] The server then uses a natural language processing (NLP) model (e.g., spaCy) to extract intent and keywords from the text data. In this case, the NLP model identifies the key keyword, "how to record TV."

[0421] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (e.g., Elasticsearch) to identify the most relevant FAQs, such as those about "how to record TV."

[0422] The identified FAQs are then passed to a generative AI (e.g., OpenAI GPT-4), which generates a natural conversational answer, such as "Press the record button and select the program you want to record."

[0423] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0424] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. When changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and relevant parties as necessary.

[0425] As a concrete example, if a customer's question is "How do I record this TV?" the system will process it as follows:

[0426] 1. The customer's voice is collected by the microphone on their smart device.

[0427] 2. The audio data is sent to the server and converted to text data using Google Cloud Speech-to-Text.

[0428] 3. The text data "How to record TV" is extracted using spaCy.

[0429] 4. Related FAQs are searched in Elasticsearch.

[0430] 5. GPT-4 generates natural conversational answers based on the identified FAQs.

[0431] 6. The generated answer is sent to the device and displayed as text or provided as audio.

[0432] An example of an input prompt is as follows:

[0433] Q: A customer is asking how to record TV. They say, "How do I record this TV?" Generate an appropriate FAQ-based answer for this question.

[0434] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0435] Step 1:

[0436] The terminal (smart device) receives a user (customer) inquiry via voice. Specifically, the user says, "Please tell me how to record this TV program." This voice is collected using the terminal's built-in microphone. Voice data is generated and sent to the server.

[0437] Step 2:

[0438] The server converts the received voice data into text data using an automatic speech recognition (ASR) model (Google Cloud Speech-to-Text). Specifically, the server passes the voice data to the ASR model, which outputs the text data "Please tell me how to record this TV." This converts the voice data into text information.

[0439] Step 3:

[0440] The server passes the converted text data to a natural language processing (NLP) model (spaCy) to extract intent and keywords. Specifically, the keyword "TV recording method" is extracted from the text data. The NLP model analyzes the input text and identifies important information.

[0441] Step 4:

[0442] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (Elasticsearch). Specifically, the server passes the keyword "TV recording method" to Elasticsearch to search for related FAQs. The most relevant FAQ is identified.

[0443] Step 5:

[0444] The server passes the identified FAQs to a generative AI (OpenAI GPT-4), which generates natural dialogue-style answers. Specifically, the FAQ data is input as a prompt into GPT-4, which outputs the answer "Press the record button and select the program you want to record." The generative AI generates appropriate dialogue-style sentences based on the input data.

[0445] Step 6:

[0446] The server sends the generated answer to the terminal, which then displays the answer to the customer in text or reads it aloud. Specifically, in the case of text display, the answer is displayed on the terminal screen, and in the case of audio, the answer is read aloud from the terminal's speaker.

[0447] Step 7:

[0448] The server collects feedback from operators and uses it to improve the system. Specifically, it collects feedback data and uses it to improve the accuracy of natural language processing models and similarity search algorithms. This allows the system's performance to continuously improve.

[0449] Step 8:

[0450] The server monitors changes to the FAQ database in real time and notifies operators and other relevant parties as necessary. Specifically, a process that monitors database updates is constantly running, and if a change is detected, a notification is automatically sent. This allows the system to always maintain the latest information.

[0451] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0452] This invention is a system that combines automatic speech recognition, natural language processing, generative AI, and an emotion engine that recognizes user emotions to provide more accurate responses to customer inquiries. This system operates as follows.

[0453] The device receives customer inquiries via voice. For example, consider a case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0454] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0455] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0456] Furthermore, the server activates an emotion engine to analyze the user's emotion from the voice data. The emotion engine analyzes the user's emotional state (e.g., joy, anger, sadness, etc.) from the voice data and obtains the result. This emotion analysis result is used to generate a response later.

[0457] The server uses the extracted keywords and the sentiment analysis results to search the FAQ database. It then uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, it identifies FAQs about "how to record TV."

[0458] The identified FAQs are passed to a generative AI, which then activates the AI. Based on the FAQs and the results of sentiment analysis, the generative AI generates a natural, conversational response that matches the user's emotions. For example, if it determines that the user is confused, it generates a response that includes reassuring language, such as, "Don't worry, just press the record button and select the program you want to record."

[0459] The generated answer is then sent back to the device, which displays it as text or reads it aloud.

[0460] The device provides customers with generated answers, which provide fast, accurate information and are emotionally sensitive, further increasing customer satisfaction.

[0461] The server also collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0462] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0463] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the customer's feelings, thereby reducing the burden on operators and significantly improving customer satisfaction.

[0464] The processing flow will be explained below.

[0465] Step 1:

[0466] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0467] Step 2:

[0468] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0469] Step 3:

[0470] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0471] Step 4:

[0472] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0473] Step 5:

[0474] The server activates an emotion engine to analyze the user's emotional state from the voice data. The emotion engine recognizes emotions such as "confused" or "angry" based on the user's tone and intonation of voice.

[0475] Step 6:

[0476] The server combines the results of the emotion engine and the NLP model, searches the FAQ database based on the extracted keywords and the sentiment analysis results, and then uses a similarity search algorithm to identify FAQs that have a high matching frequency with the keywords.

[0477] Step 7:

[0478] The server passes the identified FAQ to the generative AI, which then activates it. Based on the FAQ and the results of sentiment analysis, the generative AI generates a natural, conversational response that takes the user's emotions into consideration. For example, it might generate a response like, "Press the record button and select the program you want to record. Please also use our operation guide."

[0479] Step 8:

[0480] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0481] Step 9:

[0482] The terminal provides the customer with a generated answer, allowing them to obtain quick and accurate information. In addition, the response is sensitive to the user's feelings, improving customer satisfaction.

[0483] Step 10:

[0484] The server collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0485] Step 11:

[0486] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0487] Example 2

[0488] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] Conventional customer support systems use voice recognition and natural language processing, but lack the mechanisms to understand user emotions and respond appropriately. This has led to problems such as reduced user satisfaction and increased operator workload. Furthermore, FAQ databases are not updated in real time, which can result in delays in providing the latest information. The purpose of this invention is to solve these problems.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0491] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for analyzing the user's emotional state from the voice data, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format that matches the user's emotions using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for providing answers to customers. This enables accurate and prompt information provision that takes user emotions into consideration, thereby improving customer satisfaction and reducing the burden on operators.

[0492] "Voice data" refers to data in which the voice spoken by the user via the terminal is recorded in digital format.

[0493] "Text data" refers to voice data converted into text format using an automatic speech recognition (ASR) model.

[0494] A "natural language processing model" is a general term for algorithms and software that analyze text data and extract intent and keywords.

[0495] "Intention" refers to specific actions and intentions extracted by a natural language processing model from what the user has said.

[0496] "Keywords" refer to important words or phrases identified from text data by a natural language processing model.

[0497] "Automatic speech recognition model" is a general term for algorithms and software for converting voice data into text data.

[0498] "Emotion engine" is a general term for algorithms and software for analyzing a user's emotional state (e.g., joy, anger, sadness, etc.) from voice or text data.

[0499] The "similarity search algorithm" is an algorithm for identifying highly relevant data from within the FAQ database based on extracted keywords.

[0500] The "FAQ database" is a database that compiles frequently asked questions and their answers.

[0501] "Generative AI" is a general term for algorithms and software that generate natural, conversational answers based on given data (e.g., FAQs or sentiment analysis results).

[0502] "Operator feedback" refers to the operator's evaluation of the system's answers, and is the data used to retrain and tune the model.

[0503] "System improvement" refers to the process of improving the accuracy of automatic speech recognition models, natural language processing models, emotion engines, and generative AI based on collected feedback.

[0504] "Means of real-time notification" refers to a system that immediately transmits information to the server and related terminals when there are changes in the FAQ database.

[0505] This invention is a system that combines automatic speech recognition (ASR) technology, natural language processing (NLP) technology, generative AI technology, and emotion recognition technology to provide customer service that is appropriate to the user's emotions. This system is mainly composed of a server and a terminal, and operates as follows.

[0506] The device receives the user's voice inquiry. For example, consider the case where the user says, "Please tell me how to record this TV program." The device records the user's voice using a built-in microphone and sends the voice data in digital format to a server. Voice data is typically sent using an HTTP request over the Internet.

[0507] The server uses an automatic speech recognition (ASR) model on the received voice data to convert it into text data. This process can be performed using a voice recognition service such as Google Cloud Speech-to-Text API. For example, the server can generate text data such as "Please tell me how to record this TV program" from the voice data.

[0508] The server then uses a natural language processing (NLP) model to extract user intent and key keywords from the text data. This process can be performed using natural language processing libraries such as spaCy or BERT. For example, the NLP model might extract the key keyword "how to record TV" from the text data.

[0509] The server then activates an emotion engine to analyze the user's emotional state from the voice and text data. This can be done using emotion analysis tools such as IBM Watson Tone Analyzer. For example, the server can determine from the voice data that the user is confused.

[0510] The server uses the extracted keywords and the results of sentiment analysis to search the FAQ database. It uses a similarity search algorithm to identify the most relevant FAQs from among many. This process can be performed using a search engine such as Elasticsearch. For example, the server identifies FAQs about "how to record TV."

[0511] The identified FAQs are passed to a generative AI, which uses the FAQs and sentiment analysis results to generate natural, conversational answers that fit the user's emotions. This process can use a generative AI model such as OpenAI's GPT-3. For example, a generated answer might be, "Don't worry. Just press the record button and select the program you want to record."

[0512] The generated answer is then sent back to the device, which can then display it as text or read it aloud using a speech synthesis engine. For example, the device may display it to the user as a text message and read it aloud using speech synthesis.

[0513] The server also collects feedback from operators and uses it to improve the accuracy of the automatic speech recognition model, natural language processing model, emotion engine, and generative AI. This feedback is used to retrain the models and tune parameters. Furthermore, the server monitors changes to the FAQ database in real time and notifies operators and related devices if any changes are detected.

[0514] As a concrete example, the following prompt sentences can be input to a generative AI model:

[0515] Example prompt: "How do I record this TV show?"

[0516] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the user's feelings, thereby improving customer satisfaction and reducing the burden on operators.

[0517] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0518] Step 1:

[0519] The device acquires the user's voice. When the user speaks into the microphone, such as "Please tell me how to record this TV program," the device records the voice and generates audio data. The input is the user's speech, and the output is digital audio data.

[0520] Step 2:

[0521] The device sends the recorded audio data to the server. This transmission is performed via an HTTP request over the Internet. The input is audio data, and the output is the transmission of audio data to the server.

[0522] Step 3:

[0523] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For this conversion, it uses, for example, the Google Cloud Speech-to-Text API. The input is voice data, and the output is text data such as "Please tell me how to record this TV."

[0524] Step 4:

[0525] The server then passes the converted text data to a natural language processing (NLP) model to extract key intent and keywords. Examples of NLP models used for this process include spaCy and BERT. The input is text data, and the output is extracted keywords, such as "how to record TV."

[0526] Step 5:

[0527] The server passes the voice data to an emotion engine to analyze the user's emotional state. For this processing, IBM Watson Tone Analyzer is used, for example. The input is the voice data, and the output is the user's emotional state (e.g., confusion or relief).

[0528] Step 6:

[0529] The server uses the extracted keywords and the results of the sentiment analysis to search the FAQ database. This search is performed using a search engine such as Elasticsearch. The input is the keywords and the results of the sentiment analysis, and the output is a list of relevant FAQs about "how to record TV."

[0530] Step 7:

[0531] The server passes the FAQs identified by the search to a generative AI, which generates a natural, conversational answer that matches the user's emotions. This process uses a generative AI model such as OpenAI's GPT-3. The input is the FAQ and the results of sentiment analysis, and the output is a response such as, "Don't worry. Just press the record button and select the program you want to record."

[0532] Step 8:

[0533] The server generates a response and sends it to the device, which displays it as text on the screen or reads it aloud using a speech synthesis engine. The input is the response, and the output is a response that the user can see or hear.

[0534] Step 9:

[0535] The server collects feedback from operators and uses it to improve the system. Operators evaluate the quality and appropriateness of the answers provided, and the results are used to retrain and tune the model. The input is operator feedback, and the output is an improved model.

[0536] Step 10:

[0537] The server monitors changes to the FAQ database in real time and notifies operators and terminals if any changes are detected. This allows the system to always provide the latest information. The input is new or changed FAQ data, and the output is real-time notifications.

[0538] (Application example 2)

[0539] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0540] Conventional customer support systems are capable of voice recognition and text conversion, but they do not adequately respond to the user's emotions, resulting in low customer satisfaction. Furthermore, the generated answers do not always take the user's emotions into consideration, resulting in inappropriate responses. Furthermore, they lack a mechanism for notifying users of changes to the FAQ database in real time, making it difficult to always provide the latest information to customers.

[0541] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, and means for recognizing the user's emotional state from the voice data using an emotion engine. This makes it possible to generate appropriate answers that take the user's emotions into consideration and to respond to customers quickly and accurately. In addition, by notifying changes to the FAQ database in real time, it is possible to always provide the latest information.

[0542] "Audio data" refers to data of an audio signal provided by a user through an audio input device such as a microphone.

[0543] "Text data" is data in a format in which voice data is converted into character information.

[0544] A "natural language processing model" is an algorithm or machine learning model for analyzing semantic information and context from text data.

[0545] A "similarity search algorithm" is an algorithm for searching a database for related information based on specific keywords or phrases.

[0546] "Generative AI" is an artificial intelligence model that generates natural, conversational text based on input data.

[0547] An "emotion engine" is an algorithm or system that analyzes and recognizes a user's emotional state from voice and text data.

[0548] "Operator feedback" refers to the operator's evaluation and comments regarding the system's accuracy and response content.

[0549] The "FAQ database" is a database that collects frequently asked questions and their answers.

[0550] "Means of real-time notification" refers to a function that instantly notifies relevant parties or systems of database update information, etc.

[0551] The "means for providing an answer to the customer" is a function for presenting the generated answer to the user in text or audio format.

[0552] "User's emotional state" refers to the feelings or mood of the user when they input voice or text.

[0553] "Intent and keywords" refer to verbs and nouns that are at the core of the context and are extracted from the text data entered by the user.

[0554] A system for realizing this application example operates with the following hardware and software configuration.

[0555] Hardware and software used

[0556] Hardware: Microphone (for voice input), server, terminal

[0557] Software: SpeechRecognition library, spaCy, Transformers (Hugging Face), TextBlob

[0558] System Operation Details

[0559] 1. Voice Recognition

[0560] The user speaks their query through the microphone. The device's microphone records the voice data, and the SpeechRecognition library is used to capture the data. For example, consider a user saying, "Please tell me how to record this TV." This voice is converted into text data in real time through Google's speech recognition API.

[0561] 2. Natural Language Processing

[0562] The server analyzes the acquired text data using a natural language processing model (spaCy). Here, intent (verbs) and keywords (nouns) are extracted from the text data. For example, keywords such as "TV recording method" are extracted.

[0563] 3. Sentiment analysis

[0564] We use TextBlob to recognize the user's emotional state from the audio data, which determines whether the emotional state is positive or negative, and this information is then reflected in the generated answer.

[0565] 4. Answer generation

[0566] The server combines the extracted intent, keywords, and sentiment analysis results and passes them to a generative AI (using the Transformers library) as a prompt. The AI ​​then adjusts the format and content of the response appropriately based on the user's emotional state. For example, if the server determines that the user is confused, it generates a response such as, "Don't worry, just press the record button and select the program you want to record."

[0567] 5. Providing answers

[0568] The generated answer is then sent back to the device, which displays the answer as text or reads it aloud, allowing the user to obtain quick and accurate information.

[0569] Specific prompt examples

[0570] Below are some examples of prompts passed to the generative AI:

[0571] text

[0572] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0573] This system is expected to improve customer satisfaction by providing appropriate answers to customers through a consistent process from voice input to emotion recognition, natural language processing, and answer generation.

[0574] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0575] Step 1:

[0576] The user makes a voice inquiry. The voice data is input to the device through the microphone. The device uses the SpeechRecognition library to record the voice data. The recorded voice data is processed as input data.

[0577] Step 2:

[0578] The device converts the recorded voice data into text data through Google's speech recognition API. The input data is voice data, and the output data is text data. This allows the voice content to be expressed in text format.

[0579] Step 3:

[0580] The server receives the text data and uses a natural language processing model (spaCy) to extract intent and keywords. At this stage, the text data is the input and the intent and keywords are the output. Specifically, the text data is grammatically analyzed to identify verbs and nouns.

[0581] Step 4:

[0582] The server uses TextBlob to analyze the user's emotional state from text data. The input data is text data, and the output data is the emotional state (e.g., positive, negative, etc.). The specific operation here is to determine the emotion from the context and tone of the text.

[0583] Step 5:

[0584] The server generates a prompt based on the extracted intent, keywords, and emotional state, and inputs it into a generative AI (Transformers). The input data is the prompt, and the output data is the answer text. The specific operation is for the generative AI to generate an appropriate answer for the input prompt.

[0585] Examples of prompts:

[0586] text

[0587] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0588] Step 6:

[0589] The generated answer text is sent from the server to the terminal. The input data is the answer text, and the output data is the answer that is displayed or read aloud. The terminal provides the answer to the user, providing a highly satisfactory response.

[0590] This system automates the entire process from receiving an inquiry to providing a response, while also providing a response that takes emotions into consideration.

[0591] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0592] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0593] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0594] [Third embodiment]

[0595] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0596] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0597] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0598] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0599] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0600] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0601] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0602] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0603] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0604] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0605] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0606] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0607] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search related FAQs in response to customer inquiries and provide quick and accurate answers. This system works as follows.

[0608] The device receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0609] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0610] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0611] The server uses the extracted keywords to search the FAQ database, and uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, the FAQ about "how to record TV" is identified.

[0612] The identified FAQs are passed to a generative AI, which then generates a natural conversational response, such as "Press the record button and select the program you want to record."

[0613] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0614] The server also collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithm. Additionally, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. This ensures that the system always provides the latest information.

[0615] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0616] The processing flow will be explained below.

[0617] Step 1:

[0618] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0619] Step 2:

[0620] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0621] Step 3:

[0622] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0623] Step 4:

[0624] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0625] Step 5:

[0626] The server uses the extracted keywords to search the FAQ database, and then uses a similarity search algorithm to identify FAQs that closely match the keywords.

[0627] Step 6:

[0628] The server passes the identified FAQ to the generative AI, which then activates it. The generative AI then generates a natural dialogue-style answer based on the FAQ. For example, it generates an answer like, "Press the record button and select the program you want to record."

[0629] Step 7:

[0630] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0631] Step 8:

[0632] The terminal provides the generated answer to the customer, allowing them to get fast and accurate information.

[0633] Step 9:

[0634] The server collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithms, which are then used to retrain and tune the model.

[0635] Step 10:

[0636] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0637] Example 1

[0638] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0639] In conventional customer support systems, it takes a lot of time and effort to manually search for information and generate answers each time a customer makes an inquiry, making it difficult to provide quick and accurate answers. This also increases the burden on operators, leading to problems such as reduced customer satisfaction. Therefore, the present invention aims to utilize automatic speech recognition and generative artificial intelligence to respond to customer inquiries quickly and accurately, thereby reducing the burden on operators and improving customer satisfaction.

[0640] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0641] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative artificial intelligence, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for sending answers to a terminal and providing them to the customer by display or voice. This makes it possible to provide answers to customer inquiries quickly and accurately, reducing the burden on operators and improving customer satisfaction.

[0642] "User" refers to any individual or entity that queries the system.

[0643] A "terminal" is a device used by a user, and refers to an apparatus that includes hardware such as a microphone, speaker, and display.

[0644] "Voice data" refers to data that is a digital recording of a user's speech.

[0645] "Text data" refers to data in which voice data is expressed as text information.

[0646] A "natural language processing model" refers to a mathematical or machine learning model for analyzing and extracting meaning, intent, important keywords, etc. from text data.

[0647] A "similarity search algorithm" refers to an algorithm for searching for highly relevant data for a given query or keyword.

[0648] An "FAQ database" is a database that organizes and stores frequently asked questions and their answers.

[0649] "Generative AI" refers to artificial intelligence technology that generates new text or dialogue based on input information.

[0650] An "operator" refers to a person who manages and monitors the system and handles customer inquiries.

[0651] "Feedback" refers to evaluation information provided by operators or users regarding the performance of the system and the quality of the answers.

[0652] An "answer" is a response provided by the system to a user's inquiry.

[0653] "Real-time notification" refers to the system's ability to instantly notify relevant personnel of any changes or updates to the FAQ database.

[0654] The present invention is a system that utilizes automatic speech recognition and generative artificial intelligence to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system operates as follows.

[0655] A user makes a voice inquiry via a device. For example, the user might say, "Please tell me how to record this TV show." The device uses a built-in microphone to record the user's voice and sends the voice data to a server. The server converts the received voice data into text data using an automatic speech recognition (ASR) model (e.g., a voice recognition service). For example, the voice data "Please tell me how to record this TV show" is converted into text data "Please tell me how to record this TV show."

[0656] The server then uses a natural language processing (NLP) model (e.g., a natural language processing service) to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword. The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm (e.g., a search engine) to identify the most relevant FAQs from a large number of FAQs. As a concrete example, FAQs about "how to record TV" are identified.

[0657] The server then passes the identified FAQ to a generative AI (e.g., a generative AI model), which generates a natural, conversational answer. For example, the answer might be, "Press the record button and select the program you want to record." Here's an example prompt:

[0658] Example prompt sentence:

[0659] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0660] The generated answer is sent back to the terminal, which then displays the answer to the user in text or reads it out loud, allowing the user to quickly and accurately obtain an answer to their inquiry.

[0661] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. Furthermore, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. In this way, the system can always provide the latest information.

[0662] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0663] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0664] Step 1:

[0665] A user makes a voice inquiry via a terminal. For example, "Please tell me how to record this TV program." The input is the user's voice, and the output is the voice data picked up by the terminal's microphone.

[0666] Step 2:

[0667] The device uses a microphone to record the user's voice. The recorded voice data is saved as a digital audio file and sent to the server. The input is the voice data picked up by the device's microphone, and the output is the digital audio file sent to the server.

[0668] Step 3:

[0669] The server passes the received voice data to an automatic speech recognition (ASR) model and converts it into text data. For example, it is converted into text data such as "Please tell me how to record this TV." The input is the voice data received by the server, and the output is the text data converted by the ASR model.

[0670] Step 4:

[0671] The server uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, "TV recording methods" is recognized as an important keyword. The input is the text data output by the ASR model, and the output is the keywords extracted by the NLP model.

[0672] Step 5:

[0673] The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm to identify highly relevant FAQs. For example, it finds FAQs about "how to record TV." The input is the keywords extracted by the NLP model, and the output is the relevant FAQ data identified by the similarity search algorithm.

[0674] Step 6:

[0675] The server passes the identified FAQ to a generative AI system, which generates a natural conversational response. For example, a response such as "Press the record button and select the program you want to record" is generated. The following is an example of a prompt used for generation:

[0676] Example prompt sentence:

[0677] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0678] The input is FAQ data identified by the search algorithm, and the output is answers generated by generative artificial intelligence.

[0679] Step 7:

[0680] The generated answer is sent from the server to the device. The input is the answer generated by the generative AI, and the output is the answer data sent to the device.

[0681] Step 8:

[0682] The device displays the received answer to the user as text or reads it out loud. Specifically, in text display mode, the text is displayed on the screen, and in audio mode, the answer is played out loud from the speaker. The input is the answer data sent from the server, and the output is the answer information received by the user.

[0683] (Application example 1)

[0684] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0685] Conventional customer support systems have difficulty providing accurate and prompt answers to customer questions, and are particularly inadequate when it comes to responses to voice inquiries. Furthermore, they place a heavy burden on operators, making it difficult to improve the system in real time, and the answers often lack a natural feel. These issues often lead to a decline in customer satisfaction.

[0686] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0687] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, means for providing answers to customers, voice input means, and means for receiving customer questions as voice input using a smart device. This makes it possible to provide quick and accurate answers to customer questions using voice input, thereby improving the efficiency of customer support and customer satisfaction.

[0688] "Audio data" refers to data obtained by converting an audio signal into digital format.

[0689] "Text data" refers to data obtained by converting voice data into text information.

[0690] A "natural language processing model" is a type of artificial intelligence used to extract intent and keywords from text data.

[0691] A "similarity search algorithm" is a method for searching for data that is most related to a specified keyword from among a large amount of data.

[0692] "Generative AI" refers to artificial intelligence technology that generates natural, conversational sentences based on given information.

[0693] "Operator" means a person assigned to assist users of the System.

[0694] "Feedback" refers to evaluations and opinions collected to improve the system.

[0695] "Smart devices" refer to highly functional electronic devices such as smartphones and head-mounted displays.

[0696] "Audio input means" refers to a microphone or related technology for collecting audio as digital data.

[0697] An "FAQ database" is a database that compiles frequently asked questions and their answers.

[0698] "Real-time notification" refers to the function of instantly notifying you of changes or updates to information.

[0699] "Customer" refers to a consumer who uses a service or product.

[0700] "Dialogue style" refers to a style of expression that involves interaction in a natural, conversational format.

[0701] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system is composed of elements such as "voice data," "text data," "natural language processing model," "similarity search algorithm," "generative AI," "operator," "feedback," "smart device," "voice input means," "FAQ database," "real-time notification," "customer," and "interactive format."

[0702] The operation of the system will now be described in detail.

[0703] The terminal (a smart device such as a smartphone or head-mounted display) receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The terminal records the customer's voice via a microphone and sends the voice data to the server.

[0704] The server converts the voice data into text data using an automatic speech recognition (ASR) model (e.g., Google Cloud Speech-to-Text). The converted text data has the format "Please tell me how to record this TV."

[0705] The server then uses a natural language processing (NLP) model (e.g., spaCy) to extract intent and keywords from the text data. In this case, the NLP model identifies the key keyword, "how to record TV."

[0706] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (e.g., Elasticsearch) to identify the most relevant FAQs, such as those about "how to record TV."

[0707] The identified FAQs are then passed to a generative AI (e.g., OpenAI GPT-4), which generates a natural conversational answer, such as "Press the record button and select the program you want to record."

[0708] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0709] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. When changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and relevant parties as necessary.

[0710] As a concrete example, if a customer's question is "How do I record this TV?" the system will process it as follows:

[0711] 1. The customer's voice is collected by the microphone on their smart device.

[0712] 2. The audio data is sent to the server and converted to text data using Google Cloud Speech-to-Text.

[0713] 3. The text data "How to record TV" is extracted using spaCy.

[0714] 4. Related FAQs are searched in Elasticsearch.

[0715] 5. GPT-4 generates natural conversational answers based on the identified FAQs.

[0716] 6. The generated answer is sent to the device and displayed as text or provided as audio.

[0717] An example of an input prompt is as follows:

[0718] Q: A customer is asking how to record TV. They say, "How do I record this TV?" Generate an appropriate FAQ-based answer for this question.

[0719] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0720] Step 1:

[0721] The terminal (smart device) receives a user (customer) inquiry via voice. Specifically, the user says, "Please tell me how to record this TV program." This voice is collected using the terminal's built-in microphone. Voice data is generated and sent to the server.

[0722] Step 2:

[0723] The server converts the received voice data into text data using an automatic speech recognition (ASR) model (Google Cloud Speech-to-Text). Specifically, the server passes the voice data to the ASR model, which outputs the text data "Please tell me how to record this TV." This converts the voice data into text information.

[0724] Step 3:

[0725] The server passes the converted text data to a natural language processing (NLP) model (spaCy) to extract intent and keywords. Specifically, the keyword "TV recording method" is extracted from the text data. The NLP model analyzes the input text and identifies important information.

[0726] Step 4:

[0727] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (Elasticsearch). Specifically, the server passes the keyword "TV recording method" to Elasticsearch to search for related FAQs. The most relevant FAQ is identified.

[0728] Step 5:

[0729] The server passes the identified FAQs to a generative AI (OpenAI GPT-4), which generates natural dialogue-style answers. Specifically, the FAQ data is input as a prompt into GPT-4, which outputs the answer "Press the record button and select the program you want to record." The generative AI generates appropriate dialogue-style sentences based on the input data.

[0730] Step 6:

[0731] The server sends the generated answer to the terminal, which then displays the answer to the customer in text or reads it aloud. Specifically, in the case of text display, the answer is displayed on the terminal screen, and in the case of audio, the answer is read aloud from the terminal's speaker.

[0732] Step 7:

[0733] The server collects feedback from operators and uses it to improve the system. Specifically, it collects feedback data and uses it to improve the accuracy of natural language processing models and similarity search algorithms. This allows the system's performance to continuously improve.

[0734] Step 8:

[0735] The server monitors changes to the FAQ database in real time and notifies operators and other relevant parties as necessary. Specifically, a process that monitors database updates is constantly running, and if a change is detected, a notification is automatically sent. This allows the system to always maintain the latest information.

[0736] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0737] This invention is a system that combines automatic speech recognition, natural language processing, generative AI, and an emotion engine that recognizes user emotions to provide more accurate responses to customer inquiries. This system operates as follows.

[0738] The device receives customer inquiries via voice. For example, consider a case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0739] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0740] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0741] Furthermore, the server activates an emotion engine to analyze the user's emotion from the voice data. The emotion engine analyzes the user's emotional state (e.g., joy, anger, sadness, etc.) from the voice data and obtains the result. This emotion analysis result is used to generate a response later.

[0742] The server uses the extracted keywords and the sentiment analysis results to search the FAQ database. It then uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, it identifies FAQs about "how to record TV."

[0743] The identified FAQs are passed to a generative AI, which then activates the AI. Based on the FAQs and the results of sentiment analysis, the generative AI generates a natural, conversational response that matches the user's emotions. For example, if it determines that the user is confused, it generates a response that includes reassuring language, such as, "Don't worry, just press the record button and select the program you want to record."

[0744] The generated answer is then sent back to the device, which displays it as text or reads it aloud.

[0745] The device provides customers with generated answers, which provide fast, accurate information and are emotionally sensitive, further increasing customer satisfaction.

[0746] The server also collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0747] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0748] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the customer's feelings, thereby reducing the burden on operators and significantly improving customer satisfaction.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0752] Step 2:

[0753] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0754] Step 3:

[0755] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0756] Step 4:

[0757] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0758] Step 5:

[0759] The server activates an emotion engine to analyze the user's emotional state from the voice data. The emotion engine recognizes emotions such as "confused" or "angry" based on the user's tone and intonation of voice.

[0760] Step 6:

[0761] The server combines the results of the emotion engine and the NLP model, searches the FAQ database based on the extracted keywords and the sentiment analysis results, and then uses a similarity search algorithm to identify FAQs that have a high matching frequency with the keywords.

[0762] Step 7:

[0763] The server passes the identified FAQ to the generative AI, which then activates it. Based on the FAQ and the results of sentiment analysis, the generative AI generates a natural, conversational response that takes the user's emotions into consideration. For example, it might generate a response like, "Press the record button and select the program you want to record. Please also use our operation guide."

[0764] Step 8:

[0765] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0766] Step 9:

[0767] The terminal provides the customer with a generated answer, allowing them to obtain quick and accurate information. In addition, the response is sensitive to the user's feelings, improving customer satisfaction.

[0768] Step 10:

[0769] The server collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[0770] Step 11:

[0771] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0772] Example 2

[0773] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] Conventional customer support systems use voice recognition and natural language processing, but lack the mechanisms to understand user emotions and respond appropriately. This has led to problems such as reduced user satisfaction and increased operator workload. Furthermore, FAQ databases are not updated in real time, which can result in delays in providing the latest information. The purpose of this invention is to solve these problems.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0776] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for analyzing the user's emotional state from the voice data, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format that matches the user's emotions using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for providing answers to customers. This enables accurate and prompt information provision that takes user emotions into consideration, thereby improving customer satisfaction and reducing the burden on operators.

[0777] "Voice data" refers to data in which the voice spoken by the user via the terminal is recorded in digital format.

[0778] "Text data" refers to voice data converted into text format using an automatic speech recognition (ASR) model.

[0779] A "natural language processing model" is a general term for algorithms and software that analyze text data and extract intent and keywords.

[0780] "Intention" refers to specific actions and intentions extracted by a natural language processing model from what the user has said.

[0781] "Keywords" refer to important words or phrases identified from text data by a natural language processing model.

[0782] "Automatic speech recognition model" is a general term for algorithms and software for converting voice data into text data.

[0783] "Emotion engine" is a general term for algorithms and software for analyzing a user's emotional state (e.g., joy, anger, sadness, etc.) from voice or text data.

[0784] The "similarity search algorithm" is an algorithm for identifying highly relevant data from within the FAQ database based on extracted keywords.

[0785] The "FAQ database" is a database that compiles frequently asked questions and their answers.

[0786] "Generative AI" is a general term for algorithms and software that generate natural, conversational answers based on given data (e.g., FAQs or sentiment analysis results).

[0787] "Operator feedback" refers to the operator's evaluation of the system's answers, and is the data used to retrain and tune the model.

[0788] "System improvement" refers to the process of improving the accuracy of automatic speech recognition models, natural language processing models, emotion engines, and generative AI based on collected feedback.

[0789] "Means of real-time notification" refers to a system that immediately transmits information to the server and related terminals when there are changes in the FAQ database.

[0790] This invention is a system that combines automatic speech recognition (ASR) technology, natural language processing (NLP) technology, generative AI technology, and emotion recognition technology to provide customer service that is appropriate to the user's emotions. This system is mainly composed of a server and a terminal, and operates as follows.

[0791] The device receives the user's voice inquiry. For example, consider the case where the user says, "Please tell me how to record this TV program." The device records the user's voice using a built-in microphone and sends the voice data in digital format to a server. Voice data is typically sent using an HTTP request over the Internet.

[0792] The server uses an automatic speech recognition (ASR) model on the received voice data to convert it into text data. This process can be performed using a voice recognition service such as Google Cloud Speech-to-Text API. For example, the server can generate text data such as "Please tell me how to record this TV program" from the voice data.

[0793] The server then uses a natural language processing (NLP) model to extract user intent and key keywords from the text data. This process can be performed using natural language processing libraries such as spaCy or BERT. For example, the NLP model might extract the key keyword "how to record TV" from the text data.

[0794] The server then activates an emotion engine to analyze the user's emotional state from the voice and text data. This can be done using emotion analysis tools such as IBM Watson Tone Analyzer. For example, the server can determine from the voice data that the user is confused.

[0795] The server uses the extracted keywords and the results of sentiment analysis to search the FAQ database. It uses a similarity search algorithm to identify the most relevant FAQs from among many. This process can be performed using a search engine such as Elasticsearch. For example, the server identifies FAQs about "how to record TV."

[0796] The identified FAQs are passed to a generative AI, which uses the FAQs and sentiment analysis results to generate natural, conversational answers that fit the user's emotions. This process can use a generative AI model such as OpenAI's GPT-3. For example, a generated answer might be, "Don't worry. Just press the record button and select the program you want to record."

[0797] The generated answer is then sent back to the device, which can then display it as text or read it aloud using a speech synthesis engine. For example, the device may display it to the user as a text message and read it aloud using speech synthesis.

[0798] The server also collects feedback from operators and uses it to improve the accuracy of the automatic speech recognition model, natural language processing model, emotion engine, and generative AI. This feedback is used to retrain the models and tune parameters. Furthermore, the server monitors changes to the FAQ database in real time and notifies operators and related devices if any changes are detected.

[0799] As a concrete example, the following prompt sentences can be input to a generative AI model:

[0800] Example prompt: "How do I record this TV show?"

[0801] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the user's feelings, thereby improving customer satisfaction and reducing the burden on operators.

[0802] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0803] Step 1:

[0804] The device acquires the user's voice. When the user speaks into the microphone, such as "Please tell me how to record this TV program," the device records the voice and generates audio data. The input is the user's speech, and the output is digital audio data.

[0805] Step 2:

[0806] The device sends the recorded audio data to the server. This transmission is performed via an HTTP request over the Internet. The input is audio data, and the output is the transmission of audio data to the server.

[0807] Step 3:

[0808] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For this conversion, it uses, for example, the Google Cloud Speech-to-Text API. The input is voice data, and the output is text data such as "Please tell me how to record this TV."

[0809] Step 4:

[0810] The server then passes the converted text data to a natural language processing (NLP) model to extract key intent and keywords. Examples of NLP models used for this process include spaCy and BERT. The input is text data, and the output is extracted keywords, such as "how to record TV."

[0811] Step 5:

[0812] The server passes the voice data to an emotion engine to analyze the user's emotional state. For this processing, IBM Watson Tone Analyzer is used, for example. The input is the voice data, and the output is the user's emotional state (e.g., confusion or relief).

[0813] Step 6:

[0814] The server uses the extracted keywords and the results of the sentiment analysis to search the FAQ database. This search is performed using a search engine such as Elasticsearch. The input is the keywords and the results of the sentiment analysis, and the output is a list of relevant FAQs about "how to record TV."

[0815] Step 7:

[0816] The server passes the FAQs identified by the search to a generative AI, which generates a natural, conversational answer that matches the user's emotions. This process uses a generative AI model such as OpenAI's GPT-3. The input is the FAQ and the results of sentiment analysis, and the output is a response such as, "Don't worry. Just press the record button and select the program you want to record."

[0817] Step 8:

[0818] The server generates a response and sends it to the device, which displays it as text on the screen or reads it aloud using a speech synthesis engine. The input is the response, and the output is a response that the user can see or hear.

[0819] Step 9:

[0820] The server collects feedback from operators and uses it to improve the system. Operators evaluate the quality and appropriateness of the answers provided, and the results are used to retrain and tune the model. The input is operator feedback, and the output is an improved model.

[0821] Step 10:

[0822] The server monitors changes to the FAQ database in real time and notifies operators and terminals if any changes are detected. This allows the system to always provide the latest information. The input is new or changed FAQ data, and the output is real-time notifications.

[0823] (Application example 2)

[0824] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0825] Conventional customer support systems are capable of voice recognition and text conversion, but they do not adequately respond to the user's emotions, resulting in low customer satisfaction. Furthermore, the generated answers do not always take the user's emotions into consideration, resulting in inappropriate responses. Furthermore, they lack a mechanism for notifying users of changes to the FAQ database in real time, making it difficult to always provide the latest information to customers.

[0826] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, and means for recognizing the user's emotional state from the voice data using an emotion engine. This makes it possible to generate appropriate answers that take the user's emotions into consideration and to respond to customers quickly and accurately. In addition, by notifying changes to the FAQ database in real time, it is possible to always provide the latest information.

[0827] "Audio data" refers to data of an audio signal provided by a user through an audio input device such as a microphone.

[0828] "Text data" is data in a format in which voice data is converted into character information.

[0829] A "natural language processing model" is an algorithm or machine learning model for analyzing semantic information and context from text data.

[0830] A "similarity search algorithm" is an algorithm for searching a database for related information based on specific keywords or phrases.

[0831] "Generative AI" is an artificial intelligence model that generates natural, conversational text based on input data.

[0832] An "emotion engine" is an algorithm or system that analyzes and recognizes a user's emotional state from voice and text data.

[0833] "Operator feedback" refers to the operator's evaluation and comments regarding the system's accuracy and response content.

[0834] The "FAQ database" is a database that collects frequently asked questions and their answers.

[0835] "Means of real-time notification" refers to a function that instantly notifies relevant parties or systems of database update information, etc.

[0836] The "means for providing an answer to the customer" is a function for presenting the generated answer to the user in text or audio format.

[0837] "User's emotional state" refers to the feelings or mood of the user when they input voice or text.

[0838] "Intent and keywords" refer to verbs and nouns that are at the core of the context and are extracted from the text data entered by the user.

[0839] A system for realizing this application example operates with the following hardware and software configuration.

[0840] Hardware and software used

[0841] Hardware: Microphone (for voice input), server, terminal

[0842] Software: SpeechRecognition library, spaCy, Transformers (Hugging Face), TextBlob

[0843] System Operation Details

[0844] 1. Voice Recognition

[0845] The user speaks their query through the microphone. The device's microphone records the voice data, and the SpeechRecognition library is used to capture the data. For example, consider a user saying, "Please tell me how to record this TV." This voice is converted into text data in real time through Google's speech recognition API.

[0846] 2. Natural Language Processing

[0847] The server analyzes the acquired text data using a natural language processing model (spaCy). Here, intent (verbs) and keywords (nouns) are extracted from the text data. For example, keywords such as "TV recording method" are extracted.

[0848] 3. Sentiment analysis

[0849] We use TextBlob to recognize the user's emotional state from the audio data, which determines whether the emotional state is positive or negative, and this information is then reflected in the generated answer.

[0850] 4. Answer generation

[0851] The server combines the extracted intent, keywords, and sentiment analysis results and passes them to a generative AI (using the Transformers library) as a prompt. The AI ​​then adjusts the format and content of the response appropriately based on the user's emotional state. For example, if the server determines that the user is confused, it generates a response such as, "Don't worry, just press the record button and select the program you want to record."

[0852] 5. Providing answers

[0853] The generated answer is then sent back to the device, which displays the answer as text or reads it aloud, allowing the user to obtain quick and accurate information.

[0854] Specific prompt examples

[0855] Below are some examples of prompts passed to the generative AI:

[0856] text

[0857] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0858] This system is expected to improve customer satisfaction by providing appropriate answers to customers through a consistent process from voice input to emotion recognition, natural language processing, and answer generation.

[0859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0860] Step 1:

[0861] The user makes a voice inquiry. The voice data is input to the device through the microphone. The device uses the SpeechRecognition library to record the voice data. The recorded voice data is processed as input data.

[0862] Step 2:

[0863] The device converts the recorded voice data into text data through Google's speech recognition API. The input data is voice data, and the output data is text data. This allows the voice content to be expressed in text format.

[0864] Step 3:

[0865] The server receives the text data and uses a natural language processing model (spaCy) to extract intent and keywords. At this stage, the text data is the input and the intent and keywords are the output. Specifically, the text data is grammatically analyzed to identify verbs and nouns.

[0866] Step 4:

[0867] The server uses TextBlob to analyze the user's emotional state from text data. The input data is text data, and the output data is the emotional state (e.g., positive, negative, etc.). The specific operation here is to determine the emotion from the context and tone of the text.

[0868] Step 5:

[0869] The server generates a prompt based on the extracted intent, keywords, and emotional state, and inputs it into a generative AI (Transformers). The input data is the prompt, and the output data is the answer text. The specific operation is for the generative AI to generate an appropriate answer for the input prompt.

[0870] Examples of prompts:

[0871] text

[0872] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[0873] Step 6:

[0874] The generated answer text is sent from the server to the terminal. The input data is the answer text, and the output data is the answer that is displayed or read aloud. The terminal provides the answer to the user, providing a highly satisfactory response.

[0875] This system automates the entire process from receiving an inquiry to providing a response, while also providing a response that takes emotions into consideration.

[0876] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0877] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0878] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0879] [Fourth embodiment]

[0880] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0881] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0882] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0883] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0884] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0885] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0886] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0887] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0888] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0889] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0890] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0891] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0892] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0893] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search related FAQs in response to customer inquiries and provide quick and accurate answers. This system works as follows.

[0894] The device receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[0895] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[0896] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[0897] The server uses the extracted keywords to search the FAQ database, and uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, the FAQ about "how to record TV" is identified.

[0898] The identified FAQs are passed to a generative AI, which then generates a natural conversational response, such as "Press the record button and select the program you want to record."

[0899] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0900] The server also collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithm. Additionally, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. This ensures that the system always provides the latest information.

[0901] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0902] The processing flow will be explained below.

[0903] Step 1:

[0904] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[0905] Step 2:

[0906] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[0907] Step 3:

[0908] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[0909] Step 4:

[0910] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[0911] Step 5:

[0912] The server uses the extracted keywords to search the FAQ database, and then uses a similarity search algorithm to identify FAQs that closely match the keywords.

[0913] Step 6:

[0914] The server passes the identified FAQ to the generative AI, which then activates it. The generative AI then generates a natural dialogue-style answer based on the FAQ. For example, it generates an answer like, "Press the record button and select the program you want to record."

[0915] Step 7:

[0916] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[0917] Step 8:

[0918] The terminal provides the generated answer to the customer, allowing them to get fast and accurate information.

[0919] Step 9:

[0920] The server collects feedback from operators and uses it to improve the accuracy of the NLP model and similarity search algorithms, which are then used to retrain and tune the model.

[0921] Step 10:

[0922] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[0923] Example 1

[0924] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0925] In conventional customer support systems, it takes a lot of time and effort to manually search for information and generate answers each time a customer makes an inquiry, making it difficult to provide quick and accurate answers. This also increases the burden on operators, leading to problems such as reduced customer satisfaction. Therefore, the present invention aims to utilize automatic speech recognition and generative artificial intelligence to respond to customer inquiries quickly and accurately, thereby reducing the burden on operators and improving customer satisfaction.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0927] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative artificial intelligence, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for sending answers to a terminal and providing them to the customer by display or voice. This makes it possible to provide answers to customer inquiries quickly and accurately, reducing the burden on operators and improving customer satisfaction.

[0928] "User" refers to any individual or entity that queries the system.

[0929] A "terminal" is a device used by a user, and refers to an apparatus that includes hardware such as a microphone, speaker, and display.

[0930] "Voice data" refers to data that is a digital recording of a user's speech.

[0931] "Text data" refers to data in which voice data is expressed as text information.

[0932] A "natural language processing model" refers to a mathematical or machine learning model for analyzing and extracting meaning, intent, important keywords, etc. from text data.

[0933] A "similarity search algorithm" refers to an algorithm for searching for highly relevant data for a given query or keyword.

[0934] An "FAQ database" is a database that organizes and stores frequently asked questions and their answers.

[0935] "Generative AI" refers to artificial intelligence technology that generates new text or dialogue based on input information.

[0936] An "operator" refers to a person who manages and monitors the system and handles customer inquiries.

[0937] "Feedback" refers to evaluation information provided by operators or users regarding the performance of the system and the quality of the answers.

[0938] An "answer" is a response provided by the system to a user's inquiry.

[0939] "Real-time notification" refers to the system's ability to instantly notify relevant personnel of any changes or updates to the FAQ database.

[0940] The present invention is a system that utilizes automatic speech recognition and generative artificial intelligence to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system operates as follows.

[0941] A user makes a voice inquiry via a device. For example, the user might say, "Please tell me how to record this TV show." The device uses a built-in microphone to record the user's voice and sends the voice data to a server. The server converts the received voice data into text data using an automatic speech recognition (ASR) model (e.g., a voice recognition service). For example, the voice data "Please tell me how to record this TV show" is converted into text data "Please tell me how to record this TV show."

[0942] The server then uses a natural language processing (NLP) model (e.g., a natural language processing service) to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword. The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm (e.g., a search engine) to identify the most relevant FAQs from a large number of FAQs. As a concrete example, FAQs about "how to record TV" are identified.

[0943] The server then passes the identified FAQ to a generative AI (e.g., a generative AI model), which generates a natural, conversational answer. For example, the answer might be, "Press the record button and select the program you want to record." Here's an example prompt:

[0944] Example prompt sentence:

[0945] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0946] The generated answer is sent back to the terminal, which then displays the answer to the user in text or reads it out loud, allowing the user to quickly and accurately obtain an answer to their inquiry.

[0947] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. Furthermore, when changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and other relevant parties as necessary. In this way, the system can always provide the latest information.

[0948] The system of the present invention makes it possible to respond to customer inquiries quickly and accurately, reduce the burden on operators, and improve customer satisfaction.

[0949] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0950] Step 1:

[0951] A user makes a voice inquiry via a terminal. For example, "Please tell me how to record this TV program." The input is the user's voice, and the output is the voice data picked up by the terminal's microphone.

[0952] Step 2:

[0953] The device uses a microphone to record the user's voice. The recorded voice data is saved as a digital audio file and sent to the server. The input is the voice data picked up by the device's microphone, and the output is the digital audio file sent to the server.

[0954] Step 3:

[0955] The server passes the received voice data to an automatic speech recognition (ASR) model and converts it into text data. For example, it is converted into text data such as "Please tell me how to record this TV." The input is the voice data received by the server, and the output is the text data converted by the ASR model.

[0956] Step 4:

[0957] The server uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, "TV recording methods" is recognized as an important keyword. The input is the text data output by the ASR model, and the output is the keywords extracted by the NLP model.

[0958] Step 5:

[0959] The server uses the extracted keywords to search the FAQ database. It uses a similarity search algorithm to identify highly relevant FAQs. For example, it finds FAQs about "how to record TV." The input is the keywords extracted by the NLP model, and the output is the relevant FAQ data identified by the similarity search algorithm.

[0960] Step 6:

[0961] The server passes the identified FAQ to a generative AI system, which generates a natural conversational response. For example, a response such as "Press the record button and select the program you want to record" is generated. The following is an example of a prompt used for generation:

[0962] Example prompt sentence:

[0963] "Generate appropriate answers to user questions, such as: 'How do I record TV?'"

[0964] The input is FAQ data identified by the search algorithm, and the output is answers generated by generative artificial intelligence.

[0965] Step 7:

[0966] The generated answer is sent from the server to the device. The input is the answer generated by the generative AI, and the output is the answer data sent to the device.

[0967] Step 8:

[0968] The device displays the received answer to the user as text or reads it out loud. Specifically, in text display mode, the text is displayed on the screen, and in audio mode, the answer is played out loud from the speaker. The input is the answer data sent from the server, and the output is the answer information received by the user.

[0969] (Application example 1)

[0970] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0971] Conventional customer support systems have difficulty providing accurate and prompt answers to customer questions, and are particularly inadequate when it comes to responses to voice inquiries. Furthermore, they place a heavy burden on operators, making it difficult to improve the system in real time, and the answers often lack a natural feel. These issues often lead to a decline in customer satisfaction.

[0972] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0973] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, means for providing answers to customers, voice input means, and means for receiving customer questions as voice input using a smart device. This makes it possible to provide quick and accurate answers to customer questions using voice input, thereby improving the efficiency of customer support and customer satisfaction.

[0974] "Audio data" refers to data obtained by converting an audio signal into digital format.

[0975] "Text data" refers to data obtained by converting voice data into text information.

[0976] A "natural language processing model" is a type of artificial intelligence used to extract intent and keywords from text data.

[0977] A "similarity search algorithm" is a method for searching for data that is most related to a specified keyword from among a large amount of data.

[0978] "Generative AI" refers to artificial intelligence technology that generates natural, conversational sentences based on given information.

[0979] "Operator" means a person assigned to assist users of the System.

[0980] "Feedback" refers to evaluations and opinions collected to improve the system.

[0981] "Smart devices" refer to highly functional electronic devices such as smartphones and head-mounted displays.

[0982] "Audio input means" refers to a microphone or related technology for collecting audio as digital data.

[0983] An "FAQ database" is a database that compiles frequently asked questions and their answers.

[0984] "Real-time notification" refers to the function of instantly notifying you of changes or updates to information.

[0985] "Customer" refers to a consumer who uses a service or product.

[0986] "Dialogue style" refers to a style of expression that involves interaction in a natural, conversational format.

[0987] This invention is a system that utilizes automatic speech recognition and generative AI to automatically search for FAQs related to customer inquiries and provide quick and accurate answers. This system is composed of elements such as "voice data," "text data," "natural language processing model," "similarity search algorithm," "generative AI," "operator," "feedback," "smart device," "voice input means," "FAQ database," "real-time notification," "customer," and "interactive format."

[0988] The operation of the system will now be described in detail.

[0989] The terminal (a smart device such as a smartphone or head-mounted display) receives customer inquiries via voice. As a concrete example, consider the case where a customer says, "Please tell me how to record this TV." The terminal records the customer's voice via a microphone and sends the voice data to the server.

[0990] The server converts the voice data into text data using an automatic speech recognition (ASR) model (e.g., Google Cloud Speech-to-Text). The converted text data has the format "Please tell me how to record this TV."

[0991] The server then uses a natural language processing (NLP) model (e.g., spaCy) to extract intent and keywords from the text data. In this case, the NLP model identifies the key keyword, "how to record TV."

[0992] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (e.g., Elasticsearch) to identify the most relevant FAQs, such as those about "how to record TV."

[0993] The identified FAQs are then passed to a generative AI (e.g., OpenAI GPT-4), which generates a natural conversational answer, such as "Press the record button and select the program you want to record."

[0994] The generated answer is then sent back to the device, which then displays the answer to the customer as text or reads it out loud, allowing the customer to receive a quick and accurate answer to their inquiry.

[0995] In addition, the server collects feedback from operators and uses it to improve the accuracy of natural language processing models and similarity search algorithms. When changes are made to the FAQ database, the server monitors the changes in real time and notifies operators and relevant parties as necessary.

[0996] As a concrete example, if a customer's question is "How do I record this TV?" the system will process it as follows:

[0997] 1. The customer's voice is collected by the microphone on their smart device.

[0998] 2. The audio data is sent to the server and converted to text data using Google Cloud Speech-to-Text.

[0999] 3. The text data "How to record TV" is extracted using spaCy.

[1000] 4. Related FAQs are searched in Elasticsearch.

[1001] 5. GPT-4 generates natural conversational answers based on the identified FAQs.

[1002] 6. The generated answer is sent to the device and displayed as text or provided as audio.

[1003] An example of an input prompt is as follows:

[1004] Q: A customer is asking how to record TV. They say, "How do I record this TV?" Generate an appropriate FAQ-based answer for this question.

[1005] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1006] Step 1:

[1007] The terminal (smart device) receives a user (customer) inquiry via voice. Specifically, the user says, "Please tell me how to record this TV program." This voice is collected using the terminal's built-in microphone. Voice data is generated and sent to the server.

[1008] Step 2:

[1009] The server converts the received voice data into text data using an automatic speech recognition (ASR) model (Google Cloud Speech-to-Text). Specifically, the server passes the voice data to the ASR model, which outputs the text data "Please tell me how to record this TV." This converts the voice data into text information.

[1010] Step 3:

[1011] The server passes the converted text data to a natural language processing (NLP) model (spaCy) to extract intent and keywords. Specifically, the keyword "TV recording method" is extracted from the text data. The NLP model analyzes the input text and identifies important information.

[1012] Step 4:

[1013] The server uses the extracted keywords to search the FAQ database using a similarity search algorithm (Elasticsearch). Specifically, the server passes the keyword "TV recording method" to Elasticsearch to search for related FAQs. The most relevant FAQ is identified.

[1014] Step 5:

[1015] The server passes the identified FAQs to a generative AI (OpenAI GPT-4), which generates natural dialogue-style answers. Specifically, the FAQ data is input as a prompt into GPT-4, which outputs the answer "Press the record button and select the program you want to record." The generative AI generates appropriate dialogue-style sentences based on the input data.

[1016] Step 6:

[1017] The server sends the generated answer to the terminal, which then displays the answer to the customer in text or reads it aloud. Specifically, in the case of text display, the answer is displayed on the terminal screen, and in the case of audio, the answer is read aloud from the terminal's speaker.

[1018] Step 7:

[1019] The server collects feedback from operators and uses it to improve the system. Specifically, it collects feedback data and uses it to improve the accuracy of natural language processing models and similarity search algorithms. This allows the system's performance to continuously improve.

[1020] Step 8:

[1021] The server monitors changes to the FAQ database in real time and notifies operators and other relevant parties as necessary. Specifically, a process that monitors database updates is constantly running, and if a change is detected, a notification is automatically sent. This allows the system to always maintain the latest information.

[1022] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1023] This invention is a system that combines automatic speech recognition, natural language processing, generative AI, and an emotion engine that recognizes user emotions to provide more accurate responses to customer inquiries. This system operates as follows.

[1024] The device receives customer inquiries via voice. For example, consider a case where a customer says, "Please tell me how to record this TV." The device records the customer's voice via a microphone and sends the voice data to the server.

[1025] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For example, the voice data is converted into text data such as "Please tell me how to record this TV."

[1026] The server then uses a natural language processing (NLP) model to extract intent and keywords from the text data. In this case, the NLP model identifies "how to record TV" as an important keyword.

[1027] Furthermore, the server activates an emotion engine to analyze the user's emotion from the voice data. The emotion engine analyzes the user's emotional state (e.g., joy, anger, sadness, etc.) from the voice data and obtains the result. This emotion analysis result is used to generate a response later.

[1028] The server uses the extracted keywords and the sentiment analysis results to search the FAQ database. It then uses a similarity search algorithm to identify the most relevant FAQ from a large number of FAQs. For example, it identifies FAQs about "how to record TV."

[1029] The identified FAQs are passed to a generative AI, which then activates the AI. Based on the FAQs and the results of sentiment analysis, the generative AI generates a natural, conversational response that matches the user's emotions. For example, if it determines that the user is confused, it generates a response that includes reassuring language, such as, "Don't worry, just press the record button and select the program you want to record."

[1030] The generated answer is then sent back to the device, which displays it as text or reads it aloud.

[1031] The device provides customers with generated answers, which provide fast, accurate information and are emotionally sensitive, further increasing customer satisfaction.

[1032] The server also collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[1033] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[1034] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the customer's feelings, thereby reducing the burden on operators and significantly improving customer satisfaction.

[1035] The processing flow will be explained below.

[1036] Step 1:

[1037] The terminal receives the customer's voice via a microphone. For example, when the customer asks, "Please tell me how to record this TV program."

[1038] Step 2:

[1039] The device records the received audio, converts the audio data into an appropriate format (e.g., WAV, MP3), and sends it to the server.

[1040] Step 3:

[1041] The server invokes an automatic speech recognition (ASR) model to convert the speech data into text data, specifically, generating the text data "Please tell me how to record this TV program."

[1042] Step 4:

[1043] The server runs a natural language processing (NLP) model and inputs text data. The NLP model extracts intent and keywords from the text data. In this example, it identifies "how to record TV" as an important keyword.

[1044] Step 5:

[1045] The server activates an emotion engine to analyze the user's emotional state from the voice data. The emotion engine recognizes emotions such as "confused" or "angry" based on the user's tone and intonation of voice.

[1046] Step 6:

[1047] The server combines the results of the emotion engine and the NLP model, searches the FAQ database based on the extracted keywords and the sentiment analysis results, and then uses a similarity search algorithm to identify FAQs that have a high matching frequency with the keywords.

[1048] Step 7:

[1049] The server passes the identified FAQ to the generative AI, which then activates it. Based on the FAQ and the results of sentiment analysis, the generative AI generates a natural, conversational response that takes the user's emotions into consideration. For example, it might generate a response like, "Press the record button and select the program you want to record. Please also use our operation guide."

[1050] Step 8:

[1051] The generated answer is sent from the server to the device, which displays it as text or reads it aloud.

[1052] Step 9:

[1053] The terminal provides the customer with a generated answer, allowing them to obtain quick and accurate information. In addition, the response is sensitive to the user's feelings, improving customer satisfaction.

[1054] Step 10:

[1055] The server collects feedback from operators and uses it to improve the accuracy of NLP models, similarity search algorithms, generative AI, and emotion engines. The feedback is used to retrain and tune the models.

[1056] Step 11:

[1057] The server monitors changes to the FAQ database in real time and notifies operators and related terminals if any changes are detected, allowing the system to always provide the latest information.

[1058] Example 2

[1059] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1060] Conventional customer support systems use voice recognition and natural language processing, but lack the mechanisms to understand user emotions and respond appropriately. This has led to problems such as reduced user satisfaction and increased operator workload. Furthermore, FAQ databases are not updated in real time, which can result in delays in providing the latest information. The purpose of this invention is to solve these problems.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1062] In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, means for analyzing the user's emotional state from the voice data, means for searching for related FAQ data using a similarity search algorithm, means for generating answers in a natural dialogue format that matches the user's emotions using generative AI, means for collecting operator feedback and using it to improve the system, means for notifying changes to the FAQ database in real time, and means for providing answers to customers. This enables accurate and prompt information provision that takes user emotions into consideration, thereby improving customer satisfaction and reducing the burden on operators.

[1063] "Voice data" refers to data in which the voice spoken by the user via the terminal is recorded in digital format.

[1064] "Text data" refers to voice data converted into text format using an automatic speech recognition (ASR) model.

[1065] A "natural language processing model" is a general term for algorithms and software that analyze text data and extract intent and keywords.

[1066] "Intention" refers to specific actions and intentions extracted by a natural language processing model from what the user has said.

[1067] "Keywords" refer to important words or phrases identified from text data by a natural language processing model.

[1068] "Automatic speech recognition model" is a general term for algorithms and software for converting voice data into text data.

[1069] "Emotion engine" is a general term for algorithms and software for analyzing a user's emotional state (e.g., joy, anger, sadness, etc.) from voice or text data.

[1070] The "similarity search algorithm" is an algorithm for identifying highly relevant data from within the FAQ database based on extracted keywords.

[1071] The "FAQ database" is a database that compiles frequently asked questions and their answers.

[1072] "Generative AI" is a general term for algorithms and software that generate natural, conversational answers based on given data (e.g., FAQs or sentiment analysis results).

[1073] "Operator feedback" refers to the operator's evaluation of the system's answers, and is the data used to retrain and tune the model.

[1074] "System improvement" refers to the process of improving the accuracy of automatic speech recognition models, natural language processing models, emotion engines, and generative AI based on collected feedback.

[1075] "Means of real-time notification" refers to a system that immediately transmits information to the server and related terminals when there are changes in the FAQ database.

[1076] This invention is a system that combines automatic speech recognition (ASR) technology, natural language processing (NLP) technology, generative AI technology, and emotion recognition technology to provide customer service that is appropriate to the user's emotions. This system is mainly composed of a server and a terminal, and operates as follows.

[1077] The device receives the user's voice inquiry. For example, consider the case where the user says, "Please tell me how to record this TV program." The device records the user's voice using a built-in microphone and sends the voice data in digital format to a server. Voice data is typically sent using an HTTP request over the Internet.

[1078] The server uses an automatic speech recognition (ASR) model on the received voice data to convert it into text data. This process can be performed using a voice recognition service such as Google Cloud Speech-to-Text API. For example, the server can generate text data such as "Please tell me how to record this TV program" from the voice data.

[1079] The server then uses a natural language processing (NLP) model to extract user intent and key keywords from the text data. This process can be performed using natural language processing libraries such as spaCy or BERT. For example, the NLP model might extract the key keyword "how to record TV" from the text data.

[1080] The server then activates an emotion engine to analyze the user's emotional state from the voice and text data. This can be done using emotion analysis tools such as IBM Watson Tone Analyzer. For example, the server can determine from the voice data that the user is confused.

[1081] The server uses the extracted keywords and the results of sentiment analysis to search the FAQ database. It uses a similarity search algorithm to identify the most relevant FAQs from among many. This process can be performed using a search engine such as Elasticsearch. For example, the server identifies FAQs about "how to record TV."

[1082] The identified FAQs are passed to a generative AI, which uses the FAQs and sentiment analysis results to generate natural, conversational answers that fit the user's emotions. This process can use a generative AI model such as OpenAI's GPT-3. For example, a generated answer might be, "Don't worry. Just press the record button and select the program you want to record."

[1083] The generated answer is then sent back to the device, which can then display it as text or read it aloud using a speech synthesis engine. For example, the device may display it to the user as a text message and read it aloud using speech synthesis.

[1084] The server also collects feedback from operators and uses it to improve the accuracy of the automatic speech recognition model, natural language processing model, emotion engine, and generative AI. This feedback is used to retrain the models and tune parameters. Furthermore, the server monitors changes to the FAQ database in real time and notifies operators and related devices if any changes are detected.

[1085] As a concrete example, the following prompt sentences can be input to a generative AI model:

[1086] Example prompt: "How do I record this TV show?"

[1087] The system of the present invention can respond to customer inquiries quickly and accurately and provide answers that take into consideration the user's feelings, thereby improving customer satisfaction and reducing the burden on operators.

[1088] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1089] Step 1:

[1090] The device acquires the user's voice. When the user speaks into the microphone, such as "Please tell me how to record this TV program," the device records the voice and generates audio data. The input is the user's speech, and the output is digital audio data.

[1091] Step 2:

[1092] The device sends the recorded audio data to the server. This transmission is performed via an HTTP request over the Internet. The input is audio data, and the output is the transmission of audio data to the server.

[1093] Step 3:

[1094] The server converts the received voice data into text data using an automatic speech recognition (ASR) model. For this conversion, it uses, for example, the Google Cloud Speech-to-Text API. The input is voice data, and the output is text data such as "Please tell me how to record this TV."

[1095] Step 4:

[1096] The server then passes the converted text data to a natural language processing (NLP) model to extract key intent and keywords. Examples of NLP models used for this process include spaCy and BERT. The input is text data, and the output is extracted keywords, such as "how to record TV."

[1097] Step 5:

[1098] The server passes the voice data to an emotion engine to analyze the user's emotional state. For this processing, IBM Watson Tone Analyzer is used, for example. The input is the voice data, and the output is the user's emotional state (e.g., confusion or relief).

[1099] Step 6:

[1100] The server uses the extracted keywords and the results of the sentiment analysis to search the FAQ database. This search is performed using a search engine such as Elasticsearch. The input is the keywords and the results of the sentiment analysis, and the output is a list of relevant FAQs about "how to record TV."

[1101] Step 7:

[1102] The server passes the FAQs identified by the search to a generative AI, which generates a natural, conversational answer that matches the user's emotions. This process uses a generative AI model such as OpenAI's GPT-3. The input is the FAQ and the results of sentiment analysis, and the output is a response such as, "Don't worry. Just press the record button and select the program you want to record."

[1103] Step 8:

[1104] The server generates a response and sends it to the device, which displays it as text on the screen or reads it aloud using a speech synthesis engine. The input is the response, and the output is a response that the user can see or hear.

[1105] Step 9:

[1106] The server collects feedback from operators and uses it to improve the system. Operators evaluate the quality and appropriateness of the answers provided, and the results are used to retrain and tune the model. The input is operator feedback, and the output is an improved model.

[1107] Step 10:

[1108] The server monitors changes to the FAQ database in real time and notifies operators and terminals if any changes are detected. This allows the system to always provide the latest information. The input is new or changed FAQ data, and the output is real-time notifications.

[1109] (Application example 2)

[1110] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1111] Conventional customer support systems are capable of voice recognition and text conversion, but they do not adequately respond to the user's emotions, resulting in low customer satisfaction. Furthermore, the generated answers do not always take the user's emotions into consideration, resulting in inappropriate responses. Furthermore, they lack a mechanism for notifying users of changes to the FAQ database in real time, making it difficult to always provide the latest information to customers.

[1112] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data, means for extracting intent and keywords from the text data using a natural language processing model, and means for recognizing the user's emotional state from the voice data using an emotion engine. This makes it possible to generate appropriate answers that take the user's emotions into consideration and to respond to customers quickly and accurately. In addition, by notifying changes to the FAQ database in real time, it is possible to always provide the latest information.

[1113] "Audio data" refers to data of an audio signal provided by a user through an audio input device such as a microphone.

[1114] "Text data" is data in a format in which voice data is converted into character information.

[1115] A "natural language processing model" is an algorithm or machine learning model for analyzing semantic information and context from text data.

[1116] A "similarity search algorithm" is an algorithm for searching a database for related information based on specific keywords or phrases.

[1117] "Generative AI" is an artificial intelligence model that generates natural, conversational text based on input data.

[1118] An "emotion engine" is an algorithm or system that analyzes and recognizes a user's emotional state from voice and text data.

[1119] "Operator feedback" refers to the operator's evaluation and comments regarding the system's accuracy and response content.

[1120] The "FAQ database" is a database that collects frequently asked questions and their answers.

[1121] "Means of real-time notification" refers to a function that instantly notifies relevant parties or systems of database update information, etc.

[1122] The "means for providing an answer to the customer" is a function for presenting the generated answer to the user in text or audio format.

[1123] "User's emotional state" refers to the feelings or mood of the user when they input voice or text.

[1124] "Intent and keywords" refer to verbs and nouns that are at the core of the context and are extracted from the text data entered by the user.

[1125] A system for realizing this application example operates with the following hardware and software configuration.

[1126] Hardware and software used

[1127] Hardware: Microphone (for voice input), server, terminal

[1128] Software: SpeechRecognition library, spaCy, Transformers (Hugging Face), TextBlob

[1129] System Operation Details

[1130] 1. Voice Recognition

[1131] The user speaks their query through the microphone. The device's microphone records the voice data, and the SpeechRecognition library is used to capture the data. For example, consider a user saying, "Please tell me how to record this TV." This voice is converted into text data in real time through Google's speech recognition API.

[1132] 2. Natural Language Processing

[1133] The server analyzes the acquired text data using a natural language processing model (spaCy). Here, intent (verbs) and keywords (nouns) are extracted from the text data. For example, keywords such as "TV recording method" are extracted.

[1134] 3. Sentiment analysis

[1135] We use TextBlob to recognize the user's emotional state from the audio data, which determines whether the emotional state is positive or negative, and this information is then reflected in the generated answer.

[1136] 4. Answer generation

[1137] The server combines the extracted intent, keywords, and sentiment analysis results and passes them to a generative AI (using the Transformers library) as a prompt. The AI ​​then adjusts the format and content of the response appropriately based on the user's emotional state. For example, if the server determines that the user is confused, it generates a response such as, "Don't worry, just press the record button and select the program you want to record."

[1138] 5. Providing answers

[1139] The generated answer is then sent back to the device, which displays the answer as text or reads it aloud, allowing the user to obtain quick and accurate information.

[1140] Specific prompt examples

[1141] Below are some examples of prompts passed to the generative AI:

[1142] text

[1143] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[1144] This system is expected to improve customer satisfaction by providing appropriate answers to customers through a consistent process from voice input to emotion recognition, natural language processing, and answer generation.

[1145] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1146] Step 1:

[1147] The user makes a voice inquiry. The voice data is input to the device through the microphone. The device uses the SpeechRecognition library to record the voice data. The recorded voice data is processed as input data.

[1148] Step 2:

[1149] The device converts the recorded voice data into text data through Google's speech recognition API. The input data is voice data, and the output data is text data. This allows the voice content to be expressed in text format.

[1150] Step 3:

[1151] The server receives the text data and uses a natural language processing model (spaCy) to extract intent and keywords. At this stage, the text data is the input and the intent and keywords are the output. Specifically, the text data is grammatically analyzed to identify verbs and nouns.

[1152] Step 4:

[1153] The server uses TextBlob to analyze the user's emotional state from text data. The input data is text data, and the output data is the emotional state (e.g., positive, negative, etc.). The specific operation here is to determine the emotion from the context and tone of the text.

[1154] Step 5:

[1155] The server generates a prompt based on the extracted intent, keywords, and emotional state, and inputs it into a generative AI (Transformers). The input data is the prompt, and the output data is the answer text. The specific operation is for the generative AI to generate an appropriate answer for the input prompt.

[1156] Examples of prompts:

[1157] text

[1158] The user's emotions are negative. Please generate answers for the following keywords: TV recording method settings

[1159] Step 6:

[1160] The generated answer text is sent from the server to the terminal. The input data is the answer text, and the output data is the answer that is displayed or read aloud. The terminal provides the answer to the user, providing a highly satisfactory response.

[1161] This system automates the entire process from receiving an inquiry to providing a response, while also providing a response that takes emotions into consideration.

[1162] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1163] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1164] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1165] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1166] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1167] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1168] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1169] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1170] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1171] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1172] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1173] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1174] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1175] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1176] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1177] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1178] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1179] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1180] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1181] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1182] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1183] The following is further disclosed regarding the above embodiment.

[1184] (Claim 1)

[1185] means for converting voice data into text data;

[1186] A means for extracting intent and keywords from text data using a natural language processing model;

[1187] a means for searching related FAQ data using a similarity search algorithm;

[1188] A means of generating natural dialogue-style answers using generative AI;

[1189] a means of collecting operator feedback and using it to improve the system;

[1190] A means of real-time notification of changes to the FAQ database,

[1191] a means of providing answers to customers;

[1192] A system including:

[1193] (Claim 2)

[1194] 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

[1195] (Claim 3)

[1196] The system of claim 1, wherein the generative AI generates answers in a natural dialogue format.

[1197] "Example 1"

[1198] (Claim 1)

[1199] means for receiving a user's query by voice;

[1200] means for converting voice data into text data;

[1201] A means for extracting intent and keywords from text data using a natural language processing model;

[1202] a means for searching related FAQ data using a similarity search algorithm;

[1203] A means for generating natural dialogue-style answers using generative artificial intelligence;

[1204] a means of collecting operator feedback and using it to improve the system;

[1205] A means of real-time notification of changes to the FAQ database,

[1206] a means for transmitting the answer to the terminal and providing it to the customer by display or audio;

[1207] A system including:

[1208] (Claim 2)

[1209] 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

[1210] (Claim 3)

[1211] 2. The system of claim 1, wherein the generative artificial intelligence generates answers in a natural, interactive format.

[1212] "Application Example 1"

[1213] (Claim 1)

[1214] means for converting voice data into text data;

[1215] A means for extracting intent and keywords from text data using a natural language processing model;

[1216] a means for searching related FAQ data using a similarity search algorithm;

[1217] A means of generating natural dialogue-style answers using generative AI;

[1218] a means of collecting operator feedback and using it to improve the system;

[1219] A means of real-time notification of changes to the FAQ database,

[1220] a means of providing answers to customers;

[1221] A voice input means;

[1222] a means for receiving a customer question as voice input using a smart device;

[1223] A system including:

[1224] (Claim 2)

[1225] 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

[1226] (Claim 3)

[1227] The system of claim 1, wherein the generative AI generates answers in a natural dialogue format.

[1228] "Example 2: Combining Emotion Engines"

[1229] (Claim 1)

[1230] means for converting voice data into text data;

[1231] A means for extracting intent and keywords from text data using a natural language processing model;

[1232] means for analyzing the emotional state of a user from the voice data;

[1233] a means for searching related FAQ data using a similarity search algorithm;

[1234] A means of generating natural dialogue-style responses that match the user's emotions using generative AI;

[1235] a means of collecting operator feedback and using it to improve the system;

[1236] A means of real-time notification of changes to the FAQ database,

[1237] a means of providing answers to customers;

[1238] A system including:

[1239] (Claim 2)

[1240] 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

[1241] (Claim 3)

[1242] The system of claim 1, wherein the generative AI generates answers in a natural dialogue format that is suited to the user's emotions.

[1243] "Application example 2 when combining emotion engines"

[1244] (Claim 1)

[1245] means for converting voice data into text data;

[1246] A means for extracting intent and keywords from text data using a natural language processing model;

[1247] a means for searching related FAQ data using a similarity search algorithm;

[1248] A means of generating natural dialogue-style answers using generative AI;

[1249] means for recognizing an emotional state of a user from the speech data using an emotion engine;

[1250] A means for generating an answer based on the sentiment analysis results;

[1251] a means of collecting operator feedback and using it to improve the system;

[1252] A means of real-time notification of changes to the FAQ database,

[1253] a means of providing answers to customers;

[1254] A system including:

[1255] (Claim 2)

[1256] 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

[1257] (Claim 3)

[1258] The system of claim 1, wherein the generative AI generates answers in a natural dialogue format, taking into account the results of sentiment analysis. [Explanation of symbols]

[1259] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for converting voice data into text data; A means for extracting intent and keywords from text data using a natural language processing model; a means for searching related FAQ data using a similarity search algorithm; A means of generating natural dialogue-style answers using generative AI; a means of collecting operator feedback and using it to improve the system; A means of real-time notification of changes to the FAQ database, a means of providing answers to customers; A system including:

2. 10. The system of claim 1, wherein the system converts speech data into text data using an automatic speech recognition model.

3. The system according to claim 1, wherein the generative AI generates answers in a natural dialogue format.

Images