system

Abstract

We provide the system. [Solution] A means for receiving audio data from a customer and converting said audio data into text data, An analysis means for analyzing the aforementioned text data and understanding the content of the inquiry, A search means for searching a database related to the analysis results obtained based on the aforementioned analysis means, A generation means that generates an appropriate answer based on the results of the search means, A display means for displaying the response generated by the generation means on the terminal of the counter staff member, A recording means for recording the results of the response by the counter staff member, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At a window such as a service desk, it is required to respond to customer inquiries quickly and accurately. However, in the conventional response, there is a problem that it is difficult to analyze the customer's inquiry in real time on the spot and immediately give an appropriate response. As a result, there are delays and misunderstandings in the response, which has been a factor in reducing customer satisfaction. To improve this, a mechanism is needed to quickly grasp the content of the conversation and immediately provide the necessary information to the window staff.

Means for Solving the Problems

[0005] This invention provides a means for receiving voice data from customers and converting it into text data. Furthermore, it analyzes the converted text data to understand the content of the inquiry and searches a relevant database based on the analysis results. Based on the search results, it generates an appropriate response and displays it on the terminal of the service counter staff, enabling a quick response. This improves the accuracy and speed of service counter operations and increases customer satisfaction. In addition, the response results are automatically recorded as incident information and used as data to further improve the quality of service.

[0006] "Voice data" refers to audio information, such as customer inquiries, recorded in digital format.

[0007] "Text data" refers to digital data obtained by analyzing audio data and converting it into textual information.

[0008] "Analysis tools" refer to functions that understand text data and extract the intent of inquiries and important information based on its content.

[0009] A "database" is a collection of information that systematically stores past history and related information concerning queries, and is organized in a searchable format.

[0010] A "search method" is a function that extracts relevant information from the database based on the analysis results.

[0011] "Generation means" refers to the function of constructing appropriate responses or answers based on information obtained through search means.

[0012] "Display means" refers to a function that visually displays the generated responses and information on a terminal so that the counter staff can confirm them.

[0013] "Recording means" refers to a function that saves the relevant response results as incident information after the counter staff has handled the situation.

[0014] "Incident information" refers to data that structures and records events that occurred during customer service and the details of the responses taken. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] As an embodiment of the present invention, an autonomous predictive AI agent system for a service desk is described. This system includes a process of converting voice data into text and analyzing it in real time in order to respond quickly and accurately to customer inquiries. Its specific operation is shown below.

[0037] First, when a user becomes a customer and initiates a phone inquiry to the service desk, the server receives the call. The received audio data is immediately converted into text data by the server. This conversion process prepares the audio information into a format that can be easily analyzed as text.

[0038] Next, the server uses a generative AI to analyze the transcribed conversation. In this analysis, the server aims to extract the inquiry content and important keywords from the conversation and understand the intent of the inquiry. At this time, it performs a deeper analysis based on relevant past inquiry history and information in the knowledge base.

[0039] Once the analysis is complete, the server searches the database based on the results to find similar cases and appropriate solutions. This process comprehensively considers past response examples and related information to formulate appropriate countermeasures.

[0040] Subsequently, the server uses AI to generate the optimal response to provide to the customer based on this information. This generated response is concise and easy to understand, making it effective when customer service staff guide users.

[0041] Finally, the server displays the generated response on the employee's terminal, helping them to take appropriate action immediately with the user. After the response is complete, the server automatically records the details of the response through the terminal's operation, making it available later as incident information.

[0042] For example, if a user requests to know the details of their invoice, the server transcribes this into text, searches the knowledge base, and provides the most recent invoice information. The staff member can then view the information on their terminal and immediately inform the user of the invoice amount and payment deadline. This entire process improves the efficiency of customer service and increases customer satisfaction.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] A user contacts the service desk by phone. The server receives the call and obtains the audio data. Since the audio data is difficult to analyze directly, the server uses speech recognition technology to convert it into text data.

[0046] Step 2:

[0047] The server inputs the converted text data into the generating AI. The generating AI analyzes the text content and extracts important keywords and the intent of the inquiry. This analysis process identifies which category the inquiry belongs to.

[0048] Step 3:

[0049] Based on the analysis results, the server searches its internal database. This includes past query history and relevant information within the knowledge base. The server extracts the most relevant data and prepares it for the next stage of answer generation.

[0050] Step 4:

[0051] The generation AI generates appropriate answers and recommendations based on the searched data. This process also considers predictive answers to questions the customer might ask next. The generated answers are concise and clearly structured.

[0052] Step 5:

[0053] The server sends the generated response to the counter staff member's terminal in real time. The terminal displays this information on the screen, allowing the staff member to respond to the customer immediately.

[0054] Step 6:

[0055] After the user interaction is complete, the server automatically records the details of the interaction in a database via the user's terminal. This incident information includes the inquiry, details of the interaction, and the outcome. This data is useful for future interactions and improvements.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Responding quickly and accurately to customer inquiries is crucial for improving customer satisfaction and operational efficiency. However, traditional methods of handling inquiries were time-consuming, requiring time to understand the content of the inquiry, refer to past cases, and develop appropriate solutions, making efficient responses difficult. Therefore, there is a need for techniques that accurately grasp customer intent and provide the most suitable answer quickly based on past experience.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for receiving voice information and converting said voice information into text information; means for analyzing said text information to understand the content of the inquiry and extract important words; and means for further analyzing the analysis results obtained by the analysis means based on past response records and a knowledge database and searching for similar past cases. This enables the generation of appropriate suggestions based on customer inquiries and quick customer service responses.

[0061] "Auditory information" refers to the form of sound transmitted through sound waves, and specifically to data recorded as spoken language.

[0062] "Textual information" refers to data in written form converted from audio information, and is a string of characters processed by a computer.

[0063] "Analysis means" refers to a process or device for understanding meaning from textual information and extracting important words or inquiry content.

[0064] A "knowledge database" refers to a collection of data accumulated by an organization, including information and past response records, and is a source of information used for problem-solving.

[0065] "Recorded information" refers to data that is stored as a result of customer interactions or data obtained during those interactions, and is used for future reference and analysis.

[0066] "Event information" refers to a part of the recorded information, specifically detailed data about the events that occurred and the results of the responses.

[0067] This invention relates to an autonomous predictive AI agent system for responding quickly and accurately to customer inquiries. Specific embodiments thereof are described below.

[0068] First, the user uses a communication device to call the service desk. The server receives the voice information via the communication network. This voice information is instantly converted into text information using speech recognition technology. A speech recognition API is one example of the specific software that can be used for this purpose.

[0069] Next, the server uses a generative AI model to analyze the textual information. This extracts and accurately understands the user's inquiry and important words. Natural language processing models are commonly used as generative AI models. This analysis process also references past interaction records and knowledge databases.

[0070] Furthermore, the server performs a database search based on the analysis results. This is to identify similar past cases and appropriate countermeasures. Past data is stored in a knowledge database, allowing for quick searching and utilization.

[0071] Subsequently, the server constructs the optimal solution based on the generated information and displays it on the terminal. This solution is visually easy to understand so that the service representative can guide the user appropriately.

[0072] Finally, after the interaction is complete, the server automatically saves the details of the interaction as record information via the terminal. This accumulates data that can be used for handling future inquiries.

[0073] A concrete example is when a user inquires, "I want to know the details of my invoice." In this case, the server converts the voice information into text information and analyzes it using a generation AI model to quickly search for relevant information from the knowledge base. The information displayed on the terminal can then be confirmed by the staff member, who can immediately provide the user with the details of the invoice.

[0074] An example of a prompt message is, "Generate the best response when a user asks, 'I want to know the details of the invoice.'" This enables effective responses based on the invention.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] When a user calls the service desk using a communication device, the server receives the audio information. This received audio information is the input data. The server converts the audio into text using speech recognition technology. In this process, a speech recognition API is used, and the content of the conversation is output as text by converting the sound waveform into phonemes. Specifically, the server processes the audio in real time and generates text information with minimal error.

[0078] Step 2:

[0079] The server uses a generative AI model to analyze the textual information generated in Step 1. The input for this analysis process is textual information. The server performs natural language processing to understand the content of the textual information and extracts important words and query content. The output is a list of the analyzed query content and important words. Specifically, the server analyzes complex contexts and compares them with relevant historical records.

[0080] Step 3:

[0081] The server searches the database based on the analysis results. The input is the analysis results obtained in step 2. The server quickly searches past cases and related knowledge to identify similar past cases and solutions. The output is a list of highly relevant cases. Specifically, the server uses an efficient search algorithm to collect the most relevant information in a short amount of time.

[0082] Step 4:

[0083] The server utilizes generation AI to generate the optimal solution based on the information obtained in step 3 and displays it on the terminal. The input is the result of a database search. The server integrates this information to construct an easy-to-understand answer, which is then visually displayed on the terminal as output. Specifically, the terminal arranges the generated answer in an easily viewable format, allowing staff to respond to users quickly.

[0084] Step 5:

[0085] Once the response is complete, the server automatically saves the response results as record information via the terminal. The input is a detailed description of the response performed by the person in charge on the terminal. The server saves this as record information in the database to prepare for future inquiries. The output is a detailed response record. Specifically, the server quickly organizes the record content and makes it quickly searchable as needed.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] In modern customer service, prompt and accurate customer responses are essential, but traditional systems struggle with real-time processing of voice data and providing appropriate information. Furthermore, there is a need for a method that allows service providers to immediately obtain necessary information and respond efficiently. Solving these challenges is crucial to improving customer satisfaction and increasing operational efficiency.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes means for acquiring voice signals from customers and converting said voice signals into text data, an analysis function for analyzing the text data and understanding the content of the inquiry, and a search function for searching for information related to the analysis results acquired based on the analysis function. This enables the server to immediately present the most appropriate information in response to customer inquiries, and allows staff to respond quickly while visually confirming the information using augmented reality devices.

[0091] An "audio signal" is an electrical signal that represents audio information, and it is the foundation for converting human voices into digital data.

[0092] "Character data" refers to data that represents audio signals as a string of characters, and is information that can be processed by a computer as text.

[0093] The "analysis function" is a function that analyzes text data to understand its content and intent, and plays a role in identifying the intent of the inquiry.

[0094] The "search function" is a function that, based on the information identified by the analysis function, searches for relevant databases and information sources and retrieves the necessary data.

[0095] "Information" refers to the knowledge and data necessary to process customer inquiries, which are identified through analytical and search functions.

[0096] An "augmented reality device" is a device that overlays digital information onto the real world environment, and is intended to provide visual information.

[0097] To realize this invention, it is necessary to configure a system using a server, a user's device (terminal), and an augmented reality device. A specific embodiment of this system is shown below.

[0098] The server first receives an audio signal from the user. This audio signal is sent to the server, for example, via a telephone or voice input device. The server then uses a speech recognition library (e.g., Google® Speech-to-Text API) to convert the audio signal into text data.

[0099] Next, the server uses a generative AI model (e.g., OpenAI® GPT-4®) to analyze the text data and understand the content and intent of the inquiry. This analysis accurately extracts customer requests and related keywords.

[0100] Based on the analysis results, the server utilizes its search function to retrieve information. This allows it to quickly obtain necessary information from relevant past cases and knowledge bases. The retrieved information is then generated as an appropriate response, and the server prepares this information to be presented to the user's device.

[0101] Here, it is possible to display information within the employee's field of vision via an augmented reality device (e.g., smart glasses). The displayed information allows the employee to quickly review it and provides support for taking appropriate action with the customer.

[0102] For example, if a customer requests to "check their recent payment history," the server analyzes this request and retrieves the appropriate payment history information from the knowledge base. This information is then displayed to the representative via an augmented reality device, allowing the representative to immediately provide the information to the customer.

[0103] An example of a prompt message is, "After converting the audio data to text, retrieve the latest payment history information from the knowledge base and present it to the representative."

[0104] This system will enable faster and more accurate customer service, thereby increasing customer satisfaction.

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The server receives an audio signal from the user. The audio, acquired via the audio input device, is then transferred to the server. The input is an audio signal, and the output is that audio signal stored on the server in digital format.

[0108] Step 2:

[0109] The server uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data. The input is the audio signal obtained in step 1, and the output is text data suitable for analysis. Here, data processing is performed to convert the audio signal into a string using an algorithm.

[0110] Step 3:

[0111] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data. The goal is to understand the content and intent of the inquiry and extract important keywords. The input is the text data generated in step 2, and the output is the analyzed information and related keywords. The AI ​​model performs natural language processing to analyze the text.

[0112] Step 4:

[0113] The server performs information retrieval based on the analysis results. It uses search functions to find relevant information from knowledge bases and databases. The input is the analysis results obtained in step 3, and the output is a collection of required information. Search algorithms are utilized, performing text matching and extracting related documents.

[0114] Step 5:

[0115] Based on the information acquired by the server, information is generated for the user to review. A prompt message is generated, and a response is created. The input is the information obtained in step 4, and the output is a displayable response message. A generation AI is used to naturally combine the information and generate the response.

[0116] Step 6:

[0117] The terminal displays the generated response on the augmented reality device. The server sends data to the display device, which the terminal can then verify. The input is the response message generated in step 5, and the output is the information visually displayed on the augmented reality device. Real-time data transfer and display are performed as part of the operation.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] As an embodiment of the present invention, an autonomous predictive AI agent system incorporating an emotion engine that analyzes voice data from customers and identifies emotions is described. This system is designed to provide more accurate and emotionally empathetic responses to the customer service staff being assisted.

[0120] When a user makes an inquiry via telephone, the server receives the call and acquires the audio data. The server uses speech recognition technology to convert the acquired audio data into text data. This conversion ensures that the conversation is recorded in text format, allowing for smoother subsequent processing.

[0121] The server provides text data to the generating AI, which analyzes the inquiry. During this process, the emotion engine uses voice data to identify the user's emotions. For example, it analyzes parameters such as voice tone and speaking speed to determine whether the user is angry, confused, or calm. This emotion information, along with the inquiry category, becomes part of the analysis data.

[0122] Once the analysis is complete, the server uses the analysis results to search the database for the necessary information. This search includes past query history, similar cases, and relevant knowledge base information. The server collects this data so that the AI ​​can generate the most appropriate answer or recommended action.

[0123] The generation AI generates the optimal answer based on all the information it has acquired. During this process, the answer is adjusted according to the user's emotions, as identified by the emotion engine. For example, if the user is angry, it will generate a polite and calm-toned answer; if the user is confused, it will construct a more detailed and clear answer.

[0124] Finally, the server displays the generated response on the customer service representative's terminal. The representative can then quickly provide an appropriate response to the user while viewing the terminal. After the response is completed, the response details and emotional information confirmed on the terminal are automatically recorded by the server and organized as incident information. This entire process makes customer service more sophisticated and humane, contributing to improved customer satisfaction. For example, if a user angrily inquires about a billing error, the emotion engine can identify the anger, and the server can suggest a calmer response appropriate to that emotion.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] A user contacts the service desk by phone. The server receives the call and retrieves the audio data.

[0128] Step 2:

[0129] The server uses speech recognition technology to convert the acquired audio data into text data. This makes the conversation content analyzable in written form.

[0130] Step 3:

[0131] The server provides the converted text data to the generating AI, which then begins analysis. The generating AI extracts keywords from the text to understand the content of the inquiry.

[0132] Step 4:

[0133] The server simultaneously sends audio data to the emotion engine, which identifies the user's emotional state based on their voice tone, speaking speed, intonation, etc. The emotion engine then adds this to the analysis data.

[0134] Step 5:

[0135] The server searches the database based on the analyzed query content and sentiment information. It collects relevant query history and knowledge base information.

[0136] Step 6:

[0137] The generation AI generates appropriate responses based on database search results and sentiment information. For example, if the user is angry, a response in a calm tone will be generated.

[0138] Step 7:

[0139] The server sends the generated response and emotional considerations to the customer service representative's terminal for display. This allows the representative to respond to the user immediately.

[0140] Step 8:

[0141] After the response is complete, the server automatically records the details of the response and emotional information via the terminal and stores it in the database as incident information. This information will be used to improve future operations.

[0142] (Example 2)

[0143] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0144] Conventional information processing systems have faced challenges in adequately addressing customer inquiries emotionally, making it difficult to improve customer satisfaction. Furthermore, in the analysis of inquiry content, emotional information is often not considered, resulting in cases where optimal responses cannot be provided. This situation necessitates that customer service representatives provide prompt and accurate responses that take emotions into account.

[0145] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0146] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information and understanding the content of the inquiry, and means for searching for information resources related to the analysis results. This makes it possible to perform analysis that takes customer emotions into account and provide appropriate information.

[0147] "Audio information" refers to information that represents audio acquired from customers in a digital format.

[0148] "Text information" refers to information in string format obtained by converting audio information into a parseable form.

[0149] "Analysis means" refers to means of understanding text information and processing it to classify and convert the content of inquiries into knowledge.

[0150] A "search tool" is a means of referencing and obtaining relevant information resources based on the analysis results.

[0151] "Generation means" refers to means for constructing an appropriate response based on information obtained by search means.

[0152] "Emotion identification means" refers to a method for analyzing the characteristics of voice information to determine the emotional state of a customer.

[0153] "Display means" refers to means for visually showing the response obtained by the generation means to the terminal.

[0154] "Recording means" refers to a means of saving the results of the counter staff's interactions and related information, and organizing them in a way that allows for later reference.

[0155] "Information resources" refer to various sources of information, such as databases and knowledge bases, used for analyzing inquiries and generating responses.

[0156] This embodiment of the invention describes a system that receives customer inquiries via voice information and provides responses that take into account the user's emotions. A server receives calls from customers and acquires voice information. The acquired voice information is converted into text information using speech recognition technology. In this process, general speech recognition software is used, and for example, an open-source speech recognition library can be applied.

[0157] Next, the server analyzes the text information. This analysis utilizes a generative AI model to support the understanding of the inquiry. A general natural language processing framework can be used as the specific generative AI model. Furthermore, emotion recognition is used to identify the customer's emotions from the audio information. Based on this, the server determines whether the user's emotion falls under "anger," "joy," "sadness," or "calmness."

[0158] To retrieve information resources related to a user's query, the server references an internal database based on the analysis results to obtain the appropriate information. This typically involves a general database system, such as past query history and related knowledge bases.

[0159] Based on the generated information, a generative AI model constructs a response. This model also considers emotions identified by emotion recognition tools, and adjusts the tone to suit the user. The generated response is displayed on the terminal and used as a basis for service staff to provide appropriate assistance.

[0160] As an example, consider a case where a user angrily inquires that there is an error in their bill. The emotion recognition means efficiently detects anger, and the server uses this result to generate a calm and composed response using a generative AI model. In this way, high-quality responses that take emotions into consideration are possible, which is expected to contribute to improved customer satisfaction.

[0161] (Example of prompts for a generative AI model)

[0162] "Please generate the best possible response to inquiries regarding service interruptions, taking into account user sentiment."

[0163] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0164] Step 1:

[0165] When a user makes an inquiry via telephone, the server receives the call. The input is the user's voice data, which the server captures and manages. During this process, the data is temporarily stored in an audio file format. The output is the stored audio data.

[0166] Step 2:

[0167] The server converts the received audio data into text data. It analyzes the audio using speech recognition technology and converts it into a string format. Specifically, by using a speech recognition API, the input audio data is output as text data.

[0168] Step 3:

[0169] The server provides text data to a generating AI model, which then analyzes the query. The input is the generated text data. The AI ​​model uses natural language processing algorithms to understand the context and extracts the intent and content of the query as part of the analysis. The output is the analysis result.

[0170] Step 4:

[0171] The server uses the analysis results to identify the user's emotions using emotion recognition tools. The input is voice data and its feature extraction results. By analyzing the pitch, tone, and speaking speed of the voice, it generates emotion information as output. This information includes emotion categories such as "anger," "joy," and "calmness."

[0172] Step 5:

[0173] The server uses the analysis results and sentiment information to search the company's database and collect relevant information. The input is the analysis results and sentiment information. By executing database queries, it retrieves the necessary data from past query history and relevant knowledge bases. The output is relevant information based on the search results.

[0174] Step 6:

[0175] The generative AI model generates the optimal response based on collected information and sentiment data. The input consists of information and sentiment data obtained through searches. The generation process uses high-priority data to produce a structured response. The output is a suggested response to the user.

[0176] Step 7:

[0177] The server sends the generated response to the terminal and displays it on the counter staff's screen. The input is the generated response, which is visualized on the terminal. The counter staff refer to this output and take appropriate action in real time.

[0178] Step 8:

[0179] Once the interaction is complete, the interaction details and emotional information recorded on the terminal are sent to the server. The input consists of the interaction result and emotional information. Based on this, the server organizes the data and records it in the database as an incident report. The output is structured incident information.

[0180] (Application Example 2)

[0181] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0182] Traditional customer service systems struggle to accurately understand customers' emotional states and optimize responses. This can lead to systems generating mechanical responses and decreased customer satisfaction. Furthermore, in sensitive or urgent situations, appropriate responses may not be provided promptly. This hinders the building of trust with customers and results in lost business opportunities for companies.

[0183] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0184] In this invention, the server includes emotion identification means for identifying emotional states from voice information, analysis means for analyzing text information and understanding the content of inquiries, and adjustment means for adjusting the response according to the emotional state identified by the emotion identification means. This makes it possible to accurately grasp the customer's emotional state and provide an optimal response accordingly.

[0185] "Audio information" refers to linguistic information spoken by customers, captured as acoustic signals.

[0186] "Textual information" refers to data in text format obtained by converting audio information.

[0187] "Analysis means" refers to a device or program that has the function of analyzing textual information to understand the content of an inquiry.

[0188] A "search means" is a system that has the function of finding relevant recording media based on the analysis results obtained by the analysis means.

[0189] "Generation means" refers to a mechanism or software that has the function of creating an optimal response in accordance with the results of the search means.

[0190] "Display means" refers to a means for visually displaying the generated response on the counter staff member's device.

[0191] "Recording means" refers to a method or mechanism for long-term storage of the results of a customer service representative's interactions.

[0192] An "emotion identification means" is a device that has the function of identifying a customer's emotional state from voice information.

[0193] A "modification mechanism" is a system that has the function of adjusting the content of the response according to the customer's emotional state.

[0194] This invention is an autonomous predictive AI agent system for streamlining customer service. The server receives voice information from customers and converts it into text information using speech recognition software. For speech recognition, for example, the Google Speech Recognition API is used. The converted text information is analyzed by an analysis means to identify the emotional state associated with the inquiry. For the analysis, the Sentiment Analysis pipeline of Transformers, a natural language processing library, is used.

[0195] Based on the user's inquiry, the server searches for relevant information from its storage medium. This storage medium includes past inquiry history and a knowledge base. Based on the retrieved information and the emotional state identified by the emotion recognition system, a generative AI model generates the optimal response. The generated response is displayed on the customer service representative's terminal. This terminal facilitates the rapid and appropriate response to the user.

[0196] Furthermore, the server automatically records the results of the responses by the customer service representatives and saves them as a response history. This recording process ensures that incident information is managed systematically.

[0197] For example, if a user inquires about a security system malfunction, the emotion recognition system will determine that the user is distressed, and the generative AI model will provide a reassuring response. An example of a prompt would be, "In a situation where a customer is anxiously reporting a security issue, please show how to respond appropriately based on the tone the emotion engine perceives." This enables an emotionally sensitive response, contributing to improved customer satisfaction.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The server receives voice information from the user. This information is input into speech recognition software, which converts the voice signal into text information. This conversion process outputs the customer's inquiry as text data. Specifically, the Google Speech Recognition API is used to analyze the voice and convert it into the corresponding string.

[0201] Step 2:

[0202] The server uses the obtained text information as input and employs an analysis tool to identify the content of the inquiry and the user's emotional state. This analysis utilizes Transformers' Sentiment Analysis pipeline to analyze the text information and output the user's emotions. Specifically, it analyzes the word choices and context within the string to determine the emotional label (positive, negative, neutral).

[0203] Step 3:

[0204] The server searches the storage medium based on the user's query content identified through analysis. The storage medium outputs past history and knowledge base information related to the query. Specifically, it executes database queries to extract highly relevant documents and information.

[0205] Step 4:

[0206] The server uses an AI model to generate the optimal response, taking information obtained from the recording medium and the emotional state obtained through emotion recognition as input. The generated response is output as text data in the format most appropriate to the user's situation. In terms of operation, it constructs prompt sentences based on the emotional state and executes a process to generate appropriate response sentences.

[0207] Step 5:

[0208] The terminal displays the generated response sent from the server. The displayed response allows the service representative to quickly provide the user with an appropriate answer. This step involves visually presenting the response text on the display.

[0209] Step 6:

[0210] The server automatically records the interactions performed by the customer service representative using recording mechanisms. The interaction history is organized as incident information and used to improve future interactions. Specifically, it saves the response text, emotional state, and interaction result to a data store.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] As an embodiment of the present invention, an autonomous predictive AI agent system for a service desk is described. This system includes a process of converting voice data into text and analyzing it in real time in order to respond quickly and accurately to customer inquiries. Its specific operation is shown below.

[0228] First, when a user becomes a customer and initiates a phone inquiry to the service desk, the server receives the call. The received audio data is immediately converted into text data by the server. This conversion process prepares the audio information into a format that can be easily analyzed as text.

[0229] Next, the server uses a generative AI to analyze the transcribed conversation. In this analysis, the server aims to extract the inquiry content and important keywords from the conversation and understand the intent of the inquiry. At this time, it performs a deeper analysis based on relevant past inquiry history and information in the knowledge base.

[0230] Once the analysis is complete, the server searches the database based on the results to find similar cases and appropriate solutions. This process comprehensively considers past response examples and related information to formulate appropriate countermeasures.

[0231] Subsequently, the server uses AI to generate the optimal response to provide to the customer based on this information. This generated response is concise and easy to understand, making it effective when customer service staff guide users.

[0232] Finally, the server displays the generated response on the employee's terminal, helping them to take appropriate action immediately with the user. After the response is complete, the server automatically records the details of the response through the terminal's operation, making it available later as incident information.

[0233] For example, if a user requests to know the details of their invoice, the server transcribes this into text, searches the knowledge base, and provides the most recent invoice information. The staff member can then view the information on their terminal and immediately inform the user of the invoice amount and payment deadline. This entire process improves the efficiency of customer service and increases customer satisfaction.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] A user contacts the service desk by phone. The server receives the call and obtains the audio data. Since the audio data is difficult to analyze directly, the server uses speech recognition technology to convert it into text data.

[0237] Step 2:

[0238] The server inputs the converted text data into the generating AI. The generating AI analyzes the text content and extracts important keywords and the intent of the inquiry. This analysis process identifies which category the inquiry belongs to.

[0239] Step 3:

[0240] Based on the analysis results, the server searches its internal database. This includes past query history and relevant information within the knowledge base. The server extracts the most relevant data and prepares it for the next stage of answer generation.

[0241] Step 4:

[0242] The generation AI generates appropriate answers and recommendations based on the searched data. This process also considers predictive answers to questions the customer might ask next. The generated answers are concise and clearly structured.

[0243] Step 5:

[0244] The server sends the generated response to the counter staff member's terminal in real time. The terminal displays this information on the screen, allowing the staff member to respond to the customer immediately.

[0245] Step 6:

[0246] After the user interaction is complete, the server automatically records the details of the interaction in a database via the user's terminal. This incident information includes the inquiry, details of the interaction, and the outcome. This data is useful for future interactions and improvements.

[0247] (Example 1)

[0248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0249] Responding quickly and accurately to customer inquiries is crucial for improving customer satisfaction and operational efficiency. However, traditional methods of handling inquiries were time-consuming, requiring time to understand the content of the inquiry, refer to past cases, and develop appropriate solutions, making efficient responses difficult. Therefore, there is a need for techniques that accurately grasp customer intent and provide the most suitable answer quickly based on past experience.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0251] In this invention, the server includes means for receiving voice information and converting said voice information into text information; means for analyzing said text information to understand the content of the inquiry and extract important words; and means for further analyzing the analysis results obtained by the analysis means based on past response records and a knowledge database and searching for similar past cases. This enables the generation of appropriate suggestions based on customer inquiries and quick customer service responses.

[0252] "Auditory information" refers to the form of sound transmitted through sound waves, and specifically to data recorded as spoken language.

[0253] "Textual information" refers to data in written form converted from audio information, and is a string of characters processed by a computer.

[0254] "Analysis means" refers to a process or device for understanding meaning from textual information and extracting important words or inquiry content.

[0255] A "knowledge database" refers to a collection of data accumulated by an organization, including information and past response records, and is a source of information used for problem-solving.

[0256] "Recorded information" refers to data that is stored as a result of customer interactions or data obtained during those interactions, and is used for future reference and analysis.

[0257] "Event information" refers to a part of the recorded information, specifically detailed data about the events that occurred and the results of the responses.

[0258] This invention relates to an autonomous predictive AI agent system for responding quickly and accurately to customer inquiries. Specific embodiments thereof are described below.

[0259] First, the user uses a communication device to call the service desk. The server receives the voice information via the communication network. This voice information is instantly converted into text information using speech recognition technology. A speech recognition API is one example of the specific software that can be used for this purpose.

[0260] Next, the server uses a generative AI model to analyze the textual information. This extracts and accurately understands the user's inquiry and important words. Natural language processing models are commonly used as generative AI models. This analysis process also references past interaction records and knowledge databases.

[0261] Furthermore, the server performs a database search based on the analysis results. This is to identify similar past cases and appropriate countermeasures. Past data is stored in a knowledge database, allowing for quick searching and utilization.

[0262] Subsequently, the server constructs the optimal solution based on the generated information and displays it on the terminal. This solution is visually easy to understand so that the service representative can guide the user appropriately.

[0263] Finally, after the interaction is complete, the server automatically saves the details of the interaction as record information via the terminal. This accumulates data that can be used for handling future inquiries.

[0264] A concrete example is when a user inquires, "I want to know the details of my invoice." In this case, the server converts the voice information into text information and analyzes it using a generation AI model to quickly search for relevant information from the knowledge base. The information displayed on the terminal can then be confirmed by the staff member, who can immediately provide the user with the details of the invoice.

[0265] An example of a prompt message is, "Generate the best response when a user asks, 'I want to know the details of the invoice.'" This enables effective responses based on the invention.

[0266] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0267] Step 1:

[0268] When a user calls the service desk using a communication device, the server receives the audio information. This received audio information is the input data. The server converts the audio into text using speech recognition technology. In this process, a speech recognition API is used, and the content of the conversation is output as text by converting the sound waveform into phonemes. Specifically, the server processes the audio in real time and generates text information with minimal error.

[0269] Step 2:

[0270] The server uses a generative AI model to analyze the textual information generated in Step 1. The input for this analysis process is textual information. The server performs natural language processing to understand the content of the textual information and extracts important words and query content. The output is a list of the analyzed query content and important words. Specifically, the server analyzes complex contexts and compares them with relevant historical records.

[0271] Step 3:

[0272] The server searches the database based on the analysis results. The input is the analysis results obtained in step 2. The server quickly searches past cases and related knowledge to identify similar past cases and solutions. The output is a list of highly relevant cases. Specifically, the server uses an efficient search algorithm to collect the most relevant information in a short amount of time.

[0273] Step 4:

[0274] The server utilizes generation AI to generate the optimal solution based on the information obtained in step 3 and displays it on the terminal. The input is the result of a database search. The server integrates this information to construct an easy-to-understand answer, which is then visually displayed on the terminal as output. Specifically, the terminal arranges the generated answer in an easily viewable format, allowing staff to respond to users quickly.

[0275] Step 5:

[0276] Once the response is complete, the server automatically saves the response results as record information via the terminal. The input is a detailed description of the response performed by the person in charge on the terminal. The server saves this as record information in the database to prepare for future inquiries. The output is a detailed response record. Specifically, the server quickly organizes the record content and makes it quickly searchable as needed.

[0277] (Application Example 1)

[0278] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0279] In modern customer service, prompt and accurate customer response is required. However, in the conventional system, there is a problem that real-time processing of voice data and appropriate information provision are difficult. In addition, a method that enables the responder to immediately obtain necessary information and make an efficient response is required. It is necessary to solve this problem, improve customer satisfaction, and enhance business efficiency.

[0280] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0281] In this invention, the server includes means for acquiring a voice signal from a customer and converting the voice signal into character data, an analysis function for analyzing the character data and understanding the inquiry content, and a search function for searching for information related to the analysis result obtained based on the analysis function. As a result, it is possible to immediately present optimal information in response to a customer's inquiry and enable the person in charge to respond quickly while visually confirming using an extended reality device.

[0282] The "voice signal" is what represents the information of voice as an electrical signal and is the basis for converting human voice into digital data.

[0283] The "character data" is data that represents a voice signal as a character string and is information that can be processed by a computer as text.

[0284] The "analysis function" is a function for analyzing character data and understanding its content and intention, and plays a role in specifying the intention of the inquiry.

[0285] The "search function" is a function for searching a related database or information source based on the information specified by the analysis function and obtaining necessary data.

[0286] "Information" refers to the knowledge and data necessary to process customer inquiries, and specifically refers to those identified by analysis functions and search functions.

[0287] "Augmented reality device" is a device that overlays and displays digital information on the real-world environment and is for providing visual information.

[0288] To implement this invention, it is necessary to configure a system using a server, a user's device (terminal), and an augmented reality device. The specific embodiments are shown below.

[0289] The server first acquires an audio signal from the user. This audio signal is transmitted to the server through, for example, a phone or a voice input device. Then, the server uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into character data.

[0290] Next, the server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the character data and understand the content and intent of the inquiry. Through this analysis, the customer's requirements and related keywords are accurately extracted.

[0291] Based on the analysis results, the server utilizes a search function for retrieving information. Thereby, necessary information can be quickly obtained from relevant past cases and knowledge bases. The acquired information is generated as an appropriate response, and the server prepares to present this information to the user's device.

[0292] Here, it is possible to display information in the field of vision of the person in charge through an augmented reality device (e.g., smart glasses). The displayed information is quickly confirmed by the person in charge and provides support for taking appropriate actions towards customers.

[0293] For example, if a customer requests to "check their recent payment history," the server analyzes this request and retrieves the appropriate payment history information from the knowledge base. This information is then displayed to the representative via an augmented reality device, allowing the representative to immediately provide the information to the customer.

[0294] An example of a prompt message is, "After converting the audio data to text, retrieve the latest payment history information from the knowledge base and present it to the representative."

[0295] This system will enable faster and more accurate customer service, thereby increasing customer satisfaction.

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The server receives an audio signal from the user. The audio, acquired via the audio input device, is then transferred to the server. The input is an audio signal, and the output is that audio signal stored on the server in digital format.

[0299] Step 2:

[0300] The server uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data. The input is the audio signal obtained in step 1, and the output is text data suitable for analysis. Here, data processing is performed to convert the audio signal into a string using an algorithm.

[0301] Step 3:

[0302] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data. The purpose is to understand the content and intent of the inquiry and extract important keywords. The input is the text data generated in Step 2, and the output is the analyzed information and related keywords. The AI model performs natural language processing and analyzes the text.

[0303] Step 4:

[0304] The server conducts an information search based on the analysis results. A search function is used to find relevant information from the knowledge base or database. The input is the analysis result obtained in Step 3, and the output is the set of required information. Search algorithms are utilized to perform text matching and extract related documents.

[0305] Step 5:

[0306] Based on the information acquired by the server, information is generated so that the user can view it. Prompt sentences are generated and responses are created. The input is the information obtained in Step 4, and the output is a displayable response message. Using generative AI, an operation is performed to naturally combine the information to generate a response.

[0307] Step 6:

[0308] The terminal displays the generated response on the augmented reality device. The server transmits the data to the display device, and the terminal can view it. The input is the response message generated in Step 5, and the output is the information visually displayed on the augmented reality device. Real-time data transfer and display are performed as operations.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0310] As an embodiment of the present invention, an autonomous predictive AI agent system incorporating an emotion engine that analyzes voice data from customers and identifies emotions is described. This system is designed to provide more accurate and emotionally empathetic responses to the customer service staff being assisted.

[0311] When a user makes an inquiry via telephone, the server receives the call and acquires the audio data. The server uses speech recognition technology to convert the acquired audio data into text data. This conversion ensures that the conversation is recorded in text format, allowing for smoother subsequent processing.

[0312] The server provides text data to the generating AI, which analyzes the inquiry. During this process, the emotion engine uses voice data to identify the user's emotions. For example, it analyzes parameters such as voice tone and speaking speed to determine whether the user is angry, confused, or calm. This emotion information, along with the inquiry category, becomes part of the analysis data.

[0313] Once the analysis is complete, the server uses the analysis results to search the database for the necessary information. This search includes past query history, similar cases, and relevant knowledge base information. The server collects this data so that the AI ​​can generate the most appropriate answer or recommended action.

[0314] The generation AI generates the optimal answer based on all the information it has acquired. During this process, the answer is adjusted according to the user's emotions, as identified by the emotion engine. For example, if the user is angry, it will generate a polite and calm-toned answer; if the user is confused, it will construct a more detailed and clear answer.

[0315] Finally, the server displays the generated response on the customer service representative's terminal. The representative can then quickly provide an appropriate response to the user while viewing the terminal. After the response is completed, the response details and emotional information confirmed on the terminal are automatically recorded by the server and organized as incident information. This entire process makes customer service more sophisticated and humane, contributing to improved customer satisfaction. For example, if a user angrily inquires about a billing error, the emotion engine can identify the anger, and the server can suggest a calmer response appropriate to that emotion.

[0316] The following describes the processing flow.

[0317] Step 1:

[0318] A user contacts the service desk by phone. The server receives the call and retrieves the audio data.

[0319] Step 2:

[0320] The server uses speech recognition technology to convert the acquired audio data into text data. This makes the conversation content analyzable in written form.

[0321] Step 3:

[0322] The server provides the converted text data to the generating AI, which then begins analysis. The generating AI extracts keywords from the text to understand the content of the inquiry.

[0323] Step 4:

[0324] The server simultaneously sends audio data to the emotion engine, which identifies the user's emotional state based on their voice tone, speaking speed, intonation, etc. The emotion engine then adds this to the analysis data.

[0325] Step 5:

[0326] The server searches the database based on the analyzed query content and sentiment information. It collects relevant query history and knowledge base information.

[0327] Step 6:

[0328] The generation AI generates appropriate responses based on database search results and sentiment information. For example, if the user is angry, a response in a calm tone will be generated.

[0329] Step 7:

[0330] The server sends the generated response and emotional considerations to the customer service representative's terminal for display. This allows the representative to respond to the user immediately.

[0331] Step 8:

[0332] After the response is complete, the server automatically records the details of the response and emotional information via the terminal and stores it in the database as incident information. This information will be used to improve future operations.

[0333] (Example 2)

[0334] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0335] Conventional information processing systems have faced challenges in adequately addressing customer inquiries emotionally, making it difficult to improve customer satisfaction. Furthermore, in the analysis of inquiry content, emotional information is often not considered, resulting in cases where optimal responses cannot be provided. This situation necessitates that customer service representatives provide prompt and accurate responses that take emotions into account.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0337] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information and understanding the content of the inquiry, and means for searching for information resources related to the analysis results. This makes it possible to perform analysis that takes customer emotions into account and provide appropriate information.

[0338] "Audio information" refers to information that represents audio acquired from customers in a digital format.

[0339] "Text information" refers to information in string format obtained by converting audio information into a parseable form.

[0340] "Analysis means" refers to means of understanding text information and processing it to classify and convert the content of inquiries into knowledge.

[0341] A "search tool" is a means of referencing and obtaining relevant information resources based on the analysis results.

[0342] "Generation means" refers to means for constructing an appropriate response based on information obtained by search means.

[0343] "Emotion identification means" refers to a method for analyzing the characteristics of voice information to determine the emotional state of a customer.

[0344] "Display means" refers to means for visually showing the response obtained by the generation means to the terminal.

[0345] "Recording means" refers to a means of saving the results of the counter staff's interactions and related information, and organizing them in a way that allows for later reference.

[0346] "Information resources" refer to various sources of information, such as databases and knowledge bases, used for analyzing inquiries and generating responses.

[0347] This embodiment of the invention describes a system that receives customer inquiries via voice information and provides responses that take into account the user's emotions. A server receives calls from customers and acquires voice information. The acquired voice information is converted into text information using speech recognition technology. In this process, general speech recognition software is used, and for example, an open-source speech recognition library can be applied.

[0348] Next, the server analyzes the text information. This analysis utilizes a generative AI model to support the understanding of the inquiry. A general natural language processing framework can be used as the specific generative AI model. Furthermore, emotion recognition is used to identify the customer's emotions from the audio information. Based on this, the server determines whether the user's emotion falls under "anger," "joy," "sadness," or "calmness."

[0349] To retrieve information resources related to a user's query, the server references an internal database based on the analysis results to obtain the appropriate information. This typically involves a general database system, such as past query history and related knowledge bases.

[0350] Based on the generated information, a generative AI model constructs a response. This model also considers emotions identified by emotion recognition tools, and adjusts the tone to suit the user. The generated response is displayed on the terminal and used as a basis for service staff to provide appropriate assistance.

[0351] As an example, consider a case where a user angrily inquires that there is an error in their bill. The emotion recognition means efficiently detects anger, and the server uses this result to generate a calm and composed response using a generative AI model. In this way, high-quality responses that take emotions into consideration are possible, which is expected to contribute to improved customer satisfaction.

[0352] (Example of prompts for a generative AI model)

[0353] "Please generate the best possible response to inquiries regarding service interruptions, taking into account user sentiment."

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] When a user makes an inquiry via telephone, the server receives the call. The input is the user's voice data, which the server captures and manages. During this process, the data is temporarily stored in an audio file format. The output is the stored audio data.

[0357] Step 2:

[0358] The server converts the received audio data into text data. It analyzes the audio using speech recognition technology and converts it into a string format. Specifically, by using a speech recognition API, the input audio data is output as text data.

[0359] Step 3:

[0360] The server provides text data to a generating AI model, which then analyzes the query. The input is the generated text data. The AI ​​model uses natural language processing algorithms to understand the context and extracts the intent and content of the query as part of the analysis. The output is the analysis result.

[0361] Step 4:

[0362] The server uses the analysis results to identify the user's emotions using emotion recognition tools. The input is voice data and its feature extraction results. By analyzing the pitch, tone, and speaking speed of the voice, it generates emotion information as output. This information includes emotion categories such as "anger," "joy," and "calmness."

[0363] Step 5:

[0364] The server uses the analysis results and sentiment information to search the company's database and collect relevant information. The input is the analysis results and sentiment information. By executing database queries, it retrieves the necessary data from past query history and relevant knowledge bases. The output is relevant information based on the search results.

[0365] Step 6:

[0366] The generative AI model generates the optimal response based on collected information and sentiment data. The input consists of information and sentiment data obtained through searches. The generation process uses high-priority data to produce a structured response. The output is a suggested response to the user.

[0367] Step 7:

[0368] The server sends the generated response to the terminal and displays it on the counter staff's screen. The input is the generated response, which is visualized on the terminal. The counter staff refer to this output and take appropriate action in real time.

[0369] Step 8:

[0370] Once the interaction is complete, the interaction details and emotional information recorded on the terminal are sent to the server. The input consists of the interaction result and emotional information. Based on this, the server organizes the data and records it in the database as an incident report. The output is structured incident information.

[0371] (Application Example 2)

[0372] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0373] Traditional customer service systems struggle to accurately understand customers' emotional states and optimize responses. This can lead to systems generating mechanical responses and decreased customer satisfaction. Furthermore, in sensitive or urgent situations, appropriate responses may not be provided promptly. This hinders the building of trust with customers and results in lost business opportunities for companies.

[0374] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0375] In this invention, the server includes emotion identification means for identifying emotional states from voice information, analysis means for analyzing text information and understanding the content of inquiries, and adjustment means for adjusting the response according to the emotional state identified by the emotion identification means. This makes it possible to accurately grasp the customer's emotional state and provide an optimal response accordingly.

[0376] "Audio information" refers to linguistic information spoken by customers, captured as acoustic signals.

[0377] "Textual information" refers to data in text format obtained by converting audio information.

[0378] "Analysis means" refers to a device or program that has the function of analyzing textual information to understand the content of an inquiry.

[0379] A "search means" is a system that has the function of finding relevant recording media based on the analysis results obtained by the analysis means.

[0380] "Generation means" refers to a mechanism or software that has the function of creating an optimal response in accordance with the results of the search means.

[0381] "Display means" refers to a means for visually displaying the generated response on the counter staff member's device.

[0382] "Recording means" refers to a method or mechanism for long-term storage of the results of a customer service representative's interactions.

[0383] An "emotion identification means" is a device that has the function of identifying a customer's emotional state from voice information.

[0384] A "modification mechanism" is a system that has the function of adjusting the content of the response according to the customer's emotional state.

[0385] This invention is an autonomous predictive AI agent system for streamlining customer service. The server receives voice information from customers and converts it into text information using speech recognition software. For speech recognition, for example, the Google Speech Recognition API is used. The converted text information is analyzed by an analysis means to identify the emotional state associated with the inquiry. For the analysis, the Sentiment Analysis pipeline of Transformers, a natural language processing library, is used.

[0386] Based on the user's inquiry, the server searches for relevant information from its storage medium. This storage medium includes past inquiry history and a knowledge base. Based on the retrieved information and the emotional state identified by the emotion recognition system, a generative AI model generates the optimal response. The generated response is displayed on the customer service representative's terminal. This terminal facilitates the rapid and appropriate response to the user.

[0387] Furthermore, the server automatically records the results of the responses by the customer service representatives and saves them as a response history. This recording process ensures that incident information is managed systematically.

[0388] For example, if a user inquires about a security system malfunction, the emotion recognition system will determine that the user is distressed, and the generative AI model will provide a reassuring response. An example of a prompt would be, "In a situation where a customer is anxiously reporting a security issue, please show how to respond appropriately based on the tone the emotion engine perceives." This enables an emotionally sensitive response, contributing to improved customer satisfaction.

[0389] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0390] Step 1:

[0391] The server receives voice information from the user. This information is input into speech recognition software, which converts the voice signal into text information. This conversion process outputs the customer's inquiry as text data. Specifically, the Google Speech Recognition API is used to analyze the voice and convert it into the corresponding string.

[0392] Step 2:

[0393] The server uses the obtained text information as input and employs an analysis tool to identify the content of the inquiry and the user's emotional state. This analysis utilizes Transformers' Sentiment Analysis pipeline to analyze the text information and output the user's emotions. Specifically, it analyzes the word choices and context within the string to determine the emotional label (positive, negative, neutral).

[0394] Step 3:

[0395] The server searches the storage medium based on the user's query content identified through analysis. The storage medium outputs past history and knowledge base information related to the query. Specifically, it executes database queries to extract highly relevant documents and information.

[0396] Step 4:

[0397] The server uses an AI model to generate the optimal response, taking information obtained from the recording medium and the emotional state obtained through emotion recognition as input. The generated response is output as text data in the format most appropriate to the user's situation. In terms of operation, it constructs prompt sentences based on the emotional state and executes a process to generate appropriate response sentences.

[0398] Step 5:

[0399] The terminal displays the generated response sent from the server. The displayed response allows the service representative to quickly provide the user with an appropriate answer. This step involves visually presenting the response text on the display.

[0400] Step 6:

[0401] The server automatically records the interactions performed by the customer service representative using recording mechanisms. The interaction history is organized as incident information and used to improve future interactions. Specifically, it saves the response text, emotional state, and interaction result to a data store.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0403] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0405] [Third Embodiment]

[0406] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0407] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0408] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0410] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0412] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0413] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0414] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0417] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0418] As an embodiment of the present invention, an autonomous predictive AI agent system for a service desk is described. This system includes a process of converting voice data into text and analyzing it in real time in order to respond quickly and accurately to customer inquiries. Its specific operation is shown below.

[0419] First, when a user becomes a customer and initiates a phone inquiry to the service desk, the server receives the call. The received audio data is immediately converted into text data by the server. This conversion process prepares the audio information into a format that can be easily analyzed as text.

[0420] Next, the server uses a generative AI to analyze the transcribed conversation. In this analysis, the server aims to extract the inquiry content and important keywords from the conversation and understand the intent of the inquiry. At this time, it performs a deeper analysis based on relevant past inquiry history and information in the knowledge base.

[0421] Once the analysis is complete, the server searches the database based on the results to find similar cases and appropriate solutions. This process comprehensively considers past response examples and related information to formulate appropriate countermeasures.

[0422] Subsequently, the server uses AI to generate the optimal response to provide to the customer based on this information. This generated response is concise and easy to understand, making it effective when customer service staff guide users.

[0423] Finally, the server displays the generated response on the employee's terminal, helping them to take appropriate action immediately with the user. After the response is complete, the server automatically records the details of the response through the terminal's operation, making it available later as incident information.

[0424] For example, if a user requests to know the details of their invoice, the server transcribes this into text, searches the knowledge base, and provides the most recent invoice information. The staff member can then view the information on their terminal and immediately inform the user of the invoice amount and payment deadline. This entire process improves the efficiency of customer service and increases customer satisfaction.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] A user contacts the service desk by phone. The server receives the call and obtains the audio data. Since the audio data is difficult to analyze directly, the server uses speech recognition technology to convert it into text data.

[0428] Step 2:

[0429] The server inputs the converted text data into the generating AI. The generating AI analyzes the text content and extracts important keywords and the intent of the inquiry. This analysis process identifies which category the inquiry belongs to.

[0430] Step 3:

[0431] Based on the analysis results, the server searches its internal database. This includes past query history and relevant information within the knowledge base. The server extracts the most relevant data and prepares it for the next stage of answer generation.

[0432] Step 4:

[0433] The generation AI generates appropriate answers and recommendations based on the searched data. This process also considers predictive answers to questions the customer might ask next. The generated answers are concise and clearly structured.

[0434] Step 5:

[0435] The server sends the generated response to the counter staff member's terminal in real time. The terminal displays this information on the screen, allowing the staff member to respond to the customer immediately.

[0436] Step 6:

[0437] After the user interaction is complete, the server automatically records the details of the interaction in a database via the user's terminal. This incident information includes the inquiry, details of the interaction, and the outcome. This data is useful for future interactions and improvements.

[0438] (Example 1)

[0439] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0440] Responding quickly and accurately to customer inquiries is crucial for improving customer satisfaction and operational efficiency. However, traditional methods of handling inquiries were time-consuming, requiring time to understand the content of the inquiry, refer to past cases, and develop appropriate solutions, making efficient responses difficult. Therefore, there is a need for techniques that accurately grasp customer intent and provide the most suitable answer quickly based on past experience.

[0441] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0442] In this invention, the server includes means for receiving voice information and converting said voice information into text information; means for analyzing said text information to understand the content of the inquiry and extract important words; and means for further analyzing the analysis results obtained by the analysis means based on past response records and a knowledge database and searching for similar past cases. This enables the generation of appropriate suggestions based on customer inquiries and quick customer service responses.

[0443] "Auditory information" refers to the form of sound transmitted through sound waves, and specifically to data recorded as spoken language.

[0444] "Textual information" refers to data in written form converted from audio information, and is a string of characters processed by a computer.

[0445] "Analysis means" refers to a process or device for understanding meaning from textual information and extracting important words or inquiry content.

[0446] A "knowledge database" refers to a collection of data accumulated by an organization, including information and past response records, and is a source of information used for problem-solving.

[0447] "Recorded information" refers to data that is stored as a result of customer interactions or data obtained during those interactions, and is used for future reference and analysis.

[0448] "Event information" refers to a part of the recorded information, specifically detailed data about the events that occurred and the results of the responses.

[0449] This invention relates to an autonomous predictive AI agent system for responding quickly and accurately to customer inquiries. Specific embodiments thereof are described below.

[0450] First, the user uses a communication device to call the service desk. The server receives the voice information via the communication network. This voice information is instantly converted into text information using speech recognition technology. A speech recognition API is one example of the specific software that can be used for this purpose.

[0451] Next, the server uses a generative AI model to analyze the textual information. This extracts and accurately understands the user's inquiry and important words. Natural language processing models are commonly used as generative AI models. This analysis process also references past interaction records and knowledge databases.

[0452] Furthermore, the server performs a database search based on the analysis results. This is to identify similar past cases and appropriate countermeasures. Past data is stored in a knowledge database, allowing for quick searching and utilization.

[0453] Subsequently, the server constructs the optimal solution based on the generated information and displays it on the terminal. This solution is visually easy to understand so that the service representative can guide the user appropriately.

[0454] Finally, after the interaction is complete, the server automatically saves the details of the interaction as record information via the terminal. This accumulates data that can be used for handling future inquiries.

[0455] A concrete example is when a user inquires, "I want to know the details of my invoice." In this case, the server converts the voice information into text information and analyzes it using a generation AI model to quickly search for relevant information from the knowledge base. The information displayed on the terminal can then be confirmed by the staff member, who can immediately provide the user with the details of the invoice.

[0456] An example of a prompt message is, "Generate the best response when a user asks, 'I want to know the details of the invoice.'" This enables effective responses based on the invention.

[0457] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0458] Step 1:

[0459] When a user calls the service desk using a communication device, the server receives the audio information. This received audio information is the input data. The server converts the audio into text using speech recognition technology. In this process, a speech recognition API is used, and the content of the conversation is output as text by converting the sound waveform into phonemes. Specifically, the server processes the audio in real time and generates text information with minimal error.

[0460] Step 2:

[0461] The server uses a generative AI model to analyze the textual information generated in Step 1. The input for this analysis process is textual information. The server performs natural language processing to understand the content of the textual information and extracts important words and query content. The output is a list of the analyzed query content and important words. Specifically, the server analyzes complex contexts and compares them with relevant historical records.

[0462] Step 3:

[0463] The server searches the database based on the analysis results. The input is the analysis results obtained in step 2. The server quickly searches past cases and related knowledge to identify similar past cases and solutions. The output is a list of highly relevant cases. Specifically, the server uses an efficient search algorithm to collect the most relevant information in a short amount of time.

[0464] Step 4:

[0465] The server utilizes generation AI to generate the optimal solution based on the information obtained in step 3 and displays it on the terminal. The input is the result of a database search. The server integrates this information to construct an easy-to-understand answer, which is then visually displayed on the terminal as output. Specifically, the terminal arranges the generated answer in an easily viewable format, allowing staff to respond to users quickly.

[0466] Step 5:

[0467] Once the response is complete, the server automatically saves the response results as record information via the terminal. The input is a detailed description of the response performed by the person in charge on the terminal. The server saves this as record information in the database to prepare for future inquiries. The output is a detailed response record. Specifically, the server quickly organizes the record content and makes it quickly searchable as needed.

[0468] (Application Example 1)

[0469] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] In modern customer service, prompt and accurate customer responses are essential, but traditional systems struggle with real-time processing of voice data and providing appropriate information. Furthermore, there is a need for a method that allows service providers to immediately obtain necessary information and respond efficiently. Solving these challenges is crucial to improving customer satisfaction and increasing operational efficiency.

[0471] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0472] In this invention, the server includes means for acquiring voice signals from customers and converting said voice signals into text data, an analysis function for analyzing the text data and understanding the content of the inquiry, and a search function for searching for information related to the analysis results acquired based on the analysis function. This enables the server to immediately present the most appropriate information in response to customer inquiries, and allows staff to respond quickly while visually confirming the information using augmented reality devices.

[0473] An "audio signal" is an electrical signal that represents audio information, and it is the foundation for converting human voices into digital data.

[0474] "Character data" refers to data that represents audio signals as a string of characters, and is information that can be processed by a computer as text.

[0475] The "analysis function" is a function that analyzes text data to understand its content and intent, and plays a role in identifying the intent of the inquiry.

[0476] The "search function" is a function that, based on the information identified by the analysis function, searches for relevant databases and information sources and retrieves the necessary data.

[0477] "Information" refers to the knowledge and data necessary to process customer inquiries, which are identified through analytical and search functions.

[0478] An "augmented reality device" is a device that overlays digital information onto the real world environment, and is intended to provide visual information.

[0479] To realize this invention, it is necessary to configure a system using a server, a user's device (terminal), and an augmented reality device. A specific embodiment of this system is shown below.

[0480] The server first receives an audio signal from the user. This audio signal is sent to the server, for example, via a telephone or voice input device. The server then uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data.

[0481] Next, the server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand the content and intent of the inquiry. This analysis accurately extracts customer requests and related keywords.

[0482] Based on the analysis results, the server utilizes its search function to retrieve information. This allows it to quickly obtain necessary information from relevant past cases and knowledge bases. The retrieved information is then generated as an appropriate response, and the server prepares this information to be presented to the user's device.

[0483] Here, it is possible to display information within the employee's field of vision via an augmented reality device (e.g., smart glasses). The displayed information allows the employee to quickly review it and provides support for taking appropriate action with the customer.

[0484] For example, if a customer requests to "check their recent payment history," the server analyzes this request and retrieves the appropriate payment history information from the knowledge base. This information is then displayed to the representative via an augmented reality device, allowing the representative to immediately provide the information to the customer.

[0485] An example of a prompt message is, "After converting the audio data to text, retrieve the latest payment history information from the knowledge base and present it to the representative."

[0486] This system will enable faster and more accurate customer service, thereby increasing customer satisfaction.

[0487] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0488] Step 1:

[0489] The server receives an audio signal from the user. The audio, acquired via the audio input device, is then transferred to the server. The input is an audio signal, and the output is that audio signal stored on the server in digital format.

[0490] Step 2:

[0491] The server uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data. The input is the audio signal obtained in step 1, and the output is text data suitable for analysis. Here, data processing is performed to convert the audio signal into a string using an algorithm.

[0492] Step 3:

[0493] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data. The goal is to understand the content and intent of the inquiry and extract important keywords. The input is the text data generated in step 2, and the output is the analyzed information and related keywords. The AI ​​model performs natural language processing to analyze the text.

[0494] Step 4:

[0495] The server performs information retrieval based on the analysis results. It uses search functions to find relevant information from knowledge bases and databases. The input is the analysis results obtained in step 3, and the output is a collection of required information. Search algorithms are utilized, performing text matching and extracting related documents.

[0496] Step 5:

[0497] Based on the information acquired by the server, information is generated for the user to review. A prompt message is generated, and a response is created. The input is the information obtained in step 4, and the output is a displayable response message. A generation AI is used to naturally combine the information and generate the response.

[0498] Step 6:

[0499] The terminal displays the generated response on the augmented reality device. The server sends data to the display device, which the terminal can then verify. The input is the response message generated in step 5, and the output is the information visually displayed on the augmented reality device. Real-time data transfer and display are performed as part of the operation.

[0500] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0501] As an embodiment of the present invention, an autonomous predictive AI agent system incorporating an emotion engine that analyzes voice data from customers and identifies emotions is described. This system is designed to provide more accurate and emotionally empathetic responses to the customer service staff being assisted.

[0502] When a user makes an inquiry via telephone, the server receives the call and acquires the audio data. The server uses speech recognition technology to convert the acquired audio data into text data. This conversion ensures that the conversation is recorded in text format, allowing for smoother subsequent processing.

[0503] The server provides text data to the generating AI, which analyzes the inquiry. During this process, the emotion engine uses voice data to identify the user's emotions. For example, it analyzes parameters such as voice tone and speaking speed to determine whether the user is angry, confused, or calm. This emotion information, along with the inquiry category, becomes part of the analysis data.

[0504] Once the analysis is complete, the server uses the analysis results to search the database for the necessary information. This search includes past query history, similar cases, and relevant knowledge base information. The server collects this data so that the AI ​​can generate the most appropriate answer or recommended action.

[0505] The generation AI generates the optimal answer based on all the information it has acquired. During this process, the answer is adjusted according to the user's emotions, as identified by the emotion engine. For example, if the user is angry, it will generate a polite and calm-toned answer; if the user is confused, it will construct a more detailed and clear answer.

[0506] Finally, the server displays the generated response on the customer service representative's terminal. The representative can then quickly provide an appropriate response to the user while viewing the terminal. After the response is completed, the response details and emotional information confirmed on the terminal are automatically recorded by the server and organized as incident information. This entire process makes customer service more sophisticated and humane, contributing to improved customer satisfaction. For example, if a user angrily inquires about a billing error, the emotion engine can identify the anger, and the server can suggest a calmer response appropriate to that emotion.

[0507] The following describes the processing flow.

[0508] Step 1:

[0509] A user contacts the service desk by phone. The server receives the call and retrieves the audio data.

[0510] Step 2:

[0511] The server uses speech recognition technology to convert the acquired audio data into text data. This makes the conversation content analyzable in written form.

[0512] Step 3:

[0513] The server provides the converted text data to the generating AI, which then begins analysis. The generating AI extracts keywords from the text to understand the content of the inquiry.

[0514] Step 4:

[0515] The server simultaneously sends audio data to the emotion engine, which identifies the user's emotional state based on their voice tone, speaking speed, intonation, etc. The emotion engine then adds this to the analysis data.

[0516] Step 5:

[0517] The server searches the database based on the analyzed query content and sentiment information. It collects relevant query history and knowledge base information.

[0518] Step 6:

[0519] The generation AI generates appropriate responses based on database search results and sentiment information. For example, if the user is angry, a response in a calm tone will be generated.

[0520] Step 7:

[0521] The server sends the generated response and emotional considerations to the customer service representative's terminal for display. This allows the representative to respond to the user immediately.

[0522] Step 8:

[0523] After the response is complete, the server automatically records the details of the response and emotional information via the terminal and stores it in the database as incident information. This information will be used to improve future operations.

[0524] (Example 2)

[0525] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0526] Conventional information processing systems have faced challenges in adequately addressing customer inquiries emotionally, making it difficult to improve customer satisfaction. Furthermore, in the analysis of inquiry content, emotional information is often not considered, resulting in cases where optimal responses cannot be provided. This situation necessitates that customer service representatives provide prompt and accurate responses that take emotions into account.

[0527] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0528] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information and understanding the content of the inquiry, and means for searching for information resources related to the analysis results. This makes it possible to perform analysis that takes customer emotions into account and provide appropriate information.

[0529] "Audio information" refers to information that represents audio acquired from customers in a digital format.

[0530] "Text information" refers to information in string format obtained by converting audio information into a parseable form.

[0531] "Analysis means" refers to means of understanding text information and processing it to classify and convert the content of inquiries into knowledge.

[0532] A "search tool" is a means of referencing and obtaining relevant information resources based on the analysis results.

[0533] "Generation means" refers to means for constructing an appropriate response based on information obtained by search means.

[0534] "Emotion identification means" refers to a method for analyzing the characteristics of voice information to determine the emotional state of a customer.

[0535] "Display means" refers to means for visually showing the response obtained by the generation means to the terminal.

[0536] "Recording means" refers to a means of saving the results of the counter staff's interactions and related information, and organizing them in a way that allows for later reference.

[0537] "Information resources" refer to various sources of information, such as databases and knowledge bases, used for analyzing inquiries and generating responses.

[0538] This embodiment of the invention describes a system that receives customer inquiries via voice information and provides responses that take into account the user's emotions. A server receives calls from customers and acquires voice information. The acquired voice information is converted into text information using speech recognition technology. In this process, general speech recognition software is used, and for example, an open-source speech recognition library can be applied.

[0539] Next, the server analyzes the text information. This analysis utilizes a generative AI model to support the understanding of the inquiry. A general natural language processing framework can be used as the specific generative AI model. Furthermore, emotion recognition is used to identify the customer's emotions from the audio information. Based on this, the server determines whether the user's emotion falls under "anger," "joy," "sadness," or "calmness."

[0540] To retrieve information resources related to a user's query, the server references an internal database based on the analysis results to obtain the appropriate information. This typically involves a general database system, such as past query history and related knowledge bases.

[0541] Based on the generated information, a generative AI model constructs a response. This model also considers emotions identified by emotion recognition tools, and adjusts the tone to suit the user. The generated response is displayed on the terminal and used as a basis for service staff to provide appropriate assistance.

[0542] As an example, consider a case where a user angrily inquires that there is an error in their bill. The emotion recognition means efficiently detects anger, and the server uses this result to generate a calm and composed response using a generative AI model. In this way, high-quality responses that take emotions into consideration are possible, which is expected to contribute to improved customer satisfaction.

[0543] (Example of prompts for a generative AI model)

[0544] "Please generate the best possible response to inquiries regarding service interruptions, taking into account user sentiment."

[0545] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0546] Step 1:

[0547] When a user makes an inquiry via telephone, the server receives the call. The input is the user's voice data, which the server captures and manages. During this process, the data is temporarily stored in an audio file format. The output is the stored audio data.

[0548] Step 2:

[0549] The server converts the received audio data into text data. It analyzes the audio using speech recognition technology and converts it into a string format. Specifically, by using a speech recognition API, the input audio data is output as text data.

[0550] Step 3:

[0551] The server provides text data to a generating AI model, which then analyzes the query. The input is the generated text data. The AI ​​model uses natural language processing algorithms to understand the context and extracts the intent and content of the query as part of the analysis. The output is the analysis result.

[0552] Step 4:

[0553] The server uses the analysis results to identify the user's emotions using emotion recognition tools. The input is voice data and its feature extraction results. By analyzing the pitch, tone, and speaking speed of the voice, it generates emotion information as output. This information includes emotion categories such as "anger," "joy," and "calmness."

[0554] Step 5:

[0555] The server uses the analysis results and sentiment information to search the company's database and collect relevant information. The input is the analysis results and sentiment information. By executing database queries, it retrieves the necessary data from past query history and relevant knowledge bases. The output is relevant information based on the search results.

[0556] Step 6:

[0557] The generative AI model generates the optimal response based on collected information and sentiment data. The input consists of information and sentiment data obtained through searches. The generation process uses high-priority data to produce a structured response. The output is a suggested response to the user.

[0558] Step 7:

[0559] The server sends the generated response to the terminal and displays it on the counter staff's screen. The input is the generated response, which is visualized on the terminal. The counter staff refer to this output and take appropriate action in real time.

[0560] Step 8:

[0561] Once the interaction is complete, the interaction details and emotional information recorded on the terminal are sent to the server. The input consists of the interaction result and emotional information. Based on this, the server organizes the data and records it in the database as an incident report. The output is structured incident information.

[0562] (Application Example 2)

[0563] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] Traditional customer service systems struggle to accurately understand customers' emotional states and optimize responses. This can lead to systems generating mechanical responses and decreased customer satisfaction. Furthermore, in sensitive or urgent situations, appropriate responses may not be provided promptly. This hinders the building of trust with customers and results in lost business opportunities for companies.

[0565] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0566] In this invention, the server includes emotion identification means for identifying emotional states from voice information, analysis means for analyzing text information and understanding the content of inquiries, and adjustment means for adjusting the response according to the emotional state identified by the emotion identification means. This makes it possible to accurately grasp the customer's emotional state and provide an optimal response accordingly.

[0567] "Audio information" refers to linguistic information spoken by customers, captured as acoustic signals.

[0568] "Textual information" refers to data in text format obtained by converting audio information.

[0569] "Analysis means" refers to a device or program that has the function of analyzing textual information to understand the content of an inquiry.

[0570] A "search means" is a system that has the function of finding relevant recording media based on the analysis results obtained by the analysis means.

[0571] "Generation means" refers to a mechanism or software that has the function of creating an optimal response in accordance with the results of the search means.

[0572] "Display means" refers to a means for visually displaying the generated response on the counter staff member's device.

[0573] "Recording means" refers to a method or mechanism for long-term storage of the results of a customer service representative's interactions.

[0574] An "emotion identification means" is a device that has the function of identifying a customer's emotional state from voice information.

[0575] A "modification mechanism" is a system that has the function of adjusting the content of the response according to the customer's emotional state.

[0576] This invention is an autonomous predictive AI agent system for streamlining customer service. The server receives voice information from customers and converts it into text information using speech recognition software. For speech recognition, for example, the Google Speech Recognition API is used. The converted text information is analyzed by an analysis means to identify the emotional state associated with the inquiry. For the analysis, the Sentiment Analysis pipeline of Transformers, a natural language processing library, is used.

[0577] Based on the user's inquiry, the server searches for relevant information from its storage medium. This storage medium includes past inquiry history and a knowledge base. Based on the retrieved information and the emotional state identified by the emotion recognition system, a generative AI model generates the optimal response. The generated response is displayed on the customer service representative's terminal. This terminal facilitates the rapid and appropriate response to the user.

[0578] Furthermore, the server automatically records the results of the responses by the customer service representatives and saves them as a response history. This recording process ensures that incident information is managed systematically.

[0579] For example, if a user inquires about a security system malfunction, the emotion recognition system will determine that the user is distressed, and the generative AI model will provide a reassuring response. An example of a prompt would be, "In a situation where a customer is anxiously reporting a security issue, please show how to respond appropriately based on the tone the emotion engine perceives." This enables an emotionally sensitive response, contributing to improved customer satisfaction.

[0580] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0581] Step 1:

[0582] The server receives voice information from the user. This information is input into speech recognition software, which converts the voice signal into text information. This conversion process outputs the customer's inquiry as text data. Specifically, the Google Speech Recognition API is used to analyze the voice and convert it into the corresponding string.

[0583] Step 2:

[0584] The server uses the obtained text information as input and employs an analysis tool to identify the content of the inquiry and the user's emotional state. This analysis utilizes Transformers' Sentiment Analysis pipeline to analyze the text information and output the user's emotions. Specifically, it analyzes the word choices and context within the string to determine the emotional label (positive, negative, neutral).

[0585] Step 3:

[0586] The server searches the storage medium based on the user's query content identified through analysis. The storage medium outputs past history and knowledge base information related to the query. Specifically, it executes database queries to extract highly relevant documents and information.

[0587] Step 4:

[0588] The server uses an AI model to generate the optimal response, taking information obtained from the recording medium and the emotional state obtained through emotion recognition as input. The generated response is output as text data in the format most appropriate to the user's situation. In terms of operation, it constructs prompt sentences based on the emotional state and executes a process to generate appropriate response sentences.

[0589] Step 5:

[0590] The terminal displays the generated response sent from the server. The displayed response allows the service representative to quickly provide the user with an appropriate answer. This step involves visually presenting the response text on the display.

[0591] Step 6:

[0592] The server automatically records the interactions performed by the customer service representative using recording mechanisms. The interaction history is organized as incident information and used to improve future interactions. Specifically, it saves the response text, emotional state, and interaction result to a data store.

[0593] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0594] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0595] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0596] [Fourth Embodiment]

[0597] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0598] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0599] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0600] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0601] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0602] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0603] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0604] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0605] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0606] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0607] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0608] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0609] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0610] As an embodiment of the present invention, an autonomous predictive AI agent system for a service desk is described. This system includes a process of converting voice data into text and analyzing it in real time in order to respond quickly and accurately to customer inquiries. Its specific operation is shown below.

[0611] First, when a user becomes a customer and initiates a phone inquiry to the service desk, the server receives the call. The received audio data is immediately converted into text data by the server. This conversion process prepares the audio information into a format that can be easily analyzed as text.

[0612] Next, the server uses a generative AI to analyze the transcribed conversation. In this analysis, the server aims to extract the inquiry content and important keywords from the conversation and understand the intent of the inquiry. At this time, it performs a deeper analysis based on relevant past inquiry history and information in the knowledge base.

[0613] Once the analysis is complete, the server searches the database based on the results to find similar cases and appropriate solutions. This process comprehensively considers past response examples and related information to formulate appropriate countermeasures.

[0614] Subsequently, the server uses AI to generate the optimal response to provide to the customer based on this information. This generated response is concise and easy to understand, making it effective when customer service staff guide users.

[0615] Finally, the server displays the generated response on the employee's terminal, helping them to take appropriate action immediately with the user. After the response is complete, the server automatically records the details of the response through the terminal's operation, making it available later as incident information.

[0616] For example, if a user requests to know the details of their invoice, the server transcribes this into text, searches the knowledge base, and provides the most recent invoice information. The staff member can then view the information on their terminal and immediately inform the user of the invoice amount and payment deadline. This entire process improves the efficiency of customer service and increases customer satisfaction.

[0617] The following describes the processing flow.

[0618] Step 1:

[0619] A user contacts the service desk by phone. The server receives the call and obtains the audio data. Since the audio data is difficult to analyze directly, the server uses speech recognition technology to convert it into text data.

[0620] Step 2:

[0621] The server inputs the converted text data into the generating AI. The generating AI analyzes the text content and extracts important keywords and the intent of the inquiry. This analysis process identifies which category the inquiry belongs to.

[0622] Step 3:

[0623] Based on the analysis results, the server searches its internal database. This includes past query history and relevant information within the knowledge base. The server extracts the most relevant data and prepares it for the next stage of answer generation.

[0624] Step 4:

[0625] The generation AI generates appropriate answers and recommendations based on the searched data. This process also considers predictive answers to questions the customer might ask next. The generated answers are concise and clearly structured.

[0626] Step 5:

[0627] The server sends the generated response to the counter staff member's terminal in real time. The terminal displays this information on the screen, allowing the staff member to respond to the customer immediately.

[0628] Step 6:

[0629] After the user interaction is complete, the server automatically records the details of the interaction in a database via the user's terminal. This incident information includes the inquiry, details of the interaction, and the outcome. This data is useful for future interactions and improvements.

[0630] (Example 1)

[0631] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0632] Responding quickly and accurately to customer inquiries is crucial for improving customer satisfaction and operational efficiency. However, traditional methods of handling inquiries were time-consuming, requiring time to understand the content of the inquiry, refer to past cases, and develop appropriate solutions, making efficient responses difficult. Therefore, there is a need for techniques that accurately grasp customer intent and provide the most suitable answer quickly based on past experience.

[0633] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0634] In this invention, the server includes means for receiving voice information and converting said voice information into text information; means for analyzing said text information to understand the content of the inquiry and extract important words; and means for further analyzing the analysis results obtained by the analysis means based on past response records and a knowledge database and searching for similar past cases. This enables the generation of appropriate suggestions based on customer inquiries and quick customer service responses.

[0635] "Auditory information" refers to the form of sound transmitted through sound waves, and specifically to data recorded as spoken language.

[0636] "Textual information" refers to data in written form converted from audio information, and is a string of characters processed by a computer.

[0637] "Analysis means" refers to a process or device for understanding meaning from textual information and extracting important words or inquiry content.

[0638] A "knowledge database" refers to a collection of data accumulated by an organization, including information and past response records, and is a source of information used for problem-solving.

[0639] "Recorded information" refers to data that is stored as a result of customer interactions or data obtained during those interactions, and is used for future reference and analysis.

[0640] "Event information" refers to a part of the recorded information, specifically detailed data about the events that occurred and the results of the responses.

[0641] This invention relates to an autonomous predictive AI agent system for responding quickly and accurately to customer inquiries. Specific embodiments thereof are described below.

[0642] First, the user uses a communication device to call the service desk. The server receives the voice information via the communication network. This voice information is instantly converted into text information using speech recognition technology. A speech recognition API is one example of the specific software that can be used for this purpose.

[0643] Next, the server uses a generative AI model to analyze the textual information. This extracts and accurately understands the user's inquiry and important words. Natural language processing models are commonly used as generative AI models. This analysis process also references past interaction records and knowledge databases.

[0644] Furthermore, the server performs a database search based on the analysis results. This is to identify similar past cases and appropriate countermeasures. Past data is stored in a knowledge database, allowing for quick searching and utilization.

[0645] Subsequently, the server constructs the optimal solution based on the generated information and displays it on the terminal. This solution is visually easy to understand so that the service representative can guide the user appropriately.

[0646] Finally, after the interaction is complete, the server automatically saves the details of the interaction as record information via the terminal. This accumulates data that can be used for handling future inquiries.

[0647] A concrete example is when a user inquires, "I want to know the details of my invoice." In this case, the server converts the voice information into text information and analyzes it using a generation AI model to quickly search for relevant information from the knowledge base. The information displayed on the terminal can then be confirmed by the staff member, who can immediately provide the user with the details of the invoice.

[0648] An example of a prompt message is, "Generate the best response when a user asks, 'I want to know the details of the invoice.'" This enables effective responses based on the invention.

[0649] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0650] Step 1:

[0651] When a user calls the service desk using a communication device, the server receives the audio information. This received audio information is the input data. The server converts the audio into text using speech recognition technology. In this process, a speech recognition API is used, and the content of the conversation is output as text by converting the sound waveform into phonemes. Specifically, the server processes the audio in real time and generates text information with minimal error.

[0652] Step 2:

[0653] The server uses a generative AI model to analyze the textual information generated in Step 1. The input for this analysis process is textual information. The server performs natural language processing to understand the content of the textual information and extracts important words and query content. The output is a list of the analyzed query content and important words. Specifically, the server analyzes complex contexts and compares them with relevant historical records.

[0654] Step 3:

[0655] The server searches the database based on the analysis results. The input is the analysis results obtained in step 2. The server quickly searches past cases and related knowledge to identify similar past cases and solutions. The output is a list of highly relevant cases. Specifically, the server uses an efficient search algorithm to collect the most relevant information in a short amount of time.

[0656] Step 4:

[0657] The server utilizes generation AI to generate the optimal solution based on the information obtained in step 3 and displays it on the terminal. The input is the result of a database search. The server integrates this information to construct an easy-to-understand answer, which is then visually displayed on the terminal as output. Specifically, the terminal arranges the generated answer in an easily viewable format, allowing staff to respond to users quickly.

[0658] Step 5:

[0659] Once the response is complete, the server automatically saves the response results as record information via the terminal. The input is a detailed description of the response performed by the person in charge on the terminal. The server saves this as record information in the database to prepare for future inquiries. The output is a detailed response record. Specifically, the server quickly organizes the record content and makes it quickly searchable as needed.

[0660] (Application Example 1)

[0661] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] In modern customer service, prompt and accurate customer responses are essential, but traditional systems struggle with real-time processing of voice data and providing appropriate information. Furthermore, there is a need for a method that allows service providers to immediately obtain necessary information and respond efficiently. Solving these challenges is crucial to improving customer satisfaction and increasing operational efficiency.

[0663] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0664] In this invention, the server includes means for acquiring voice signals from customers and converting said voice signals into text data, an analysis function for analyzing the text data and understanding the content of the inquiry, and a search function for searching for information related to the analysis results acquired based on the analysis function. This enables the server to immediately present the most appropriate information in response to customer inquiries, and allows staff to respond quickly while visually confirming the information using augmented reality devices.

[0665] An "audio signal" is an electrical signal that represents audio information, and it is the foundation for converting human voices into digital data.

[0666] "Character data" refers to data that represents audio signals as a string of characters, and is information that can be processed by a computer as text.

[0667] The "analysis function" is a function that analyzes text data to understand its content and intent, and plays a role in identifying the intent of the inquiry.

[0668] The "search function" is a function that, based on the information identified by the analysis function, searches for relevant databases and information sources and retrieves the necessary data.

[0669] "Information" refers to the knowledge and data necessary to process customer inquiries, which are identified through analytical and search functions.

[0670] An "augmented reality device" is a device that overlays digital information onto the real world environment, and is intended to provide visual information.

[0671] To realize this invention, it is necessary to configure a system using a server, a user's device (terminal), and an augmented reality device. A specific embodiment of this system is shown below.

[0672] The server first receives an audio signal from the user. This audio signal is sent to the server, for example, via a telephone or voice input device. The server then uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data.

[0673] Next, the server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand the content and intent of the inquiry. This analysis accurately extracts customer requests and related keywords.

[0674] Based on the analysis results, the server utilizes its search function to retrieve information. This allows it to quickly obtain necessary information from relevant past cases and knowledge bases. The retrieved information is then generated as an appropriate response, and the server prepares this information to be presented to the user's device.

[0675] Here, it is possible to display information within the employee's field of vision via an augmented reality device (e.g., smart glasses). The displayed information allows the employee to quickly review it and provides support for taking appropriate action with the customer.

[0676] For example, if a customer requests to "check their recent payment history," the server analyzes this request and retrieves the appropriate payment history information from the knowledge base. This information is then displayed to the representative via an augmented reality device, allowing the representative to immediately provide the information to the customer.

[0677] An example of a prompt message is, "After converting the audio data to text, retrieve the latest payment history information from the knowledge base and present it to the representative."

[0678] This system will enable faster and more accurate customer service, thereby increasing customer satisfaction.

[0679] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0680] Step 1:

[0681] The server receives an audio signal from the user. The audio, acquired via the audio input device, is then transferred to the server. The input is an audio signal, and the output is that audio signal stored on the server in digital format.

[0682] Step 2:

[0683] The server uses a speech recognition library (e.g., Google Speech-to-Text API) to convert the audio signal into text data. The input is the audio signal obtained in step 1, and the output is text data suitable for analysis. Here, data processing is performed to convert the audio signal into a string using an algorithm.

[0684] Step 3:

[0685] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data. The goal is to understand the content and intent of the inquiry and extract important keywords. The input is the text data generated in step 2, and the output is the analyzed information and related keywords. The AI ​​model performs natural language processing to analyze the text.

[0686] Step 4:

[0687] The server performs information retrieval based on the analysis results. It uses search functions to find relevant information from knowledge bases and databases. The input is the analysis results obtained in step 3, and the output is a collection of required information. Search algorithms are utilized, performing text matching and extracting related documents.

[0688] Step 5:

[0689] Based on the information acquired by the server, information is generated for the user to review. A prompt message is generated, and a response is created. The input is the information obtained in step 4, and the output is a displayable response message. A generation AI is used to naturally combine the information and generate the response.

[0690] Step 6:

[0691] The terminal displays the generated response on the augmented reality device. The server sends data to the display device, which the terminal can then verify. The input is the response message generated in step 5, and the output is the information visually displayed on the augmented reality device. Real-time data transfer and display are performed as part of the operation.

[0692] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0693] As an embodiment of the present invention, an autonomous predictive AI agent system incorporating an emotion engine that analyzes voice data from customers and identifies emotions is described. This system is designed to provide more accurate and emotionally empathetic responses to the customer service staff being assisted.

[0694] When a user makes an inquiry via telephone, the server receives the call and acquires the audio data. The server uses speech recognition technology to convert the acquired audio data into text data. This conversion ensures that the conversation is recorded in text format, allowing for smoother subsequent processing.

[0695] The server provides text data to the generating AI, which analyzes the inquiry. During this process, the emotion engine uses voice data to identify the user's emotions. For example, it analyzes parameters such as voice tone and speaking speed to determine whether the user is angry, confused, or calm. This emotion information, along with the inquiry category, becomes part of the analysis data.

[0696] Once the analysis is complete, the server uses the analysis results to search the database for the necessary information. This search includes past query history, similar cases, and relevant knowledge base information. The server collects this data so that the AI ​​can generate the most appropriate answer or recommended action.

[0697] The generation AI generates the optimal answer based on all the information it has acquired. During this process, the answer is adjusted according to the user's emotions, as identified by the emotion engine. For example, if the user is angry, it will generate a polite and calm-toned answer; if the user is confused, it will construct a more detailed and clear answer.

[0698] Finally, the server displays the generated response on the customer service representative's terminal. The representative can then quickly provide an appropriate response to the user while viewing the terminal. After the response is completed, the response details and emotional information confirmed on the terminal are automatically recorded by the server and organized as incident information. This entire process makes customer service more sophisticated and humane, contributing to improved customer satisfaction. For example, if a user angrily inquires about a billing error, the emotion engine can identify the anger, and the server can suggest a calmer response appropriate to that emotion.

[0699] The following describes the processing flow.

[0700] Step 1:

[0701] A user contacts the service desk by phone. The server receives the call and retrieves the audio data.

[0702] Step 2:

[0703] The server uses speech recognition technology to convert the acquired audio data into text data. This makes the conversation content analyzable in written form.

[0704] Step 3:

[0705] The server provides the converted text data to the generating AI, which then begins analysis. The generating AI extracts keywords from the text to understand the content of the inquiry.

[0706] Step 4:

[0707] The server simultaneously sends audio data to the emotion engine, which identifies the user's emotional state based on their voice tone, speaking speed, intonation, etc. The emotion engine then adds this to the analysis data.

[0708] Step 5:

[0709] The server searches the database based on the analyzed query content and sentiment information. It collects relevant query history and knowledge base information.

[0710] Step 6:

[0711] The generation AI generates appropriate responses based on database search results and sentiment information. For example, if the user is angry, a response in a calm tone will be generated.

[0712] Step 7:

[0713] The server sends the generated response and emotional considerations to the customer service representative's terminal for display. This allows the representative to respond to the user immediately.

[0714] Step 8:

[0715] After the response is complete, the server automatically records the details of the response and emotional information via the terminal and stores it in the database as incident information. This information will be used to improve future operations.

[0716] (Example 2)

[0717] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0718] Conventional information processing systems have faced challenges in adequately addressing customer inquiries emotionally, making it difficult to improve customer satisfaction. Furthermore, in the analysis of inquiry content, emotional information is often not considered, resulting in cases where optimal responses cannot be provided. This situation necessitates that customer service representatives provide prompt and accurate responses that take emotions into account.

[0719] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0720] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information and understanding the content of the inquiry, and means for searching for information resources related to the analysis results. This makes it possible to perform analysis that takes customer emotions into account and provide appropriate information.

[0721] "Audio information" refers to information that represents audio acquired from customers in a digital format.

[0722] "Text information" refers to information in string format obtained by converting audio information into a parseable form.

[0723] "Analysis means" refers to means of understanding text information and processing it to classify and convert the content of inquiries into knowledge.

[0724] A "search tool" is a means of referencing and obtaining relevant information resources based on the analysis results.

[0725] "Generation means" refers to means for constructing an appropriate response based on information obtained by search means.

[0726] "Emotion identification means" refers to a method for analyzing the characteristics of voice information to determine the emotional state of a customer.

[0727] "Display means" refers to means for visually showing the response obtained by the generation means to the terminal.

[0728] "Recording means" refers to a means of saving the results of the counter staff's interactions and related information, and organizing them in a way that allows for later reference.

[0729] "Information resources" refer to various sources of information, such as databases and knowledge bases, used for analyzing inquiries and generating responses.

[0730] This embodiment of the invention describes a system that receives customer inquiries via voice information and provides responses that take into account the user's emotions. A server receives calls from customers and acquires voice information. The acquired voice information is converted into text information using speech recognition technology. In this process, general speech recognition software is used, and for example, an open-source speech recognition library can be applied.

[0731] Next, the server analyzes the text information. This analysis utilizes a generative AI model to support the understanding of the inquiry. A general natural language processing framework can be used as the specific generative AI model. Furthermore, emotion recognition is used to identify the customer's emotions from the audio information. Based on this, the server determines whether the user's emotion falls under "anger," "joy," "sadness," or "calmness."

[0732] To retrieve information resources related to a user's query, the server references an internal database based on the analysis results to obtain the appropriate information. This typically involves a general database system, such as past query history and related knowledge bases.

[0733] Based on the generated information, a generative AI model constructs a response. This model also considers emotions identified by emotion recognition tools, and adjusts the tone to suit the user. The generated response is displayed on the terminal and used as a basis for service staff to provide appropriate assistance.

[0734] As an example, consider a case where a user angrily inquires that there is an error in their bill. The emotion recognition means efficiently detects anger, and the server uses this result to generate a calm and composed response using a generative AI model. In this way, high-quality responses that take emotions into consideration are possible, which is expected to contribute to improved customer satisfaction.

[0735] (Example of prompts for a generative AI model)

[0736] "Please generate the best possible response to inquiries regarding service interruptions, taking into account user sentiment."

[0737] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0738] Step 1:

[0739] When a user makes an inquiry via telephone, the server receives the call. The input is the user's voice data, which the server captures and manages. During this process, the data is temporarily stored in an audio file format. The output is the stored audio data.

[0740] Step 2:

[0741] The server converts the received audio data into text data. It analyzes the audio using speech recognition technology and converts it into a string format. Specifically, by using a speech recognition API, the input audio data is output as text data.

[0742] Step 3:

[0743] The server provides text data to a generating AI model, which then analyzes the query. The input is the generated text data. The AI ​​model uses natural language processing algorithms to understand the context and extracts the intent and content of the query as part of the analysis. The output is the analysis result.

[0744] Step 4:

[0745] The server uses the analysis results to identify the user's emotions using emotion recognition tools. The input is voice data and its feature extraction results. By analyzing the pitch, tone, and speaking speed of the voice, it generates emotion information as output. This information includes emotion categories such as "anger," "joy," and "calmness."

[0746] Step 5:

[0747] The server uses the analysis results and sentiment information to search the company's database and collect relevant information. The input is the analysis results and sentiment information. By executing database queries, it retrieves the necessary data from past query history and relevant knowledge bases. The output is relevant information based on the search results.

[0748] Step 6:

[0749] The generative AI model generates the optimal response based on collected information and sentiment data. The input consists of information and sentiment data obtained through searches. The generation process uses high-priority data to produce a structured response. The output is a suggested response to the user.

[0750] Step 7:

[0751] The server sends the generated response to the terminal and displays it on the counter staff's screen. The input is the generated response, which is visualized on the terminal. The counter staff refer to this output and take appropriate action in real time.

[0752] Step 8:

[0753] Once the interaction is complete, the interaction details and emotional information recorded on the terminal are sent to the server. The input consists of the interaction result and emotional information. Based on this, the server organizes the data and records it in the database as an incident report. The output is structured incident information.

[0754] (Application Example 2)

[0755] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0756] Traditional customer service systems struggle to accurately understand customers' emotional states and optimize responses. This can lead to systems generating mechanical responses and decreased customer satisfaction. Furthermore, in sensitive or urgent situations, appropriate responses may not be provided promptly. This hinders the building of trust with customers and results in lost business opportunities for companies.

[0757] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0758] In this invention, the server includes emotion identification means for identifying emotional states from voice information, analysis means for analyzing text information and understanding the content of inquiries, and adjustment means for adjusting the response according to the emotional state identified by the emotion identification means. This makes it possible to accurately grasp the customer's emotional state and provide an optimal response accordingly.

[0759] "Audio information" refers to linguistic information spoken by customers, captured as acoustic signals.

[0760] "Textual information" refers to data in text format obtained by converting audio information.

[0761] "Analysis means" refers to a device or program that has the function of analyzing textual information to understand the content of an inquiry.

[0762] A "search means" is a system that has the function of finding relevant recording media based on the analysis results obtained by the analysis means.

[0763] "Generation means" refers to a mechanism or software that has the function of creating an optimal response in accordance with the results of the search means.

[0764] "Display means" refers to a means for visually displaying the generated response on the counter staff member's device.

[0765] "Recording means" refers to a method or mechanism for long-term storage of the results of a customer service representative's interactions.

[0766] An "emotion identification means" is a device that has the function of identifying a customer's emotional state from voice information.

[0767] A "modification mechanism" is a system that has the function of adjusting the content of the response according to the customer's emotional state.

[0768] This invention is an autonomous predictive AI agent system for streamlining customer service. The server receives voice information from customers and converts it into text information using speech recognition software. For speech recognition, for example, the Google Speech Recognition API is used. The converted text information is analyzed by an analysis means to identify the emotional state associated with the inquiry. For the analysis, the Sentiment Analysis pipeline of Transformers, a natural language processing library, is used.

[0769] Based on the user's inquiry, the server searches for relevant information from its storage medium. This storage medium includes past inquiry history and a knowledge base. Based on the retrieved information and the emotional state identified by the emotion recognition system, a generative AI model generates the optimal response. The generated response is displayed on the customer service representative's terminal. This terminal facilitates the rapid and appropriate response to the user.

[0770] Furthermore, the server automatically records the results of the responses by the customer service representatives and saves them as a response history. This recording process ensures that incident information is managed systematically.

[0771] For example, if a user inquires about a security system malfunction, the emotion recognition system will determine that the user is distressed, and the generative AI model will provide a reassuring response. An example of a prompt would be, "In a situation where a customer is anxiously reporting a security issue, please show how to respond appropriately based on the tone the emotion engine perceives." This enables an emotionally sensitive response, contributing to improved customer satisfaction.

[0772] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0773] Step 1:

[0774] The server receives voice information from the user. This information is input into speech recognition software, which converts the voice signal into text information. This conversion process outputs the customer's inquiry as text data. Specifically, the Google Speech Recognition API is used to analyze the voice and convert it into the corresponding string.

[0775] Step 2:

[0776] The server uses the obtained text information as input and employs an analysis tool to identify the content of the inquiry and the user's emotional state. This analysis utilizes Transformers' Sentiment Analysis pipeline to analyze the text information and output the user's emotions. Specifically, it analyzes the word choices and context within the string to determine the emotional label (positive, negative, neutral).

[0777] Step 3:

[0778] The server searches the storage medium based on the user's query content identified through analysis. The storage medium outputs past history and knowledge base information related to the query. Specifically, it executes database queries to extract highly relevant documents and information.

[0779] Step 4:

[0780] The server uses an AI model to generate the optimal response, taking information obtained from the recording medium and the emotional state obtained through emotion recognition as input. The generated response is output as text data in the format most appropriate to the user's situation. In terms of operation, it constructs prompt sentences based on the emotional state and executes a process to generate appropriate response sentences.

[0781] Step 5:

[0782] The terminal displays the generated response sent from the server. The displayed response allows the service representative to quickly provide the user with an appropriate answer. This step involves visually presenting the response text on the display.

[0783] Step 6:

[0784] The server automatically records the interactions performed by the customer service representative using recording mechanisms. The interaction history is organized as incident information and used to improve future interactions. Specifically, it saves the response text, emotional state, and interaction result to a data store.

[0785] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0786] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0787] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0788] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0789] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0790] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0791] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0792] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0793] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0794] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0795] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0796] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0797] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0798] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0799] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0800] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0801] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0802] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0803] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0804] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0805] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0806] The following is further disclosed regarding the embodiments described above.

[0807] (Claim 1)

[0808] A means for receiving audio data from a customer and converting said audio data into text data,

[0809] An analysis means for analyzing the aforementioned text data and understanding the content of the inquiry,

[0810] A search means for searching a database related to the analysis results obtained based on the aforementioned analysis means,

[0811] A generation means that generates an appropriate answer based on the results of the search means,

[0812] A display means for displaying the response generated by the generation means on the terminal of the counter staff member,

[0813] A recording means for recording the results of the response by the counter staff member,

[0814] A system that includes this.

[0815] (Claim 2)

[0816] The system according to claim 1, further comprising means for identifying an inquiry category based on the text data analyzed by the analysis means.

[0817] (Claim 3)

[0818] The system according to claim 1, wherein the recording means automatically records the response results as incident information.

[0819] "Example 1"

[0820] (Claim 1)

[0821] A means for receiving audio information and converting said audio information into text information,

[0822] An analysis means for analyzing the aforementioned textual information, understanding the content of the inquiry, and extracting important words,

[0823] A means for further analyzing the analysis results obtained by the aforementioned analysis means based on past response records and knowledge databases, and for searching for similar past cases,

[0824] Means for generating an appropriate solution based on the results of the search means,

[0825] A means for displaying the generated solution on a display device,

[0826] A means for automatically saving the results of the actions taken using the aforementioned display device as recorded information,

[0827] A system that includes this.

[0828] (Claim 2)

[0829] The system according to claim 1, further comprising means for identifying the type of inquiry based on the character information analyzed by the analysis means.

[0830] (Claim 3)

[0831] The system according to claim 1, wherein the recorded information is stored as event information, with the corresponding results being preserved.

[0832] "Application Example 1"

[0833] (Claim 1)

[0834] A means for acquiring audio signals from customers and converting said audio signals into text data,

[0835] The aforementioned text data is analyzed using an analysis function to understand the content of the inquiry,

[0836] A search function for searching for information related to the analysis results obtained based on the aforementioned analysis function,

[0837] A generation function that generates an appropriate response based on the results of the aforementioned search function,

[0838] A display function that presents the response generated by the generation function to the user's display device,

[0839] A recording function for recording the results of the user's response,

[0840] A function for presenting the information displayed by the aforementioned display function to the person in charge on an augmented reality device,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, further comprising a function to identify the type of inquiry based on the character data analyzed by the aforementioned analysis function.

[0844] (Claim 3)

[0845] The system according to claim 1, wherein the recording function automatically records the response results as incident information and stores the information in a database for future use.

[0846] "Example 2 of combining an emotion engine"

[0847] (Claim 1)

[0848] A means for receiving voice information from a customer and converting said voice information into text information,

[0849] An analysis means for analyzing the aforementioned text information and understanding the content of the inquiry,

[0850] A search means for searching for information resources related to the analysis results obtained based on the aforementioned analysis means,

[0851] A generation means that generates an appropriate response based on the results of the search means,

[0852] An emotion identification method for identifying emotions from audio data,

[0853] A display means for displaying the response generated by the generation means on the terminal of the counter staff member,

[0854] A recording means for recording the results of the response by the counter staff member,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, further comprising means for identifying an inquiry category based on text information analyzed by the analysis means and sentiment information identified by the sentiment identification means.

[0858] (Claim 3)

[0859] The system according to claim 1, wherein the recording means automatically records the response results and emotional information as incident information.

[0860] "Application example 2 when combining with an emotional engine"

[0861] (Claim 1)

[0862] A means for receiving voice information from a customer and converting said voice information into text information,

[0863] An analysis means for analyzing the aforementioned textual information and understanding the content of the inquiry,

[0864] A search means for searching a recording medium related to the analysis results obtained based on the analysis means,

[0865] A generation means that generates an appropriate response based on the results of the search means,

[0866] A display means for displaying the response generated by the generation means on the counter staff's device,

[0867] A recording means for recording the results of the response by the counter staff member,

[0868] An emotion identification means for identifying an emotional state from the aforementioned audio information,

[0869] An adjustment means that adjusts the response according to the emotional state identified by the emotion identification means,

[0870] A system that includes this.

[0871] (Claim 2)

[0872] The system according to claim 1, further comprising means for identifying a query classification based on the character information analyzed by the analysis means.

[0873] (Claim 3)

[0874] The system according to claim 1, wherein the recording means automatically records the response results as incident information. [Explanation of Symbols]

[0875] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving audio data from a customer and converting said audio data into text data, An analysis means for analyzing the aforementioned text data and understanding the content of the inquiry, A search means for searching a database related to the analysis results obtained based on the aforementioned analysis means, A generation means that generates an appropriate answer based on the results of the search means, A display means for displaying the response generated by the generation means on the terminal of the counter staff member, A recording means for recording the results of the response by the counter staff member, A system that includes this.

2. The system according to claim 1, further comprising means for identifying an inquiry category based on the text data analyzed by the analysis means.

3. The system according to claim 1, wherein the recording means automatically records the response results as incident information.

Images