System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring a user's voice input; means for converting the acquired voice into text data; means for analyzing the converted text data and classifying consultation contents; means for routing to a specialized department based on the classified consultation contents; means for generating an automatic response message based on the classified consultation contents; means for converting the generated automatic response message into voice and responding to the user; and optionally means for summarizing the user's consultation contents and displaying the summary on an operator's terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When contacting a call center, customers have to press multiple buttons to select the topic of their inquiry, which is time-consuming and time-consuming. Furthermore, call centers are facing a shortage of staff and a high number of similar inquiries, making their work cumbersome. Furthermore, if customers are unable to clearly communicate the content of their inquiry, it becomes difficult to respond quickly. There is a need to solve these issues, improve the efficiency of call center operations, and increase customer convenience. [Means for solving the problem]

[0005] The present invention is a system that includes means for acquiring user voice input, speech recognition means for converting the acquired voice into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automatic response message based on the classified consultation content, means for converting the generated automatic response message into voice and responding to the user, and means for summarizing the user's consultation content and displaying it on an operator's terminal as needed. This allows users to communicate the consultation content without having to press any buttons, enabling call centers to respond efficiently.

[0006] "Means for acquiring user voice input" refers to a device or software interface for acquiring user-uttered voice as digital data.

[0007] "Speech recognition means" refers to hardware and software for analyzing acquired voice data and converting it into text format data.

[0008] "Converting into text data" refers to the process of converting voice data into a character string in a natural language using a voice recognition means.

[0009] "Means of analyzing and classifying the consultation content" refers to the process of analyzing text data based on specific rules and algorithms and sorting it into categories or segments according to the consultation content.

[0010] "Means for routing to specialized departments" refers to a mechanism for determining which specialized department should handle an inquiry based on the classified content of the inquiry and transferring the inquiry to that department.

[0011] "Means for generating automatic response messages" refers to algorithms or databases that automatically create appropriate responses based on the classified content of inquiries.

[0012] The "means for converting the automatic response message into voice" refers to a mechanism for converting the generated text-format automatic response message into voice data using voice synthesis technology.

[0013] "Means for responding to the user" refers to the hardware and software interfaces for transmitting the converted automated response message to the user.

[0014] "Means for summarizing and displaying on the operator's terminal" refers to a mechanism for summarizing the user's consultation content and displaying the summary on the terminal so that the operator can respond efficiently. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that uses voice recognition to improve the operational efficiency of call centers. This system uses voice recognition technology to appropriately classify customer inquiries, enabling automatic responses and efficient support for operators.

[0037] What the program does

[0038] 1. Capture user voice input

[0039] (User)

[0040] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0041] 2. Acquiring and transmitting audio data

[0042] (Terminal)

[0043] The terminal captures the user's voice and transmits the voice data to the server.

[0044] 3. Speech Recognition and Text Conversion

[0045] (server)

[0046] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text into "Internet connection not possible."

[0047] 4. Text Data Analysis and Classification

[0048] (server)

[0049] The server analyzes the text data and classifies the issue based on pre-defined rules and machine learning models. In this case, it is classified as a "connection problem."

[0050] 5. Routing to specialized departments

[0051] (server)

[0052] Route inquiries to network support departments based on the classified inquiry content.

[0053] 6. Generate Auto-Responses

[0054] (server)

[0055] The server generates an auto-reply message based on the classified inquiry, for example, "Please restart your modem."

[0056] 7. Speech conversion and response to the user

[0057] (server)

[0058] The generated automatic response message is converted into voice and responded to the user via the terminal.

[0059] 8. Summary and Operator Assistance

[0060] (server)

[0061] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. In some cases, the operator will respond to the user directly, but the summary information allows for a quicker response.

[0062] Specific examples

[0063] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the device and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" and routed to the network support department.

[0064] The server generates an automatic response message for a frequently asked question, such as "Please try restarting the modem," and converts it into voice to respond to the user. If the automatic response is not appropriate, the server summarizes the inquiry and displays it on the operator's terminal. The operator can respond quickly based on the displayed summary and related FAQs.

[0065] In this way, the system of the present invention makes full use of voice recognition and automatic response technology to enhance user convenience and significantly improve the operational efficiency of call centers.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0069] Step 2:

[0070] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0071] Step 3:

[0072] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[0073] Step 4:

[0074] The server analyzes the text data obtained from the speech recognition engine and classifies the consultation content using pre-set rules and machine learning models. In this example, it is classified as a "connection problem."

[0075] Step 5:

[0076] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[0077] Step 6:

[0078] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[0079] Step 7:

[0080] The server generates the prepared auto-answer message and converts it into voice using speech synthesis technology. Specifically, the voice message generated is "Please try restarting your modem."

[0081] Step 8:

[0082] The server sends a voice-converted automated response message to the terminal, which then relays it to the user, who receives the advice "Try restarting your modem."

[0083] Step 9:

[0084] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[0085] Step 10:

[0086] Based on the summary displayed on the terminal and related FAQs, the operator can provide the user with a quick and specific response, such as giving appropriate advice such as "Check the modem settings."

[0087] Through these steps, the system enables quick and accurate responses to user inquiries, while at the same time significantly improving the operational efficiency of call centers.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In conventional call centers, operators manually handle inquiries from users, which takes time and manpower, resulting in low operational efficiency. It was particularly difficult to respond quickly when multiple inquiries occurred simultaneously. Furthermore, there was a lack of technology to efficiently convert voice data into text data and properly classify the content, making it difficult to respond accurately or assign calls to the appropriate department.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for acquiring user voice input, means for transmitting the acquired voice to the server as digital data, speech recognition means for converting the acquired voice data into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automatic response message based on the classified consultation content, means for converting the generated automatic response message into voice and responding to the user, and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to process user inquiries quickly and accurately, and to provide an appropriate automatic response or route to a specialized department, thereby significantly improving business efficiency.

[0093] "Means for acquiring user voice input" refers to devices or software for receiving voice from the user, such as a microphone installed on a telephone or smartphone.

[0094] The "means for transmitting the captured audio as digital data to the server" refers to a communication means for converting analog audio into digital form and transmitting it to the server over a network. Specifically, this includes an audio signal converter and a communication protocol.

[0095] "Speech recognition means for converting acquired voice data into text data" refers to technology for analyzing voice and converting it into text information, and a voice recognition engine is an example of this.

[0096] "Means for analyzing the converted text data and classifying the consultation content" refers to algorithms or software that interpret the text data using natural language processing technology and classify it based on pre-set categories.

[0097] "Means of routing to specialized departments based on classified consultation content" refers to a system that automatically assigns inquiries to the appropriate department or person in charge based on the analysis results.

[0098] "A means for generating automatic response messages based on classified inquiry content" refers to a function that automatically creates responses to user inquiries by referring to an FAQ database or template.

[0099] "Means for converting the generated automated response message into speech and responding to the user" refers to technology for converting text-based messages into speech, such as a speech synthesis engine.

[0100] "Means of summarizing the user's inquiry content and displaying it on the operator's terminal as the case may be" refers to a function for concisely summarizing the user's inquiry data and displaying it in a way that the operator can quickly understand.

[0101] The present invention relates to a system that uses a voice recognition function to improve the operational efficiency of a call center. The system acquires voice input from a user, analyzes the voice, and responds appropriately.

[0102] Hardware and software used

[0103] Hardware

[0104] Devices: Phones, smartphones, PCs with headsets

[0105] Server: A server machine with a powerful processor and sufficient memory

[0106] software

[0107] Speech recognition engine: Technology that converts voice data into text data. For example, a general-purpose speech recognition API or in-house developed speech recognition software.

[0108] Natural language processing engine (NLP): A technology that analyzes text data and classifies the content of inquiries.

[0109] Speech synthesis engine: Technology that converts auto-response messages into speech, such as in-house developed speech synthesis software.

[0110] Database: A database system for storing FAQs and template answers.

[0111] Data processing and calculation

[0112] 1. Audio capture and transmission

[0113] A user calls a call center using a telephone or smartphone. The device picks up the user's voice through a microphone and sends the voice data to a server. The sent voice data is then passed to a speech recognition engine within the server.

[0114] 2. Speech Recognition and Text Conversion

[0115] The server uses a speech recognition engine to convert the voice data into text data. For example, the speech recognition engine converts the voice "I can't connect to the Internet" into text. This conversion process involves noise filtering and language modeling.

[0116] 3. Analysis and Classification of Text Data

[0117] The converted text data is passed to an NLP engine on the server, where its content is analyzed. The NLP engine then classifies the text data based on pre-defined rules and machine learning models. For example, it may be classified as a "connection problem."

[0118] 4. Routing to specialized departments

[0119] The server routes the inquiry to the appropriate department based on the classified inquiry content, and uses an internal routing algorithm to transfer the inquiry to the most appropriate department or person.

[0120] 5. Generate Auto-Response Messages

[0121] The server generates an automatic response message from a FAQ database or template based on the classified inquiry content, such as "Please restart your modem."

[0122] 6. Speech synthesis and response to the user

[0123] The generated auto-answer message is passed to a speech synthesis engine and converted into voice data, which is then sent to the user via the terminal in a voice response. For example, the message tells the user to "restart the modem."

[0124] 7. Summary and Operator Assistance

[0125] The server further analyzes the user's inquiry and provides a concise summary. This summary and related FAQs are displayed on the operator's terminal. The operator can then respond quickly based on the displayed summary information.

[0126] Specific examples

[0127] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" by an NLP engine and routed to the network support department. The server generates an automated response message from the FAQ database saying, "Please restart your modem," and responds to the user by converting it into speech using a speech synthesis engine. If the automated response is not appropriate, the server summarizes the consultation and displays it on the operator terminal. The operator can respond based on the displayed summary.

[0128] Example prompts for generative AI models

[0129] A user uses their smartphone and says, "I can't connect to the Internet."

[0130] The server uses a speech recognition engine to convert the speech into text data.

[0131] The text data is classified by the NLP engine as a "connection issue" and routed to the network support department.

[0132] The server generates an automatic response message from the FAQ database and converts "Please restart your modem" into speech using a speech synthesis engine.

[0133] The terminal plays the audio data and responds to the user.

[0134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0135] Step 1:

[0136] The user begins speaking

[0137] The user calls the call center using a telephone or smartphone and verbally communicates the inquiry through a microphone.

[0138] Input: User speech (e.g. "I can't connect to the Internet")

[0139] Output: Audio data (analog signal)

[0140] Step 2:

[0141] The device converts the voice data into digital data and sends it to the server.

[0142] The device converts the audio picked up by the microphone into a digital format and transmits it to a server using a secure communication protocol (e.g., HTTPS).

[0143] Input: Audio data (analog signal)

[0144] Output: Digital audio data

[0145] Step 3:

[0146] The server receives the voice data and passes it to the voice recognition engine.

[0147] The server passes the received digital voice data to a voice recognition engine (e.g., a general-purpose voice recognition API) and converts the voice into text data. It also performs preprocessing such as noise filtering.

[0148] Input: Digital audio data

[0149] Output: Text data (e.g. "Internet connection not available")

[0150] Step 4:

[0151] The server analyzes the text data and classifies the content of the consultation.

[0152] The server uses an NLP engine to analyze the text data, and based on the analysis, classifies the text data based on pre-defined rules and machine learning models.

[0153] Input: Text data (e.g. "Cannot connect to the Internet")

[0154] Output: Classification result (e.g. "connectivity problem")

[0155] Step 5:

[0156] The server routes the request to a specialist department based on the content of the request.

[0157] The server automatically routes inquiries to the most appropriate department based on the classification results, using a routing algorithm to forward them to the network support department, etc.

[0158] Input: Classification result (e.g. "Connectivity problem")

[0159] Output: Routing instructions (e.g., Network Support Department)

[0160] Step 6:

[0161] The server generates an auto-reply message

[0162] The server generates an appropriate auto-reply message from a FAQ database or template, such as "Please restart your modem."

[0163] Input: Classification result (e.g. "Connectivity problem")

[0164] Output: Auto-answer message (e.g. "Please restart your modem")

[0165] Step 7:

[0166] The server converts the auto-answer message into voice and responds to the user.

[0167] The generated automated response message is passed to a speech synthesis engine (e.g., speech synthesis software) and converted into voice data. This voice data is then sent to the terminal, which then responds to the user.

[0168] Input: Auto-answer message (e.g. "Please restart your modem")

[0169] Output: Audio data

[0170] Step 8:

[0171] The server summarizes the user's consultation and displays it on the operator's terminal.

[0172] The server further analyzes and summarizes the user's inquiry, and displays the summary information, including related FAQs, on the operator's terminal, helping the operator respond quickly.

[0173] Input: User inquiry information, related FAQ

[0174] Output: Summary, operator display information

[0175] By implementing the above steps, the system of the present invention can efficiently process user inquiries and enable automatic responses and rapid routing to specialized departments, thereby significantly improving the operational efficiency of call centers and enhancing user convenience.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Conventional self-driving vehicles lack a system that allows passengers to smoothly respond to their requests and questions. This often results in a lack of convenience and comfort for passengers inside the vehicle. Another problem is that there are insufficient means to quickly obtain emergency or specific information. To solve these issues, a highly functional support system that uses voice input is needed.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for acquiring a user's voice input, speech recognition means for converting the acquired voice into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automated response message based on the classified consultation content, means for converting the generated automated response message into voice and responding to the user, means for summarizing the user's consultation content and displaying it on an operator's terminal as needed, and means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle. This enables passengers to receive prompt and appropriate responses to their inquiries and requests within the autonomous vehicle.

[0181] "Means for obtaining user voice input" refers to devices or software that recognize voice signals emitted by the user and capture them as digital data.

[0182] "Speech recognition means for converting acquired speech into text data" refers to technology that analyzes speech signals and converts them into corresponding text data, and is generally composed of a microphone and a speech recognition engine.

[0183] The "means for analyzing the converted text data and classifying the consultation content" is a method for analyzing the meaning of the text data using natural language processing technology and classifying the consultation content into predefined categories.

[0184] "Means for routing to specialized departments based on classified consultation content" is a system for automatically distributing inquiries to appropriate specialized departments based on the analyzed consultation content.

[0185] The "means for generating automatic response messages" is a system that automatically creates appropriate response messages in response to user inquiries based on a database or template responses.

[0186] The "means for converting the generated automated response message into speech and responding to the user" refers to a technology for converting the text-based automated response message into speech and providing audible feedback to the user.

[0187] The "means for summarizing the user's consultation content and displaying it on the operator's terminal" is a system that has the function of concisely summarizing the user's consultation content and displaying it on the operator's display device.

[0188] "Means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle" refers to a technology that transmits voice instructions given within the vehicle to an in-vehicle server in real time and provides a comfortable in-vehicle environment based on that.

[0189] This invention is a system for significantly improving passenger convenience in an autonomous vehicle. A specific implementation method of this system will be described below.

[0190] Overall system overview

[0191] The server receives voice input from the user through the infotainment system in the vehicle and generates appropriate responses. The system includes the following main functions:

[0192] Acquiring voice input

[0193] Speech-to-text

[0194] Text Analysis and Classification

[0195] Routing to specialized departments

[0196] Generate auto-response messages

[0197] Speech transcription and response

[0198] Summary of user inquiries and operator support

[0199] Passenger voice commands are sent to an in-vehicle server to provide convenience within the vehicle.

[0200] Hardware and software used

[0201] 1. Hardware:

[0202] Autonomous vehicle infotainment systems

[0203] In-vehicle server

[0204] 2. Software:

[0205] Speech recognition engine: To convert voice input into text data, for example, Amazon Transcribe or Google® Cloud Speech-to-Text are used.

[0206] Natural Language Processing: For analyzing text data and classifying content, use AWS® SageMaker and NLTK, for example.

[0207] Response message generation engine: Refers to FAQs and template answers to automatically create appropriate response messages.

[0208] Text conversion engine: For example, using Amazon Polly or Google Cloud Text-to-Speech to convert the generated text response into speech.

[0209] Example

[0210] 1. Acquiring voice input:

[0211] A passenger issues a voice command such as, "Where is the next service area?" This speech is picked up by the infotainment system.

[0212] 2. Speech to text transcription:

[0213] The acquired voice data is sent to an in-vehicle server and converted into text data such as "Where is the next service area?" by a voice recognition engine (e.g., Amazon Transcribe, Google Cloud Speech-to-Text).

[0214] 3. Text Analysis and Classification:

[0215] The server uses natural language processing technology (e.g., AWS SageMaker, NLTK) to analyze the text data and classify it into categories called "nearby facility information."

[0216] 4. Routing to specialized departments:

[0217] Based on the classification, the call is routed to a specialized department (e.g., vehicle support center) if necessary.

[0218] 5. Generate auto-response messages:

[0219] The server refers to FAQs and template answers in the database and generates an automated response message saying, "The next service area is approximately 15 kilometers away."

[0220] 6. Voice conversion and response:

[0221] The generated automated response message is converted into speech using a text conversion engine (e.g., Amazon Polly, Google Cloud Text-to-Speech) and provided to passengers through the infotainment system.

[0222] 7. Summary of user inquiries and operator support:

[0223] If the consultation is complicated, the server will summarize the content and display it on the operator's terminal along with related information, allowing the operator to respond quickly.

[0224] Examples and prompts

[0225] Example: When a passenger asks, "The car is shaking a little. Is everything okay?", the system automatically responds, "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[0226] Example prompt sentence:

[0227] Where is the next service area?

[0228] The car is shaking a little, is everything okay?

[0229] As a result, the system of the present invention can increase passenger convenience and enable comfortable travel in an autonomous vehicle.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] The user provides voice input. The passenger speaks a question or request. For example, "Where is the next service area?" The input is the user's voice, and the infotainment system captures the voice data. The output is the voice data.

[0233] Step 2:

[0234] The infotainment system sends the voice data to the in-vehicle server, which receives it in digital format. The input is the voice data, and the output is the voice data in digital format.

[0235] Step 3:

[0236] The in-vehicle server passes the voice data to a voice recognition engine, which converts the voice into text data. The software used is Amazon Transcribe or Google Cloud Speech-to-Text. The input is voice data, which is processed by voice recognition, and the output is text data. Specifically, the text data obtained is, "Where is the next service area?"

[0237] Step 4:

[0238] The server analyzes the text data and classifies the consultation content based on pre-defined rules and machine learning models. The software used is AWS SageMaker and NLTK. The input is text data, an automatic classification algorithm is applied, and the output is a category such as "information on nearby facilities."

[0239] Step 5:

[0240] The server routes inquiries to specialized departments as needed based on the classified content. The input is the classification result, and the output is routing information. For example, if the inquiry is for "information on nearby facilities," the inquiry will be routed to a specific database or department.

[0241] Step 6:

[0242] The server generates an automatic response message based on the classified inquiry content, referencing FAQs and template answers in the database. The input is the category of the inquiry content and information in the database, and the output is the message "The next service area is approximately 15 kilometers away."

[0243] Step 7:

[0244] The generated automated response message is converted into speech and responded to the user via the infotainment system. The software used is Amazon Polly and Google Cloud Text-to-Speech. The input is a text message, which is converted into speech, and the output is a speech response.

[0245] Step 8:

[0246] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. The input is text data and analysis results, and the output is summary information and display content. As a specific example, in an emergency, a consultation such as "The car is shaking a little. Are you OK?" is summarized and displayed to the operator as "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[0247] Through the above steps, smooth responses and services based on passenger voice input are realized.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] This invention relates to a system that improves the operational efficiency of call centers by using voice recognition functions, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more appropriate responses. This system uses voice recognition technology and emotion recognition technology to understand the content of the user's inquiry and their emotional state, and provides appropriate automatic responses and support to operators.

[0250] What the program does

[0251] 1. Capture user voice input

[0252] (User)

[0253] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0254] 2. Acquiring and transmitting audio data

[0255] (Terminal)

[0256] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0257] 3. Speech Recognition and Text Conversion

[0258] (server)

[0259] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text data into "Cannot connect to the Internet."

[0260] 4. Emotion Recognition by Emotion Engine

[0261] (server)

[0262] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[0263] 5. Text Data Analysis and Classification

[0264] (server)

[0265] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[0266] 6. Routing to specialized departments

[0267] (server)

[0268] Based on the classified inquiry, the user's inquiry is routed to the appropriate specialist department. In this case, it is routed to the network support department as a "connection problem."

[0269] 7. Generate Auto-Responses

[0270] (server)

[0271] The server generates an automatic response message based on the classified consultation content and also refers to the emotion recognition results. For example, if the user is angry, the tone will be adjusted to say, "We apologize for the inconvenience. Please try restarting your modem."

[0272] 8. Speech conversion and response to the user

[0273] (server)

[0274] The generated automated answering message is converted into voice and responded to the user via the terminal, who receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[0275] 9. Summary and Operator Assistance

[0276] (server)

[0277] The server further analyzes and summarizes the user's inquiry. This summary, along with the emotion recognition results and related FAQs, is displayed on the operator's terminal. In some cases, the operator will respond directly to the user, but the summary information and emotional state provided enable a quick and appropriate response.

[0278] Specific examples

[0279] A user calls a call center and says, "I can't connect to the Internet." This voice is picked up by the device and sent to the server. The server uses a speech recognition engine to convert the voice into text data saying, "I can't connect to the Internet." At the same time, an emotion engine recognizes the emotion "impatience" from the user's voice data. The converted text data is classified as a "connection problem" and routed to the network support department.

[0280] The server, taking into account the emotion recognition results, generates an automatic response message such as "Please try restarting the modem," converts it into speech, and responds to the user. If the automatic response is not appropriate, the server summarizes the consultation content and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state.

[0281] In this way, the system of the present invention makes full use of voice recognition and emotion recognition technologies to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that correspond to the user's emotions.

[0282] The processing flow will be explained below.

[0283] Step 1:

[0284] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0285] Step 2:

[0286] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0287] Step 3:

[0288] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[0289] Step 4:

[0290] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[0291] Step 5:

[0292] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[0293] Step 6:

[0294] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[0295] Step 7:

[0296] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[0297] Step 8:

[0298] The server generates a prepared automated response message while referring to the emotion recognition results. If the user is in a state of impatience, it creates a tone-adjusted message such as "We apologize for the inconvenience. Please try restarting your modem."

[0299] Step 9:

[0300] The server converts the generated automated response message into voice and responds to the user via the terminal. Specifically, the voice message "We apologize for the inconvenience. Please try restarting your modem" is transmitted to the user.

[0301] Step 10:

[0302] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[0303] Step 11:

[0304] The server displays the user's emotional state (e.g., impatience, anger, sadness, etc.) obtained by the emotion engine on the operator's terminal, allowing the operator to understand the user's emotional state and respond accordingly.

[0305] Step 12:

[0306] Based on the summary displayed on the terminal, relevant FAQs, and the user's emotional state, the operator can provide the user with a quick and specific response, such as "Check your modem settings."

[0307] In this way, this system makes full use of voice recognition and emotion recognition technology to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that reflect the user's emotions.

[0308] Example 2

[0309] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] Modern call center operations require accurate understanding of user inquiries and prompt and appropriate responses. However, current systems lack the ability to recognize the user's emotional state, which can lead to lower user satisfaction. In particular, when a user is anxious or irritated, it is difficult to respond appropriately, increasing the burden on operators.

[0311] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for acquiring a user's voice input; speech recognition means for converting the acquired voice into text data; emotion recognition means for recognizing the user's emotional state from the voice data; means for adjusting the tone of the automatic response message based on the emotion recognition result; means for analyzing the converted text data and classifying the consultation content; means for routing to a specialized department based on the classified consultation content; means for generating an automatic response message based on the classified consultation content; means for converting the generated automatic response message into voice and responding to the user; and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to accurately grasp not only the content of the user's inquiry but also their emotional state, enabling a prompt and appropriate response.

[0312] A "user" is a person who contacts the call center to make an inquiry or ask for advice.

[0313] "Audio input" refers to audio signals emitted by a user.

[0314] A "terminal" is a device used by a user, and is a device that acquires voice data and transmits it to a server.

[0315] "Speech recognition" refers to the process of analyzing voice data and converting it into text data.

[0316] "Emotion recognition" is a technology that analyzes characteristics such as a user's voice tone, speed, and intonation to determine their emotional state.

[0317] An "automatic response message" is an automatic response text generated by the server to a user's inquiry.

[0318] "Text-to-speech" refers to the process of converting text data into audio data.

[0319] "Routing" is a method of transferring inquiries to appropriate specialist departments based on the classified content of the inquiries.

[0320] "Categorizing consultation content" is the process of analyzing the content of a user's inquiry and classifying it into pre-defined categories.

[0321] "Operator support" is a function that provides the operator with the user's consultation content and emotional state, and supports a quick and appropriate response.

[0322] This invention relates to a system that combines speech recognition technology and emotion recognition technology to improve the operational efficiency of call centers. When a user makes an inquiry to a call center, the system understands the inquiry content and emotional state of the user, and provides appropriate automated responses and support to operators, thereby achieving a fast and accurate response.

[0323] This system is composed of the following elements: users, terminals, and servers. Specifically, it uses the following hardware and software:

[0324] 1. Acquiring voice input

[0325] (User)

[0326] Users contact the call center using devices such as telephones or smartphones. For example, if a user cannot connect to the Internet at home, they will say, "I can't connect to the Internet."

[0327] 2. Acquiring and sending audio data

[0328] (Terminal)

[0329] The device picks up the user's voice using a built-in microphone or an externally connected headset, and the voice data is recorded as digital data in real time and immediately sent to the server.

[0330] 3. Speech Recognition and Text Conversion

[0331] (server)

[0332] The server passes the received voice data to a speech recognition engine (for example, Google Cloud Speech-to-Text API). This engine converts the voice data into text data. Specifically, it converts it into text data that says, "Internet connection not available."

[0333] 4. Emotion recognition

[0334] (server)

[0335] The server uses an emotion recognition engine (e.g., IBM Watson® Tone Analyzer) to analyze the voice data for features such as the user's voice tone, rate, and intonation, thereby determining the user's emotional state (e.g., anger, sadness, impatience).

[0336] 5. Text Data Analysis and Classification

[0337] (server)

[0338] The server analyzes the text data using a natural language processing engine (e.g., OpenAI's GPT-3®). The engine extracts intent and keywords from the text and classifies them into predefined categories (e.g., "connection problems").

[0339] 6. Routing to specialized departments

[0340] (server)

[0341] The server routes the user's inquiry to the appropriate department based on the classification of the problem, for example, forwarding it to the network support department as a "connection problem."

[0342] 7. Generate Auto-Responses

[0343] (server)

[0344] The server generates an automatic response message based on the classified consultation content and the emotion recognition results. For example, if the user is angry, the server generates a message saying, "We apologize for the inconvenience. Please try restarting your modem."

[0345] 8. Speech conversion and response to the user

[0346] (server)

[0347] The generated automated response message is converted into speech using a speech synthesis engine (e.g., Amazon Polly) and responded to the user via the terminal. The user receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[0348] 9. Summary and Operator Assistance

[0349] (server)

[0350] The server further analyzes the user's inquiry and creates a summary. This summary, along with the emotion recognition results and related FAQs, are displayed on the operator's terminal. This allows the operator to respond quickly based on the displayed summary and the user's emotional state.

[0351] Specific examples

[0352] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert the speech into text data saying, "I can't connect to the Internet." At the same time, an emotion recognition engine recognizes the emotion "impatience" from the user's speech data. The converted text data is classified as a "connection problem" and routed to the network support department. The server, taking into account the emotion recognition results, generates an automated response message saying, "Please try restarting your modem," converts this into speech, and responds to the user. If the automated response is inappropriate, the server summarizes the consultation and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state. This system increases user convenience and improves call center efficiency.

[0353] Prompt Sentence Examples

[0354] Create a program that recognizes user questions and emotions in a call center system. Explain with a concrete example how the system would respond if a user says, "I can't connect to the Internet."

[0355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0356] Step 1:

[0357] The user calls the call center and speaks their inquiry, for example, "I can't connect to the Internet." This voice input is sent from the user to the terminal.

[0358] Step 2:

[0359] The device captures the user's voice using a built-in microphone or an externally connected headset. The captured voice data is recorded as digital data in real time. This digital voice data is then sent from the device to the server.

[0360] Step 3:

[0361] The server passes the received voice data to a speech recognition engine. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data. In this process, the voice waveform is analyzed and converted into a string of characters based on a language model. The output text data is, for example, "Internet connection not available."

[0362] Step 4:

[0363] The server passes the received voice data to an emotion recognition engine. Specifically, it uses IBM Watson Tone Analyzer to analyze the voice tone, speed, and intonation characteristics. This analysis determines the user's emotional state. The output data includes emotional states such as "impatience."

[0364] Step 5:

[0365] The server passes the text data to a natural language processing engine for analysis. Specifically, it uses OpenAI's GPT-3 to extract intent and keywords from the text. Based on the input text, it classifies it into categories such as "connection problems." The output of this process is called a "connection problem."

[0366] Step 6:

[0367] The server routes the call to the appropriate department based on the classified text data, for example, "Connection Problems" and forwards it to the network support department. A custom rules-based routing engine is used to determine the route to the department.

[0368] Step 7:

[0369] The server references the classified text data and emotion recognition results to generate an automated response message. This uses FAQs and template answers stored in the database. The inputs are "Connection problem" and "Impatience," and the output is a message that reads, "We apologize for the inconvenience. Please try restarting your modem."

[0370] Step 8:

[0371] The server passes the generated automated response message to a speech synthesis engine and converts it into voice data. Specifically, Amazon Polly is used to convert the text message into voice. The generated voice data is sent to the user via the terminal. The user receives a voice response saying, "We apologize for the inconvenience. Please try restarting your modem."

[0372] Step 9:

[0373] The server further analyzes and summarizes the user's consultation content and the emotion recognition results. The created summary information, emotional state, and related FAQs are displayed on the operator's terminal. This allows the operator to respond quickly based on the summary content and emotional state. Specifically, the operator's screen displays the message, "The user is anxious about an Internet connection problem. We advise you to try restarting the modem."

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional security systems have had issues with users reporting abnormalities late and being unable to respond appropriately in emergencies. Furthermore, conventional voice recognition systems are unable to take into account the user's emotional state, making it difficult to respond quickly to highly urgent situations. Furthermore, because reporting to appropriate external agencies is not automated, prompt assistance may not be available.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0378] In this invention, the server includes an emotion recognition means for identifying an emotional state from the user's voice data, a means for determining an optimal response based on the identified emotion, and a means for detecting an emergency and notifying an appropriate external agency, thereby enabling the user's emergency to be quickly detected and an appropriate response to be taken.

[0379] "Voice input" refers to the user speaking into a device.

[0380] "Speech recognition" refers to the technology of analyzing captured speech as digital data and converting it into text data.

[0381] "Text data" refers to text information converted using voice recognition technology.

[0382] "Emotion recognition" refers to the technology of identifying a user's emotional state from voice data.

[0383] "Emotional state" is information that indicates the mental state of the user, and specifically refers to anger, sadness, impatience, and the like.

[0384] "Rational response" refers to the optimal response generated based on the classified consultation content and the user's emotional state.

[0385] A "specialized department" refers to a department that handles specific consultation matters.

[0386] "Routing" refers to the series of processes that send classified consultation content to the appropriate department or external agency.

[0387] An "automatic response message" refers to an automatically generated reply message to a user.

[0388] "Text-to-speech" refers to the process of converting text messages back into audio data.

[0389] An "emergency" is any situation that requires immediate action.

[0390] "External agencies" refer to support organizations outside the system, such as police and security companies.

[0391] This system combines voice recognition technology and emotion recognition technology to enhance user convenience and respond quickly and appropriately to emergency situations.

[0392] First, when a user speaks into a smart device (smartphone, smart glasses, head-mounted display, etc.), the device picks up the user's voice and sends it as digital data to a cloud server. The voice data is collected in real time and sent to the server immediately.

[0393] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the captured speech into text data. Once the text data is generated, an emotion recognition engine (e.g., Microsoft® Azure® Emotion API) analyzes the speech data and identifies the user's emotional state. Emotional states include anger, sadness, impatience, and fear.

[0394] The generated text data and emotion recognition results are then used to classify the user's consultation and, if necessary, automatically notify the appropriate external agency (such as the police or security company). For example, if a user says, "This is an emergency, someone please help," the emotion recognition engine will determine this as "fear," and based on that, a call will be made immediately to the security company or police.

[0395] Furthermore, the server generates an automatic response message for the user to ensure safety. This generated message is converted into voice and returned to the user through the device. For example, an automatic response message such as "The police have been notified. Please evacuate to a safe place immediately" may be generated and notify the user.

[0396] This allows users to receive prompt and appropriate responses, allowing them to deal with emergencies with peace of mind. In addition, if operator assistance is required, the server will summarize the user's consultation and display it on the operator's terminal along with their emotional state, allowing the operator to respond promptly according to the situation.

[0397] Examples of concrete examples and prompts

[0398] For example, if a user suddenly discovers an intruder in their home and speaks into the smart glasses, "This is an emergency, someone please help me," the voice will be sent to the server in real time, and based on the converted text and emotion recognition results, the police will be immediately notified, and a message will be generated stating, "The police have been notified. Please evacuate to a safe place immediately."

[0399] Prompt Sentence Examples

[0400] "When a user speaks into a smart device, saying, 'This is an emergency, someone please help,' the voice data is sent to a cloud server, where a speech recognition engine converts it into text data. The emotion recognition engine then recognizes the user's emotional state as 'fear,' and based on that, calls the police and generates an automatic response to the user saying, 'The police have been called. Please evacuate to a safe place immediately.'"

[0401] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0402] Step 1:

[0403] A user speaks to a smart device, for example, "This is an emergency, please help me." Voice data is acquired as input. This voice data is captured through the device's microphone.

[0404] Step 2:

[0405] The device captures audio data, records it as digital data in real time, and sends it to a cloud server. The captured audio data is the input, and the digital data is sent to the cloud server as the output. Specifically, the device digitizes the audio signal and transmits it over the network.

[0406] Step 3:

[0407] The server passes the received voice data to a voice recognition engine. The input is digital voice data, and the voice recognition engine (e.g., Google Cloud Speech-to-Text API) converts the voice into text data. The output is text data. Specifically, the voice waveform is analyzed and words are identified.

[0408] Step 4:

[0409] The server passes the text data to an emotion recognition engine. The input is the text data and speech features, and the emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's speech tone, speed, and intonation to identify their emotional state. The output is the user's emotional state. Specifically, emotion feature vectors are extracted and classified.

[0410] Step 5:

[0411] The server analyzes the user's consultation content based on the generated text data and emotion recognition results, and classifies it as necessary. The input is text data and emotional state, and the machine learning model analyzes the consultation content and classifies it into the appropriate category. The output is the classification result. Specifically, a text classifier, a natural language processing algorithm, is used.

[0412] Step 6:

[0413] Based on the classified consultation content, the server detects an emergency and notifies the necessary external agencies (e.g., police or security companies). The inputs are the classification result and emotional state, and the urgency is evaluated to generate routing information to external agencies. The output is a report message that is generated and sent. Specifically, a report is sent to an external system via an API.

[0414] Step 7:

[0415] The server generates an automatic response message for the user to ensure safety. The inputs are the classification result and the emotional state, and a template response message is selected from a database and adjusted. The output is the generated automatic response message. Specifically, a condition-based message generation algorithm is executed.

[0416] Step 8:

[0417] The generated automated response message is converted into voice and responded to the user via the terminal. The input is the generated text message, and the voice conversion engine converts the text into voice data. The output is a voice message that is conveyed to the user. Specifically, voice synthesis technology is used.

[0418] Step 9:

[0419] Depending on the situation, the server summarizes the user's consultation content and the emotion recognition results, and displays them on the operator's terminal. The input is text data and emotional state, and a summary is generated using a text summarization algorithm. The output is the summary information displayed on the operator's terminal. Specifically, the summary is generated using natural language processing technology.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0421] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0423] [Second embodiment]

[0424] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0425] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0426] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0428] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0430] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0431] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0432] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0435] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0436] This invention relates to a system that uses voice recognition to improve the operational efficiency of call centers. This system uses voice recognition technology to appropriately classify customer inquiries, enabling automatic responses and efficient support for operators.

[0437] What the program does

[0438] 1. Capture user voice input

[0439] (User)

[0440] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0441] 2. Acquiring and transmitting audio data

[0442] (Terminal)

[0443] The terminal captures the user's voice and transmits the voice data to the server.

[0444] 3. Speech Recognition and Text Conversion

[0445] (server)

[0446] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text into "Internet connection not possible."

[0447] 4. Text Data Analysis and Classification

[0448] (server)

[0449] The server analyzes the text data and classifies the issue based on pre-defined rules and machine learning models. In this case, it is classified as a "connection problem."

[0450] 5. Routing to specialized departments

[0451] (server)

[0452] Route inquiries to network support departments based on the classified inquiry content.

[0453] 6. Generate Auto-Responses

[0454] (server)

[0455] The server generates an auto-reply message based on the classified inquiry, for example, "Please restart your modem."

[0456] 7. Speech conversion and response to the user

[0457] (server)

[0458] The generated automatic response message is converted into voice and responded to the user via the terminal.

[0459] 8. Summary and Operator Assistance

[0460] (server)

[0461] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. In some cases, the operator will respond to the user directly, but the summary information allows for a quicker response.

[0462] Specific examples

[0463] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the device and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" and routed to the network support department.

[0464] The server generates an automatic response message for a frequently asked question, such as "Please try restarting the modem," and converts it into voice to respond to the user. If the automatic response is not appropriate, the server summarizes the inquiry and displays it on the operator's terminal. The operator can respond quickly based on the displayed summary and related FAQs.

[0465] In this way, the system of the present invention makes full use of voice recognition and automatic response technology to enhance user convenience and significantly improve the operational efficiency of call centers.

[0466] The processing flow will be explained below.

[0467] Step 1:

[0468] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0469] Step 2:

[0470] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0471] Step 3:

[0472] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[0473] Step 4:

[0474] The server analyzes the text data obtained from the speech recognition engine and classifies the consultation content using pre-set rules and machine learning models. In this example, it is classified as a "connection problem."

[0475] Step 5:

[0476] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[0477] Step 6:

[0478] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[0479] Step 7:

[0480] The server generates the prepared auto-answer message and converts it into voice using speech synthesis technology. Specifically, the voice message generated is "Please try restarting your modem."

[0481] Step 8:

[0482] The server sends a voice-converted automated response message to the terminal, which then relays it to the user, who receives the advice "Try restarting your modem."

[0483] Step 9:

[0484] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[0485] Step 10:

[0486] Based on the summary displayed on the terminal and related FAQs, the operator can provide the user with a quick and specific response, such as giving appropriate advice such as "Check the modem settings."

[0487] Through these steps, the system enables quick and accurate responses to user inquiries, while at the same time significantly improving the operational efficiency of call centers.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] In conventional call centers, operators manually handle inquiries from users, which takes time and manpower, resulting in low operational efficiency. It was particularly difficult to respond quickly when multiple inquiries occurred simultaneously. Furthermore, there was a lack of technology to efficiently convert voice data into text data and properly classify the content, making it difficult to respond accurately or assign calls to the appropriate department.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes means for acquiring user voice input, means for transmitting the acquired voice to the server as digital data, speech recognition means for converting the acquired voice data into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automatic response message based on the classified consultation content, means for converting the generated automatic response message into voice and responding to the user, and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to process user inquiries quickly and accurately, and to provide an appropriate automatic response or route to a specialized department, thereby significantly improving business efficiency.

[0493] "Means for acquiring user voice input" refers to devices or software for receiving voice from the user, such as a microphone installed on a telephone or smartphone.

[0494] The "means for transmitting the captured audio as digital data to the server" refers to a communication means for converting analog audio into digital form and transmitting it to the server over a network. Specifically, this includes an audio signal converter and a communication protocol.

[0495] "Speech recognition means for converting acquired voice data into text data" refers to technology for analyzing voice and converting it into text information, and a voice recognition engine is an example of this.

[0496] "Means for analyzing the converted text data and classifying the consultation content" refers to algorithms or software that interpret the text data using natural language processing technology and classify it based on pre-set categories.

[0497] "Means of routing to specialized departments based on classified consultation content" refers to a system that automatically assigns inquiries to the appropriate department or person in charge based on the analysis results.

[0498] "A means for generating automatic response messages based on classified inquiry content" refers to a function that automatically creates responses to user inquiries by referring to an FAQ database or template.

[0499] "Means for converting the generated automated response message into speech and responding to the user" refers to technology for converting text-based messages into speech, such as a speech synthesis engine.

[0500] "Means of summarizing the user's inquiry content and displaying it on the operator's terminal as the case may be" refers to a function for concisely summarizing the user's inquiry data and displaying it in a way that the operator can quickly understand.

[0501] The present invention relates to a system that uses a voice recognition function to improve the operational efficiency of a call center. The system acquires voice input from a user, analyzes the voice, and responds appropriately.

[0502] Hardware and software used

[0503] Hardware

[0504] Devices: Phones, smartphones, PCs with headsets

[0505] Server: A server machine with a powerful processor and sufficient memory

[0506] software

[0507] Speech recognition engine: Technology that converts voice data into text data. For example, a general-purpose speech recognition API or in-house developed speech recognition software.

[0508] Natural language processing engine (NLP): A technology that analyzes text data and classifies the content of inquiries.

[0509] Speech synthesis engine: Technology that converts auto-response messages into speech, such as in-house developed speech synthesis software.

[0510] Database: A database system for storing FAQs and template answers.

[0511] Data processing and calculation

[0512] 1. Audio capture and transmission

[0513] A user calls a call center using a telephone or smartphone. The device picks up the user's voice through a microphone and sends the voice data to a server. The sent voice data is then passed to a speech recognition engine within the server.

[0514] 2. Speech Recognition and Text Conversion

[0515] The server uses a speech recognition engine to convert the voice data into text data. For example, the speech recognition engine converts the voice "I can't connect to the Internet" into text. This conversion process involves noise filtering and language modeling.

[0516] 3. Analysis and Classification of Text Data

[0517] The converted text data is passed to an NLP engine on the server, where its content is analyzed. The NLP engine then classifies the text data based on pre-defined rules and machine learning models. For example, it may be classified as a "connection problem."

[0518] 4. Routing to specialized departments

[0519] The server routes the inquiry to the appropriate department based on the classified inquiry content, and uses an internal routing algorithm to transfer the inquiry to the most appropriate department or person.

[0520] 5. Generate Auto-Response Messages

[0521] The server generates an automatic response message from a FAQ database or template based on the classified inquiry content, such as "Please restart your modem."

[0522] 6. Speech synthesis and response to the user

[0523] The generated auto-answer message is passed to a speech synthesis engine and converted into voice data, which is then sent to the user via the terminal in a voice response. For example, the message tells the user to "restart the modem."

[0524] 7. Summary and Operator Assistance

[0525] The server further analyzes the user's inquiry and provides a concise summary. This summary and related FAQs are displayed on the operator's terminal. The operator can then respond quickly based on the displayed summary information.

[0526] Specific examples

[0527] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" by an NLP engine and routed to the network support department. The server generates an automated response message from the FAQ database saying, "Please restart your modem," and responds to the user by converting it into speech using a speech synthesis engine. If the automated response is not appropriate, the server summarizes the consultation and displays it on the operator terminal. The operator can respond based on the displayed summary.

[0528] Example prompts for generative AI models

[0529] A user uses their smartphone and says, "I can't connect to the Internet."

[0530] The server uses a speech recognition engine to convert the speech into text data.

[0531] The text data is classified by the NLP engine as a "connection issue" and routed to the network support department.

[0532] The server generates an automatic response message from the FAQ database and converts "Please restart your modem" into speech using a speech synthesis engine.

[0533] The terminal plays the audio data and responds to the user.

[0534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0535] Step 1:

[0536] The user begins speaking

[0537] The user calls the call center using a telephone or smartphone and verbally communicates the inquiry through a microphone.

[0538] Input: User speech (e.g. "I can't connect to the Internet")

[0539] Output: Audio data (analog signal)

[0540] Step 2:

[0541] The device converts the voice data into digital data and sends it to the server.

[0542] The device converts the audio picked up by the microphone into a digital format and transmits it to a server using a secure communication protocol (e.g., HTTPS).

[0543] Input: Audio data (analog signal)

[0544] Output: Digital audio data

[0545] Step 3:

[0546] The server receives the voice data and passes it to the voice recognition engine.

[0547] The server passes the received digital voice data to a voice recognition engine (e.g., a general-purpose voice recognition API) and converts the voice into text data. It also performs preprocessing such as noise filtering.

[0548] Input: Digital audio data

[0549] Output: Text data (e.g. "Internet connection not available")

[0550] Step 4:

[0551] The server analyzes the text data and classifies the content of the consultation.

[0552] The server uses an NLP engine to analyze the text data, and based on the analysis, classifies the text data based on pre-defined rules and machine learning models.

[0553] Input: Text data (e.g. "Cannot connect to the Internet")

[0554] Output: Classification result (e.g. "connectivity problem")

[0555] Step 5:

[0556] The server routes the request to a specialist department based on the content of the request.

[0557] The server automatically routes inquiries to the most appropriate department based on the classification results, using a routing algorithm to forward them to the network support department, etc.

[0558] Input: Classification result (e.g. "Connectivity problem")

[0559] Output: Routing instructions (e.g., Network Support Department)

[0560] Step 6:

[0561] The server generates an auto-reply message

[0562] The server generates an appropriate auto-reply message from a FAQ database or template, such as "Please restart your modem."

[0563] Input: Classification result (e.g. "Connectivity problem")

[0564] Output: Auto-answer message (e.g. "Please restart your modem")

[0565] Step 7:

[0566] The server converts the auto-answer message into voice and responds to the user.

[0567] The generated automated response message is passed to a speech synthesis engine (e.g., speech synthesis software) and converted into voice data. This voice data is then sent to the terminal, which then responds to the user.

[0568] Input: Auto-answer message (e.g. "Please restart your modem")

[0569] Output: Audio data

[0570] Step 8:

[0571] The server summarizes the user's consultation and displays it on the operator's terminal.

[0572] The server further analyzes and summarizes the user's inquiry, and displays the summary information, including related FAQs, on the operator's terminal, helping the operator respond quickly.

[0573] Input: User inquiry information, related FAQ

[0574] Output: Summary, operator display information

[0575] By implementing the above steps, the system of the present invention can efficiently process user inquiries and enable automatic responses and rapid routing to specialized departments, thereby significantly improving the operational efficiency of call centers and enhancing user convenience.

[0576] (Application example 1)

[0577] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] Conventional self-driving vehicles lack a system that allows passengers to smoothly respond to their requests and questions. This often results in a lack of convenience and comfort for passengers inside the vehicle. Another problem is that there are insufficient means to quickly obtain emergency or specific information. To solve these issues, a highly functional support system that uses voice input is needed.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0580] In this invention, the server includes means for acquiring a user's voice input, speech recognition means for converting the acquired voice into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automated response message based on the classified consultation content, means for converting the generated automated response message into voice and responding to the user, means for summarizing the user's consultation content and displaying it on an operator's terminal as needed, and means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle. This enables passengers to receive prompt and appropriate responses to their inquiries and requests within the autonomous vehicle.

[0581] "Means for obtaining user voice input" refers to devices or software that recognize voice signals emitted by the user and capture them as digital data.

[0582] "Speech recognition means for converting acquired speech into text data" refers to technology that analyzes speech signals and converts them into corresponding text data, and is generally composed of a microphone and a speech recognition engine.

[0583] The "means for analyzing the converted text data and classifying the consultation content" is a method for analyzing the meaning of the text data using natural language processing technology and classifying the consultation content into predefined categories.

[0584] "Means for routing to specialized departments based on classified consultation content" is a system for automatically distributing inquiries to appropriate specialized departments based on the analyzed consultation content.

[0585] The "means for generating automatic response messages" is a system that automatically creates appropriate response messages in response to user inquiries based on a database or template responses.

[0586] The "means for converting the generated automated response message into speech and responding to the user" refers to a technology for converting the text-based automated response message into speech and providing audible feedback to the user.

[0587] The "means for summarizing the user's consultation content and displaying it on the operator's terminal" is a system that has the function of concisely summarizing the user's consultation content and displaying it on the operator's display device.

[0588] "Means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle" refers to a technology that transmits voice instructions given within the vehicle to an in-vehicle server in real time and provides a comfortable in-vehicle environment based on that.

[0589] This invention is a system for significantly improving passenger convenience in an autonomous vehicle. A specific implementation method of this system will be described below.

[0590] Overall system overview

[0591] The server receives voice input from the user through the infotainment system in the vehicle and generates appropriate responses. The system includes the following main functions:

[0592] Acquiring voice input

[0593] Speech-to-text

[0594] Text Analysis and Classification

[0595] Routing to specialized departments

[0596] Generate auto-response messages

[0597] Speech transcription and response

[0598] Summary of user inquiries and operator support

[0599] Passenger voice commands are sent to an in-vehicle server to provide convenience within the vehicle.

[0600] Hardware and software used

[0601] 1. Hardware:

[0602] Autonomous vehicle infotainment systems

[0603] In-vehicle server

[0604] 2. Software:

[0605] Speech recognition engine: To convert voice input into text data, use services such as Amazon Transcribe or Google Cloud Speech-to-Text.

[0606] Natural Language Processing: To analyze text data and classify its content, use, for example, AWS SageMaker and NLTK.

[0607] Response message generation engine: Refers to FAQs and template answers to automatically create appropriate response messages.

[0608] Text conversion engine: For example, using Amazon Polly or Google Cloud Text-to-Speech to convert the generated text response into speech.

[0609] Example

[0610] 1. Acquiring voice input:

[0611] A passenger issues a voice command such as, "Where is the next service area?" This speech is picked up by the infotainment system.

[0612] 2. Speech to text transcription:

[0613] The acquired voice data is sent to an in-vehicle server and converted into text data such as "Where is the next service area?" by a voice recognition engine (e.g., Amazon Transcribe, Google Cloud Speech-to-Text).

[0614] 3. Text Analysis and Classification:

[0615] The server uses natural language processing technology (e.g., AWS SageMaker, NLTK) to analyze the text data and classify it into categories called "nearby facility information."

[0616] 4. Routing to specialized departments:

[0617] Based on the classification, the call is routed to a specialized department (e.g., vehicle support center) if necessary.

[0618] 5. Generate auto-response messages:

[0619] The server refers to FAQs and template answers in the database and generates an automated response message saying, "The next service area is approximately 15 kilometers away."

[0620] 6. Voice conversion and response:

[0621] The generated automated response message is converted into speech using a text conversion engine (e.g., Amazon Polly, Google Cloud Text-to-Speech) and provided to passengers through the infotainment system.

[0622] 7. Summary of user inquiries and operator support:

[0623] If the consultation is complicated, the server will summarize the content and display it on the operator's terminal along with related information, allowing the operator to respond quickly.

[0624] Examples and prompts

[0625] Example: When a passenger asks, "The car is shaking a little. Is everything okay?", the system automatically responds, "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[0626] Example prompt sentence:

[0627] Where is the next service area?

[0628] The car is shaking a little, is everything okay?

[0629] As a result, the system of the present invention can increase passenger convenience and enable comfortable travel in an autonomous vehicle.

[0630] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0631] Step 1:

[0632] The user provides voice input. The passenger speaks a question or request. For example, "Where is the next service area?" The input is the user's voice, and the infotainment system captures the voice data. The output is the voice data.

[0633] Step 2:

[0634] The infotainment system sends the voice data to the in-vehicle server, which receives it in digital format. The input is the voice data, and the output is the voice data in digital format.

[0635] Step 3:

[0636] The in-vehicle server passes the voice data to a voice recognition engine, which converts the voice into text data. The software used is Amazon Transcribe or Google Cloud Speech-to-Text. The input is voice data, which is processed by voice recognition, and the output is text data. Specifically, the text data obtained is, "Where is the next service area?"

[0637] Step 4:

[0638] The server analyzes the text data and classifies the consultation content based on pre-defined rules and machine learning models. The software used is AWS SageMaker and NLTK. The input is text data, an automatic classification algorithm is applied, and the output is a category such as "information on nearby facilities."

[0639] Step 5:

[0640] The server routes inquiries to specialized departments as needed based on the classified content. The input is the classification result, and the output is routing information. For example, if the inquiry is for "information on nearby facilities," the inquiry will be routed to a specific database or department.

[0641] Step 6:

[0642] The server generates an automatic response message based on the classified inquiry content, referencing FAQs and template answers in the database. The input is the category of the inquiry content and information in the database, and the output is the message "The next service area is approximately 15 kilometers away."

[0643] Step 7:

[0644] The generated automated response message is converted into speech and responded to the user via the infotainment system. The software used is Amazon Polly and Google Cloud Text-to-Speech. The input is a text message, which is converted into speech, and the output is a speech response.

[0645] Step 8:

[0646] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. The input is text data and analysis results, and the output is summary information and display content. As a specific example, in an emergency, a consultation such as "The car is shaking a little. Are you OK?" is summarized and displayed to the operator as "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[0647] Through the above steps, smooth responses and services based on passenger voice input are realized.

[0648] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0649] This invention relates to a system that improves the operational efficiency of call centers by using voice recognition functions, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more appropriate responses. This system uses voice recognition technology and emotion recognition technology to understand the content of the user's inquiry and their emotional state, and provides appropriate automatic responses and support to operators.

[0650] What the program does

[0651] 1. Capture user voice input

[0652] (User)

[0653] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0654] 2. Acquiring and transmitting audio data

[0655] (Terminal)

[0656] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0657] 3. Speech Recognition and Text Conversion

[0658] (server)

[0659] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text data into "Cannot connect to the Internet."

[0660] 4. Emotion Recognition by Emotion Engine

[0661] (server)

[0662] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[0663] 5. Text Data Analysis and Classification

[0664] (server)

[0665] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[0666] 6. Routing to specialized departments

[0667] (server)

[0668] Based on the classified inquiry, the user's inquiry is routed to the appropriate specialist department. In this case, it is routed to the network support department as a "connection problem."

[0669] 7. Generate Auto-Responses

[0670] (server)

[0671] The server generates an automatic response message based on the classified consultation content and also refers to the emotion recognition results. For example, if the user is angry, the tone will be adjusted to say, "We apologize for the inconvenience. Please try restarting your modem."

[0672] 8. Speech conversion and response to the user

[0673] (server)

[0674] The generated automated answering message is converted into voice and responded to the user via the terminal, who receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[0675] 9. Summary and Operator Assistance

[0676] (server)

[0677] The server further analyzes and summarizes the user's inquiry. This summary, along with the emotion recognition results and related FAQs, is displayed on the operator's terminal. In some cases, the operator will respond directly to the user, but the summary information and emotional state provided enable a quick and appropriate response.

[0678] Specific examples

[0679] A user calls a call center and says, "I can't connect to the Internet." This voice is picked up by the device and sent to the server. The server uses a speech recognition engine to convert the voice into text data saying, "I can't connect to the Internet." At the same time, an emotion engine recognizes the emotion "impatience" from the user's voice data. The converted text data is classified as a "connection problem" and routed to the network support department.

[0680] The server, taking into account the emotion recognition results, generates an automatic response message such as "Please try restarting the modem," converts it into speech, and responds to the user. If the automatic response is not appropriate, the server summarizes the consultation content and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state.

[0681] In this way, the system of the present invention makes full use of voice recognition and emotion recognition technologies to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that correspond to the user's emotions.

[0682] The processing flow will be explained below.

[0683] Step 1:

[0684] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0685] Step 2:

[0686] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0687] Step 3:

[0688] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[0689] Step 4:

[0690] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[0691] Step 5:

[0692] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[0693] Step 6:

[0694] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[0695] Step 7:

[0696] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[0697] Step 8:

[0698] The server generates a prepared automated response message while referring to the emotion recognition results. If the user is in a state of impatience, it creates a tone-adjusted message such as "We apologize for the inconvenience. Please try restarting your modem."

[0699] Step 9:

[0700] The server converts the generated automated response message into voice and responds to the user via the terminal. Specifically, the voice message "We apologize for the inconvenience. Please try restarting your modem" is transmitted to the user.

[0701] Step 10:

[0702] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[0703] Step 11:

[0704] The server displays the user's emotional state (e.g., impatience, anger, sadness, etc.) obtained by the emotion engine on the operator's terminal, allowing the operator to understand the user's emotional state and respond accordingly.

[0705] Step 12:

[0706] Based on the summary displayed on the terminal, relevant FAQs, and the user's emotional state, the operator can provide the user with a quick and specific response, such as "Check your modem settings."

[0707] In this way, this system makes full use of voice recognition and emotion recognition technology to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that reflect the user's emotions.

[0708] Example 2

[0709] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0710] Modern call center operations require accurate understanding of user inquiries and prompt and appropriate responses. However, current systems lack the ability to recognize the user's emotional state, which can lead to lower user satisfaction. In particular, when a user is anxious or irritated, it is difficult to respond appropriately, increasing the burden on operators.

[0711] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for acquiring a user's voice input; speech recognition means for converting the acquired voice into text data; emotion recognition means for recognizing the user's emotional state from the voice data; means for adjusting the tone of the automatic response message based on the emotion recognition result; means for analyzing the converted text data and classifying the consultation content; means for routing to a specialized department based on the classified consultation content; means for generating an automatic response message based on the classified consultation content; means for converting the generated automatic response message into voice and responding to the user; and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to accurately grasp not only the content of the user's inquiry but also their emotional state, enabling a prompt and appropriate response.

[0712] A "user" is a person who contacts the call center to make an inquiry or ask for advice.

[0713] "Audio input" refers to audio signals emitted by a user.

[0714] A "terminal" is a device used by a user, and is a device that acquires voice data and transmits it to a server.

[0715] "Speech recognition" refers to the process of analyzing voice data and converting it into text data.

[0716] "Emotion recognition" is a technology that analyzes characteristics such as a user's voice tone, speed, and intonation to determine their emotional state.

[0717] An "automatic response message" is an automatic response text generated by the server to a user's inquiry.

[0718] "Text-to-speech" refers to the process of converting text data into audio data.

[0719] "Routing" is a method of transferring inquiries to appropriate specialist departments based on the classified content of the inquiries.

[0720] "Categorizing consultation content" is the process of analyzing the content of a user's inquiry and classifying it into pre-defined categories.

[0721] "Operator support" is a function that provides the operator with the user's consultation content and emotional state, and supports a quick and appropriate response.

[0722] This invention relates to a system that combines speech recognition technology and emotion recognition technology to improve the operational efficiency of call centers. When a user makes an inquiry to a call center, the system understands the inquiry content and emotional state of the user, and provides appropriate automated responses and support to operators, thereby achieving a fast and accurate response.

[0723] This system is composed of the following elements: users, terminals, and servers. Specifically, it uses the following hardware and software:

[0724] 1. Acquiring voice input

[0725] (User)

[0726] Users contact the call center using devices such as telephones or smartphones. For example, if a user cannot connect to the Internet at home, they will say, "I can't connect to the Internet."

[0727] 2. Acquiring and sending audio data

[0728] (Terminal)

[0729] The device picks up the user's voice using a built-in microphone or an externally connected headset, and the voice data is recorded as digital data in real time and immediately sent to the server.

[0730] 3. Speech Recognition and Text Conversion

[0731] (server)

[0732] The server passes the received voice data to a speech recognition engine (for example, Google Cloud Speech-to-Text API). This engine converts the voice data into text data. Specifically, it converts it into text data that says, "Internet connection not available."

[0733] 4. Emotion recognition

[0734] (server)

[0735] The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the user's voice characteristics, such as tone, speed, and intonation, from the voice data, thereby determining the user's emotional state (e.g., anger, sadness, impatience).

[0736] 5. Text Data Analysis and Classification

[0737] (server)

[0738] The server analyzes the text data using a natural language processing engine (e.g., OpenAI's GPT-3), which extracts intent and keywords from the text and classifies them into predefined categories (e.g., "connection problem").

[0739] 6. Routing to specialized departments

[0740] (server)

[0741] The server routes the user's inquiry to the appropriate department based on the classification of the problem, for example, forwarding it to the network support department as a "connection problem."

[0742] 7. Generate Auto-Responses

[0743] (server)

[0744] The server generates an automatic response message based on the classified consultation content and the emotion recognition results. For example, if the user is angry, the server generates a message saying, "We apologize for the inconvenience. Please try restarting your modem."

[0745] 8. Speech conversion and response to the user

[0746] (server)

[0747] The generated automated response message is converted into speech using a speech synthesis engine (e.g., Amazon Polly) and responded to the user via the terminal. The user receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[0748] 9. Summary and Operator Assistance

[0749] (server)

[0750] The server further analyzes the user's inquiry and creates a summary. This summary, along with the emotion recognition results and related FAQs, are displayed on the operator's terminal. This allows the operator to respond quickly based on the displayed summary and the user's emotional state.

[0751] Specific examples

[0752] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert the speech into text data saying, "I can't connect to the Internet." At the same time, an emotion recognition engine recognizes the emotion "impatience" from the user's speech data. The converted text data is classified as a "connection problem" and routed to the network support department. The server, taking into account the emotion recognition results, generates an automated response message saying, "Please try restarting your modem," converts this into speech, and responds to the user. If the automated response is inappropriate, the server summarizes the consultation and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state. This system increases user convenience and improves call center efficiency.

[0753] Prompt Sentence Examples

[0754] Create a program that recognizes user questions and emotions in a call center system. Explain with a concrete example how the system would respond if a user says, "I can't connect to the Internet."

[0755] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0756] Step 1:

[0757] The user calls the call center and speaks their inquiry, for example, "I can't connect to the Internet." This voice input is sent from the user to the terminal.

[0758] Step 2:

[0759] The device captures the user's voice using a built-in microphone or an externally connected headset. The captured voice data is recorded as digital data in real time. This digital voice data is then sent from the device to the server.

[0760] Step 3:

[0761] The server passes the received voice data to a speech recognition engine. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data. In this process, the voice waveform is analyzed and converted into a string of characters based on a language model. The output text data is, for example, "Internet connection not available."

[0762] Step 4:

[0763] The server passes the received voice data to an emotion recognition engine. Specifically, it uses IBM Watson Tone Analyzer to analyze the voice tone, speed, and intonation characteristics. This analysis determines the user's emotional state. The output data includes emotional states such as "impatience."

[0764] Step 5:

[0765] The server passes the text data to a natural language processing engine for analysis. Specifically, it uses OpenAI's GPT-3 to extract intent and keywords from the text. Based on the input text, it classifies it into categories such as "connection problems." The output of this process is called a "connection problem."

[0766] Step 6:

[0767] The server routes the call to the appropriate department based on the classified text data, for example, "Connection Problems" and forwards it to the network support department. A custom rules-based routing engine is used to determine the route to the department.

[0768] Step 7:

[0769] The server references the classified text data and emotion recognition results to generate an automated response message. This uses FAQs and template answers stored in the database. The inputs are "Connection problem" and "Impatience," and the output is a message that reads, "We apologize for the inconvenience. Please try restarting your modem."

[0770] Step 8:

[0771] The server passes the generated automated response message to a speech synthesis engine and converts it into voice data. Specifically, Amazon Polly is used to convert the text message into voice. The generated voice data is sent to the user via the terminal. The user receives a voice response saying, "We apologize for the inconvenience. Please try restarting your modem."

[0772] Step 9:

[0773] The server further analyzes and summarizes the user's consultation content and the emotion recognition results. The created summary information, emotional state, and related FAQs are displayed on the operator's terminal. This allows the operator to respond quickly based on the summary content and emotional state. Specifically, the operator's screen displays the message, "The user is anxious about an Internet connection problem. We advise you to try restarting the modem."

[0774] (Application example 2)

[0775] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] Conventional security systems have had issues with users reporting abnormalities late and being unable to respond appropriately in emergencies. Furthermore, conventional voice recognition systems are unable to take into account the user's emotional state, making it difficult to respond quickly to highly urgent situations. Furthermore, because reporting to appropriate external agencies is not automated, prompt assistance may not be available.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0778] In this invention, the server includes an emotion recognition means for identifying an emotional state from the user's voice data, a means for determining an optimal response based on the identified emotion, and a means for detecting an emergency and notifying an appropriate external agency, thereby enabling the user's emergency to be quickly detected and an appropriate response to be taken.

[0779] "Voice input" refers to the user speaking into a device.

[0780] "Speech recognition" refers to the technology of analyzing captured speech as digital data and converting it into text data.

[0781] "Text data" refers to text information converted using voice recognition technology.

[0782] "Emotion recognition" refers to the technology of identifying a user's emotional state from voice data.

[0783] "Emotional state" is information that indicates the mental state of the user, and specifically refers to anger, sadness, impatience, and the like.

[0784] "Rational response" refers to the optimal response generated based on the classified consultation content and the user's emotional state.

[0785] A "specialized department" refers to a department that handles specific consultation matters.

[0786] "Routing" refers to the series of processes that send classified consultation content to the appropriate department or external agency.

[0787] An "automatic response message" refers to an automatically generated reply message to a user.

[0788] "Text-to-speech" refers to the process of converting text messages back into audio data.

[0789] An "emergency" is any situation that requires immediate action.

[0790] "External agencies" refer to support organizations outside the system, such as police and security companies.

[0791] This system combines voice recognition technology and emotion recognition technology to enhance user convenience and respond quickly and appropriately to emergency situations.

[0792] First, when a user speaks into a smart device (smartphone, smart glasses, head-mounted display, etc.), the device picks up the user's voice and sends it as digital data to a cloud server. The voice data is collected in real time and sent to the server immediately.

[0793] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the captured speech into text data. Once the text data is generated, an emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the speech data and identifies the user's emotional state. Emotional states include anger, sadness, impatience, and fear.

[0794] The generated text data and emotion recognition results are then used to classify the user's consultation and, if necessary, automatically notify the appropriate external agency (such as the police or security company). For example, if a user says, "This is an emergency, someone please help," the emotion recognition engine will determine this as "fear," and based on that, a call will be made immediately to the security company or police.

[0795] Furthermore, the server generates an automatic response message for the user to ensure safety. This generated message is converted into voice and returned to the user through the device. For example, an automatic response message such as "The police have been notified. Please evacuate to a safe place immediately" may be generated and notify the user.

[0796] This allows users to receive prompt and appropriate responses, allowing them to deal with emergencies with peace of mind. In addition, if operator assistance is required, the server will summarize the user's consultation and display it on the operator's terminal along with their emotional state, allowing the operator to respond promptly according to the situation.

[0797] Examples of concrete examples and prompts

[0798] For example, if a user suddenly discovers an intruder in their home and speaks into the smart glasses, "This is an emergency, someone please help me," the voice will be sent to the server in real time, and based on the converted text and emotion recognition results, the police will be immediately notified, and a message will be generated stating, "The police have been notified. Please evacuate to a safe place immediately."

[0799] Prompt Sentence Examples

[0800] "When a user speaks into a smart device, saying, 'This is an emergency, someone please help,' the voice data is sent to a cloud server, where a speech recognition engine converts it into text data. The emotion recognition engine then recognizes the user's emotional state as 'fear,' and based on that, calls the police and generates an automatic response to the user saying, 'The police have been called. Please evacuate to a safe place immediately.'"

[0801] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0802] Step 1:

[0803] A user speaks to a smart device, for example, "This is an emergency, please help me." Voice data is acquired as input. This voice data is captured through the device's microphone.

[0804] Step 2:

[0805] The device captures audio data, records it as digital data in real time, and sends it to a cloud server. The captured audio data is the input, and the digital data is sent to the cloud server as the output. Specifically, the device digitizes the audio signal and transmits it over the network.

[0806] Step 3:

[0807] The server passes the received voice data to a voice recognition engine. The input is digital voice data, and the voice recognition engine (e.g., Google Cloud Speech-to-Text API) converts the voice into text data. The output is text data. Specifically, the voice waveform is analyzed and words are identified.

[0808] Step 4:

[0809] The server passes the text data to an emotion recognition engine. The input is the text data and speech features, and the emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's speech tone, speed, and intonation to identify their emotional state. The output is the user's emotional state. Specifically, emotion feature vectors are extracted and classified.

[0810] Step 5:

[0811] The server analyzes the user's consultation content based on the generated text data and emotion recognition results, and classifies it as necessary. The input is text data and emotional state, and the machine learning model analyzes the consultation content and classifies it into the appropriate category. The output is the classification result. Specifically, a text classifier, a natural language processing algorithm, is used.

[0812] Step 6:

[0813] Based on the classified consultation content, the server detects an emergency and notifies the necessary external agencies (e.g., police or security companies). The inputs are the classification result and emotional state, and the urgency is evaluated to generate routing information to external agencies. The output is a report message that is generated and sent. Specifically, a report is sent to an external system via an API.

[0814] Step 7:

[0815] The server generates an automatic response message for the user to ensure safety. The inputs are the classification result and the emotional state, and a template response message is selected from a database and adjusted. The output is the generated automatic response message. Specifically, a condition-based message generation algorithm is executed.

[0816] Step 8:

[0817] The generated automated response message is converted into voice and responded to the user via the terminal. The input is the generated text message, and the voice conversion engine converts the text into voice data. The output is a voice message that is conveyed to the user. Specifically, voice synthesis technology is used.

[0818] Step 9:

[0819] Depending on the situation, the server summarizes the user's consultation content and the emotion recognition results, and displays them on the operator's terminal. The input is text data and emotional state, and a summary is generated using a text summarization algorithm. The output is the summary information displayed on the operator's terminal. Specifically, the summary is generated using natural language processing technology.

[0820] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0821] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0822] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0823] [Third embodiment]

[0824] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0825] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0826] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0827] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0828] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0829] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0830] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0831] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0832] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0833] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0834] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0835] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0836] This invention relates to a system that uses voice recognition to improve the operational efficiency of call centers. This system uses voice recognition technology to appropriately classify customer inquiries, enabling automatic responses and efficient support for operators.

[0837] What the program does

[0838] 1. Capture user voice input

[0839] (User)

[0840] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0841] 2. Acquiring and transmitting audio data

[0842] (Terminal)

[0843] The terminal captures the user's voice and transmits the voice data to the server.

[0844] 3. Speech Recognition and Text Conversion

[0845] (server)

[0846] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text into "Internet connection not possible."

[0847] 4. Text Data Analysis and Classification

[0848] (server)

[0849] The server analyzes the text data and classifies the issue based on pre-defined rules and machine learning models. In this case, it is classified as a "connection problem."

[0850] 5. Routing to specialized departments

[0851] (server)

[0852] Route inquiries to network support departments based on the classified inquiry content.

[0853] 6. Generate Auto-Responses

[0854] (server)

[0855] The server generates an auto-reply message based on the classified inquiry, for example, "Please restart your modem."

[0856] 7. Speech conversion and response to the user

[0857] (server)

[0858] The generated automatic response message is converted into voice and responded to the user via the terminal.

[0859] 8. Summary and Operator Assistance

[0860] (server)

[0861] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. In some cases, the operator will respond to the user directly, but the summary information allows for a quicker response.

[0862] Specific examples

[0863] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the device and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" and routed to the network support department.

[0864] The server generates an automatic response message for a frequently asked question, such as "Please try restarting the modem," and converts it into voice to respond to the user. If the automatic response is not appropriate, the server summarizes the inquiry and displays it on the operator's terminal. The operator can respond quickly based on the displayed summary and related FAQs.

[0865] In this way, the system of the present invention makes full use of voice recognition and automatic response technology to enhance user convenience and significantly improve the operational efficiency of call centers.

[0866] The processing flow will be explained below.

[0867] Step 1:

[0868] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[0869] Step 2:

[0870] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[0871] Step 3:

[0872] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[0873] Step 4:

[0874] The server analyzes the text data obtained from the speech recognition engine and classifies the consultation content using pre-set rules and machine learning models. In this example, it is classified as a "connection problem."

[0875] Step 5:

[0876] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[0877] Step 6:

[0878] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[0879] Step 7:

[0880] The server generates the prepared auto-answer message and converts it into voice using speech synthesis technology. Specifically, the voice message generated is "Please try restarting your modem."

[0881] Step 8:

[0882] The server sends a voice-converted automated response message to the terminal, which then relays it to the user, who receives the advice "Try restarting your modem."

[0883] Step 9:

[0884] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[0885] Step 10:

[0886] Based on the summary displayed on the terminal and related FAQs, the operator can provide the user with a quick and specific response, such as giving appropriate advice such as "Check the modem settings."

[0887] Through these steps, the system enables quick and accurate responses to user inquiries, while at the same time significantly improving the operational efficiency of call centers.

[0888] Example 1

[0889] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0890] In conventional call centers, operators manually handle inquiries from users, which takes time and manpower, resulting in low operational efficiency. It was particularly difficult to respond quickly when multiple inquiries occurred simultaneously. Furthermore, there was a lack of technology to efficiently convert voice data into text data and properly classify the content, making it difficult to respond accurately or assign calls to the appropriate department.

[0891] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0892] In this invention, the server includes means for acquiring user voice input, means for transmitting the acquired voice to the server as digital data, speech recognition means for converting the acquired voice data into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automatic response message based on the classified consultation content, means for converting the generated automatic response message into voice and responding to the user, and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to process user inquiries quickly and accurately, and to provide an appropriate automatic response or route to a specialized department, thereby significantly improving business efficiency.

[0893] "Means for acquiring user voice input" refers to devices or software for receiving voice from the user, such as a microphone installed on a telephone or smartphone.

[0894] The "means for transmitting the captured audio as digital data to the server" refers to a communication means for converting analog audio into digital form and transmitting it to the server over a network. Specifically, this includes an audio signal converter and a communication protocol.

[0895] "Speech recognition means for converting acquired voice data into text data" refers to technology for analyzing voice and converting it into text information, and a voice recognition engine is an example of this.

[0896] "Means for analyzing the converted text data and classifying the consultation content" refers to algorithms or software that interpret the text data using natural language processing technology and classify it based on pre-set categories.

[0897] "Means of routing to specialized departments based on classified consultation content" refers to a system that automatically assigns inquiries to the appropriate department or person in charge based on the analysis results.

[0898] "A means for generating automatic response messages based on classified inquiry content" refers to a function that automatically creates responses to user inquiries by referring to an FAQ database or template.

[0899] "Means for converting the generated automated response message into speech and responding to the user" refers to technology for converting text-based messages into speech, such as a speech synthesis engine.

[0900] "Means of summarizing the user's inquiry content and displaying it on the operator's terminal as the case may be" refers to a function for concisely summarizing the user's inquiry data and displaying it in a way that the operator can quickly understand.

[0901] The present invention relates to a system that uses a voice recognition function to improve the operational efficiency of a call center. The system acquires voice input from a user, analyzes the voice, and responds appropriately.

[0902] Hardware and software used

[0903] Hardware

[0904] Devices: Phones, smartphones, PCs with headsets

[0905] Server: A server machine with a powerful processor and sufficient memory

[0906] software

[0907] Speech recognition engine: Technology that converts voice data into text data. For example, a general-purpose speech recognition API or in-house developed speech recognition software.

[0908] Natural language processing engine (NLP): A technology that analyzes text data and classifies the content of inquiries.

[0909] Speech synthesis engine: Technology that converts auto-response messages into speech, such as in-house developed speech synthesis software.

[0910] Database: A database system for storing FAQs and template answers.

[0911] Data processing and calculation

[0912] 1. Audio capture and transmission

[0913] A user calls a call center using a telephone or smartphone. The device picks up the user's voice through a microphone and sends the voice data to a server. The sent voice data is then passed to a speech recognition engine within the server.

[0914] 2. Speech Recognition and Text Conversion

[0915] The server uses a speech recognition engine to convert the voice data into text data. For example, the speech recognition engine converts the voice "I can't connect to the Internet" into text. This conversion process involves noise filtering and language modeling.

[0916] 3. Analysis and Classification of Text Data

[0917] The converted text data is passed to an NLP engine on the server, where its content is analyzed. The NLP engine then classifies the text data based on pre-defined rules and machine learning models. For example, it may be classified as a "connection problem."

[0918] 4. Routing to specialized departments

[0919] The server routes the inquiry to the appropriate department based on the classified inquiry content, and uses an internal routing algorithm to transfer the inquiry to the most appropriate department or person.

[0920] 5. Generate Auto-Response Messages

[0921] The server generates an automatic response message from a FAQ database or template based on the classified inquiry content, such as "Please restart your modem."

[0922] 6. Speech synthesis and response to the user

[0923] The generated auto-answer message is passed to a speech synthesis engine and converted into voice data, which is then sent to the user via the terminal in a voice response. For example, the message tells the user to "restart the modem."

[0924] 7. Summary and Operator Assistance

[0925] The server further analyzes the user's inquiry and provides a concise summary. This summary and related FAQs are displayed on the operator's terminal. The operator can then respond quickly based on the displayed summary information.

[0926] Specific examples

[0927] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" by an NLP engine and routed to the network support department. The server generates an automated response message from the FAQ database saying, "Please restart your modem," and responds to the user by converting it into speech using a speech synthesis engine. If the automated response is not appropriate, the server summarizes the consultation and displays it on the operator terminal. The operator can respond based on the displayed summary.

[0928] Example prompts for generative AI models

[0929] A user uses their smartphone and says, "I can't connect to the Internet."

[0930] The server uses a speech recognition engine to convert the speech into text data.

[0931] The text data is classified by the NLP engine as a "connection issue" and routed to the network support department.

[0932] The server generates an automatic response message from the FAQ database and converts "Please restart your modem" into speech using a speech synthesis engine.

[0933] The terminal plays the audio data and responds to the user.

[0934] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0935] Step 1:

[0936] The user begins speaking

[0937] The user calls the call center using a telephone or smartphone and verbally communicates the inquiry through a microphone.

[0938] Input: User speech (e.g. "I can't connect to the Internet")

[0939] Output: Audio data (analog signal)

[0940] Step 2:

[0941] The device converts the voice data into digital data and sends it to the server.

[0942] The device converts the audio picked up by the microphone into a digital format and transmits it to a server using a secure communication protocol (e.g., HTTPS).

[0943] Input: Audio data (analog signal)

[0944] Output: Digital audio data

[0945] Step 3:

[0946] The server receives the voice data and passes it to the voice recognition engine.

[0947] The server passes the received digital voice data to a voice recognition engine (e.g., a general-purpose voice recognition API) and converts the voice into text data. It also performs preprocessing such as noise filtering.

[0948] Input: Digital audio data

[0949] Output: Text data (e.g. "Internet connection not available")

[0950] Step 4:

[0951] The server analyzes the text data and classifies the content of the consultation.

[0952] The server uses an NLP engine to analyze the text data, and based on the analysis, classifies the text data based on pre-defined rules and machine learning models.

[0953] Input: Text data (e.g. "Cannot connect to the Internet")

[0954] Output: Classification result (e.g. "connectivity problem")

[0955] Step 5:

[0956] The server routes the request to a specialist department based on the content of the request.

[0957] The server automatically routes inquiries to the most appropriate department based on the classification results, using a routing algorithm to forward them to the network support department, etc.

[0958] Input: Classification result (e.g. "Connectivity problem")

[0959] Output: Routing instructions (e.g., Network Support Department)

[0960] Step 6:

[0961] The server generates an auto-reply message

[0962] The server generates an appropriate auto-reply message from a FAQ database or template, such as "Please restart your modem."

[0963] Input: Classification result (e.g. "Connectivity problem")

[0964] Output: Auto-answer message (e.g. "Please restart your modem")

[0965] Step 7:

[0966] The server converts the auto-answer message into voice and responds to the user.

[0967] The generated automated response message is passed to a speech synthesis engine (e.g., speech synthesis software) and converted into voice data. This voice data is then sent to the terminal, which then responds to the user.

[0968] Input: Auto-answer message (e.g. "Please restart your modem")

[0969] Output: Audio data

[0970] Step 8:

[0971] The server summarizes the user's consultation and displays it on the operator's terminal.

[0972] The server further analyzes and summarizes the user's inquiry, and displays the summary information, including related FAQs, on the operator's terminal, helping the operator respond quickly.

[0973] Input: User inquiry information, related FAQ

[0974] Output: Summary, operator display information

[0975] By implementing the above steps, the system of the present invention can efficiently process user inquiries and enable automatic responses and rapid routing to specialized departments, thereby significantly improving the operational efficiency of call centers and enhancing user convenience.

[0976] (Application example 1)

[0977] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0978] Conventional self-driving vehicles lack a system that allows passengers to smoothly respond to their requests and questions. This often results in a lack of convenience and comfort for passengers inside the vehicle. Another problem is that there are insufficient means to quickly obtain emergency or specific information. To solve these issues, a highly functional support system that uses voice input is needed.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0980] In this invention, the server includes means for acquiring a user's voice input, speech recognition means for converting the acquired voice into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automated response message based on the classified consultation content, means for converting the generated automated response message into voice and responding to the user, means for summarizing the user's consultation content and displaying it on an operator's terminal as needed, and means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle. This enables passengers to receive prompt and appropriate responses to their inquiries and requests within the autonomous vehicle.

[0981] "Means for obtaining user voice input" refers to devices or software that recognize voice signals emitted by the user and capture them as digital data.

[0982] "Speech recognition means for converting acquired speech into text data" refers to technology that analyzes speech signals and converts them into corresponding text data, and is generally composed of a microphone and a speech recognition engine.

[0983] The "means for analyzing the converted text data and classifying the consultation content" is a method for analyzing the meaning of the text data using natural language processing technology and classifying the consultation content into predefined categories.

[0984] "Means for routing to specialized departments based on classified consultation content" is a system for automatically distributing inquiries to appropriate specialized departments based on the analyzed consultation content.

[0985] The "means for generating automatic response messages" is a system that automatically creates appropriate response messages in response to user inquiries based on a database or template responses.

[0986] The "means for converting the generated automated response message into speech and responding to the user" refers to a technology for converting the text-based automated response message into speech and providing audible feedback to the user.

[0987] The "means for summarizing the user's consultation content and displaying it on the operator's terminal" is a system that has the function of concisely summarizing the user's consultation content and displaying it on the operator's display device.

[0988] "Means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle" refers to a technology that transmits voice instructions given within the vehicle to an in-vehicle server in real time and provides a comfortable in-vehicle environment based on that.

[0989] This invention is a system for significantly improving passenger convenience in an autonomous vehicle. A specific implementation method of this system will be described below.

[0990] Overall system overview

[0991] The server receives voice input from the user through the infotainment system in the vehicle and generates appropriate responses. The system includes the following main functions:

[0992] Acquiring voice input

[0993] Speech-to-text

[0994] Text Analysis and Classification

[0995] Routing to specialized departments

[0996] Generate auto-response messages

[0997] Speech transcription and response

[0998] Summary of user inquiries and operator support

[0999] Passenger voice commands are sent to an in-vehicle server to provide convenience within the vehicle.

[1000] Hardware and software used

[1001] 1. Hardware:

[1002] Autonomous vehicle infotainment systems

[1003] In-vehicle server

[1004] 2. Software:

[1005] Speech recognition engine: To convert voice input into text data, use services such as Amazon Transcribe or Google Cloud Speech-to-Text.

[1006] Natural Language Processing: To analyze text data and classify its content, use, for example, AWS SageMaker and NLTK.

[1007] Response message generation engine: Refers to FAQs and template answers to automatically create appropriate response messages.

[1008] Text conversion engine: For example, using Amazon Polly or Google Cloud Text-to-Speech to convert the generated text response into speech.

[1009] Example

[1010] 1. Acquiring voice input:

[1011] A passenger issues a voice command such as, "Where is the next service area?" This speech is picked up by the infotainment system.

[1012] 2. Speech to text transcription:

[1013] The acquired voice data is sent to an in-vehicle server and converted into text data such as "Where is the next service area?" by a voice recognition engine (e.g., Amazon Transcribe, Google Cloud Speech-to-Text).

[1014] 3. Text Analysis and Classification:

[1015] The server uses natural language processing technology (e.g., AWS SageMaker, NLTK) to analyze the text data and classify it into categories called "nearby facility information."

[1016] 4. Routing to specialized departments:

[1017] Based on the classification, the call is routed to a specialized department (e.g., vehicle support center) if necessary.

[1018] 5. Generate auto-response messages:

[1019] The server refers to FAQs and template answers in the database and generates an automated response message saying, "The next service area is approximately 15 kilometers away."

[1020] 6. Voice conversion and response:

[1021] The generated automated response message is converted into speech using a text conversion engine (e.g., Amazon Polly, Google Cloud Text-to-Speech) and provided to passengers through the infotainment system.

[1022] 7. Summary of user inquiries and operator support:

[1023] If the consultation is complicated, the server will summarize the content and display it on the operator's terminal along with related information, allowing the operator to respond quickly.

[1024] Examples and prompts

[1025] Example: When a passenger asks, "The car is shaking a little. Is everything okay?", the system automatically responds, "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[1026] Example prompt sentence:

[1027] Where is the next service area?

[1028] The car is shaking a little, is everything okay?

[1029] As a result, the system of the present invention can increase passenger convenience and enable comfortable travel in an autonomous vehicle.

[1030] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1031] Step 1:

[1032] The user provides voice input. The passenger speaks a question or request. For example, "Where is the next service area?" The input is the user's voice, and the infotainment system captures the voice data. The output is the voice data.

[1033] Step 2:

[1034] The infotainment system sends the voice data to the in-vehicle server, which receives it in digital format. The input is the voice data, and the output is the voice data in digital format.

[1035] Step 3:

[1036] The in-vehicle server passes the voice data to a voice recognition engine, which converts the voice into text data. The software used is Amazon Transcribe or Google Cloud Speech-to-Text. The input is voice data, which is processed by voice recognition, and the output is text data. Specifically, the text data obtained is, "Where is the next service area?"

[1037] Step 4:

[1038] The server analyzes the text data and classifies the consultation content based on pre-defined rules and machine learning models. The software used is AWS SageMaker and NLTK. The input is text data, an automatic classification algorithm is applied, and the output is a category such as "information on nearby facilities."

[1039] Step 5:

[1040] The server routes inquiries to specialized departments as needed based on the classified content. The input is the classification result, and the output is routing information. For example, if the inquiry is for "information on nearby facilities," the inquiry will be routed to a specific database or department.

[1041] Step 6:

[1042] The server generates an automatic response message based on the classified inquiry content, referencing FAQs and template answers in the database. The input is the category of the inquiry content and information in the database, and the output is the message "The next service area is approximately 15 kilometers away."

[1043] Step 7:

[1044] The generated automated response message is converted into speech and responded to the user via the infotainment system. The software used is Amazon Polly and Google Cloud Text-to-Speech. The input is a text message, which is converted into speech, and the output is a speech response.

[1045] Step 8:

[1046] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. The input is text data and analysis results, and the output is summary information and display content. As a specific example, in an emergency, a consultation such as "The car is shaking a little. Are you OK?" is summarized and displayed to the operator as "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[1047] Through the above steps, smooth responses and services based on passenger voice input are realized.

[1048] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1049] This invention relates to a system that improves the operational efficiency of call centers by using voice recognition functions, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more appropriate responses. This system uses voice recognition technology and emotion recognition technology to understand the content of the user's inquiry and their emotional state, and provides appropriate automatic responses and support to operators.

[1050] What the program does

[1051] 1. Capture user voice input

[1052] (User)

[1053] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1054] 2. Acquiring and transmitting audio data

[1055] (Terminal)

[1056] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[1057] 3. Speech Recognition and Text Conversion

[1058] (server)

[1059] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text data into "Cannot connect to the Internet."

[1060] 4. Emotion Recognition by Emotion Engine

[1061] (server)

[1062] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[1063] 5. Text Data Analysis and Classification

[1064] (server)

[1065] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[1066] 6. Routing to specialized departments

[1067] (server)

[1068] Based on the classified inquiry, the user's inquiry is routed to the appropriate specialist department. In this case, it is routed to the network support department as a "connection problem."

[1069] 7. Generate Auto-Responses

[1070] (server)

[1071] The server generates an automatic response message based on the classified consultation content and also refers to the emotion recognition results. For example, if the user is angry, the tone will be adjusted to say, "We apologize for the inconvenience. Please try restarting your modem."

[1072] 8. Speech conversion and response to the user

[1073] (server)

[1074] The generated automated answering message is converted into voice and responded to the user via the terminal, who receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[1075] 9. Summary and Operator Assistance

[1076] (server)

[1077] The server further analyzes and summarizes the user's inquiry. This summary, along with the emotion recognition results and related FAQs, is displayed on the operator's terminal. In some cases, the operator will respond directly to the user, but the summary information and emotional state provided enable a quick and appropriate response.

[1078] Specific examples

[1079] A user calls a call center and says, "I can't connect to the Internet." This voice is picked up by the device and sent to the server. The server uses a speech recognition engine to convert the voice into text data saying, "I can't connect to the Internet." At the same time, an emotion engine recognizes the emotion "impatience" from the user's voice data. The converted text data is classified as a "connection problem" and routed to the network support department.

[1080] The server, taking into account the emotion recognition results, generates an automatic response message such as "Please try restarting the modem," converts it into speech, and responds to the user. If the automatic response is not appropriate, the server summarizes the consultation content and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state.

[1081] In this way, the system of the present invention makes full use of voice recognition and emotion recognition technologies to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that correspond to the user's emotions.

[1082] The processing flow will be explained below.

[1083] Step 1:

[1084] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1085] Step 2:

[1086] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[1087] Step 3:

[1088] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[1089] Step 4:

[1090] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[1091] Step 5:

[1092] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[1093] Step 6:

[1094] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[1095] Step 7:

[1096] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[1097] Step 8:

[1098] The server generates a prepared automated response message while referring to the emotion recognition results. If the user is in a state of impatience, it creates a tone-adjusted message such as "We apologize for the inconvenience. Please try restarting your modem."

[1099] Step 9:

[1100] The server converts the generated automated response message into voice and responds to the user via the terminal. Specifically, the voice message "We apologize for the inconvenience. Please try restarting your modem" is transmitted to the user.

[1101] Step 10:

[1102] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[1103] Step 11:

[1104] The server displays the user's emotional state (e.g., impatience, anger, sadness, etc.) obtained by the emotion engine on the operator's terminal, allowing the operator to understand the user's emotional state and respond accordingly.

[1105] Step 12:

[1106] Based on the summary displayed on the terminal, relevant FAQs, and the user's emotional state, the operator can provide the user with a quick and specific response, such as "Check your modem settings."

[1107] In this way, this system makes full use of voice recognition and emotion recognition technology to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that reflect the user's emotions.

[1108] Example 2

[1109] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1110] Modern call center operations require accurate understanding of user inquiries and prompt and appropriate responses. However, current systems lack the ability to recognize the user's emotional state, which can lead to lower user satisfaction. In particular, when a user is anxious or irritated, it is difficult to respond appropriately, increasing the burden on operators.

[1111] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for acquiring a user's voice input; speech recognition means for converting the acquired voice into text data; emotion recognition means for recognizing the user's emotional state from the voice data; means for adjusting the tone of the automatic response message based on the emotion recognition result; means for analyzing the converted text data and classifying the consultation content; means for routing to a specialized department based on the classified consultation content; means for generating an automatic response message based on the classified consultation content; means for converting the generated automatic response message into voice and responding to the user; and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to accurately grasp not only the content of the user's inquiry but also their emotional state, enabling a prompt and appropriate response.

[1112] A "user" is a person who contacts the call center to make an inquiry or ask for advice.

[1113] "Audio input" refers to audio signals emitted by a user.

[1114] A "terminal" is a device used by a user, and is a device that acquires voice data and transmits it to a server.

[1115] "Speech recognition" refers to the process of analyzing voice data and converting it into text data.

[1116] "Emotion recognition" is a technology that analyzes characteristics such as a user's voice tone, speed, and intonation to determine their emotional state.

[1117] An "automatic response message" is an automatic response text generated by the server to a user's inquiry.

[1118] "Text-to-speech" refers to the process of converting text data into audio data.

[1119] "Routing" is a method of transferring inquiries to appropriate specialist departments based on the classified content of the inquiries.

[1120] "Categorizing consultation content" is the process of analyzing the content of a user's inquiry and classifying it into pre-defined categories.

[1121] "Operator support" is a function that provides the operator with the user's consultation content and emotional state, and supports a quick and appropriate response.

[1122] This invention relates to a system that combines speech recognition technology and emotion recognition technology to improve the operational efficiency of call centers. When a user makes an inquiry to a call center, the system understands the inquiry content and emotional state of the user, and provides appropriate automated responses and support to operators, thereby achieving a fast and accurate response.

[1123] This system is composed of the following elements: users, terminals, and servers. Specifically, it uses the following hardware and software:

[1124] 1. Acquiring voice input

[1125] (User)

[1126] Users contact the call center using devices such as telephones or smartphones. For example, if a user cannot connect to the Internet at home, they will say, "I can't connect to the Internet."

[1127] 2. Acquiring and sending audio data

[1128] (Terminal)

[1129] The device picks up the user's voice using a built-in microphone or an externally connected headset, and the voice data is recorded as digital data in real time and immediately sent to the server.

[1130] 3. Speech Recognition and Text Conversion

[1131] (server)

[1132] The server passes the received voice data to a speech recognition engine (for example, Google Cloud Speech-to-Text API). This engine converts the voice data into text data. Specifically, it converts it into text data that says, "Internet connection not available."

[1133] 4. Emotion recognition

[1134] (server)

[1135] The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the user's voice characteristics, such as tone, speed, and intonation, from the voice data, thereby determining the user's emotional state (e.g., anger, sadness, impatience).

[1136] 5. Text Data Analysis and Classification

[1137] (server)

[1138] The server analyzes the text data using a natural language processing engine (e.g., OpenAI's GPT-3), which extracts intent and keywords from the text and classifies them into predefined categories (e.g., "connection problem").

[1139] 6. Routing to specialized departments

[1140] (server)

[1141] The server routes the user's inquiry to the appropriate department based on the classification of the problem, for example, forwarding it to the network support department as a "connection problem."

[1142] 7. Generate Auto-Responses

[1143] (server)

[1144] The server generates an automatic response message based on the classified consultation content and the emotion recognition results. For example, if the user is angry, the server generates a message saying, "We apologize for the inconvenience. Please try restarting your modem."

[1145] 8. Speech conversion and response to the user

[1146] (server)

[1147] The generated automated response message is converted into speech using a speech synthesis engine (e.g., Amazon Polly) and responded to the user via the terminal. The user receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[1148] 9. Summary and Operator Assistance

[1149] (server)

[1150] The server further analyzes the user's inquiry and creates a summary. This summary, along with the emotion recognition results and related FAQs, are displayed on the operator's terminal. This allows the operator to respond quickly based on the displayed summary and the user's emotional state.

[1151] Specific examples

[1152] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert the speech into text data saying, "I can't connect to the Internet." At the same time, an emotion recognition engine recognizes the emotion "impatience" from the user's speech data. The converted text data is classified as a "connection problem" and routed to the network support department. The server, taking into account the emotion recognition results, generates an automated response message saying, "Please try restarting your modem," converts this into speech, and responds to the user. If the automated response is inappropriate, the server summarizes the consultation and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state. This system increases user convenience and improves call center efficiency.

[1153] Prompt Sentence Examples

[1154] Create a program that recognizes user questions and emotions in a call center system. Explain with a concrete example how the system would respond if a user says, "I can't connect to the Internet."

[1155] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1156] Step 1:

[1157] The user calls the call center and speaks their inquiry, for example, "I can't connect to the Internet." This voice input is sent from the user to the terminal.

[1158] Step 2:

[1159] The device captures the user's voice using a built-in microphone or an externally connected headset. The captured voice data is recorded as digital data in real time. This digital voice data is then sent from the device to the server.

[1160] Step 3:

[1161] The server passes the received voice data to a speech recognition engine. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data. In this process, the voice waveform is analyzed and converted into a string of characters based on a language model. The output text data is, for example, "Internet connection not available."

[1162] Step 4:

[1163] The server passes the received voice data to an emotion recognition engine. Specifically, it uses IBM Watson Tone Analyzer to analyze the voice tone, speed, and intonation characteristics. This analysis determines the user's emotional state. The output data includes emotional states such as "impatience."

[1164] Step 5:

[1165] The server passes the text data to a natural language processing engine for analysis. Specifically, it uses OpenAI's GPT-3 to extract intent and keywords from the text. Based on the input text, it classifies it into categories such as "connection problems." The output of this process is called a "connection problem."

[1166] Step 6:

[1167] The server routes the call to the appropriate department based on the classified text data, for example, "Connection Problems" and forwards it to the network support department. A custom rules-based routing engine is used to determine the route to the department.

[1168] Step 7:

[1169] The server references the classified text data and emotion recognition results to generate an automated response message. This uses FAQs and template answers stored in the database. The inputs are "Connection problem" and "Impatience," and the output is a message that reads, "We apologize for the inconvenience. Please try restarting your modem."

[1170] Step 8:

[1171] The server passes the generated automated response message to a speech synthesis engine and converts it into voice data. Specifically, Amazon Polly is used to convert the text message into voice. The generated voice data is sent to the user via the terminal. The user receives a voice response saying, "We apologize for the inconvenience. Please try restarting your modem."

[1172] Step 9:

[1173] The server further analyzes and summarizes the user's consultation content and the emotion recognition results. The created summary information, emotional state, and related FAQs are displayed on the operator's terminal. This allows the operator to respond quickly based on the summary content and emotional state. Specifically, the operator's screen displays the message, "The user is anxious about an Internet connection problem. We advise you to try restarting the modem."

[1174] (Application example 2)

[1175] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] Conventional security systems have had issues with users reporting abnormalities late and being unable to respond appropriately in emergencies. Furthermore, conventional voice recognition systems are unable to take into account the user's emotional state, making it difficult to respond quickly to highly urgent situations. Furthermore, because reporting to appropriate external agencies is not automated, prompt assistance may not be available.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1178] In this invention, the server includes an emotion recognition means for identifying an emotional state from the user's voice data, a means for determining an optimal response based on the identified emotion, and a means for detecting an emergency and notifying an appropriate external agency, thereby enabling the user's emergency to be quickly detected and an appropriate response to be taken.

[1179] "Voice input" refers to the user speaking into a device.

[1180] "Speech recognition" refers to the technology of analyzing captured speech as digital data and converting it into text data.

[1181] "Text data" refers to text information converted using voice recognition technology.

[1182] "Emotion recognition" refers to the technology of identifying a user's emotional state from voice data.

[1183] "Emotional state" is information that indicates the mental state of the user, and specifically refers to anger, sadness, impatience, and the like.

[1184] "Rational response" refers to the optimal response generated based on the classified consultation content and the user's emotional state.

[1185] A "specialized department" refers to a department that handles specific consultation matters.

[1186] "Routing" refers to the series of processes that send classified consultation content to the appropriate department or external agency.

[1187] An "automatic response message" refers to an automatically generated reply message to a user.

[1188] "Text-to-speech" refers to the process of converting text messages back into audio data.

[1189] An "emergency" is any situation that requires immediate action.

[1190] "External agencies" refer to support organizations outside the system, such as police and security companies.

[1191] This system combines voice recognition technology and emotion recognition technology to enhance user convenience and respond quickly and appropriately to emergency situations.

[1192] First, when a user speaks into a smart device (smartphone, smart glasses, head-mounted display, etc.), the device picks up the user's voice and sends it as digital data to a cloud server. The voice data is collected in real time and sent to the server immediately.

[1193] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the captured speech into text data. Once the text data is generated, an emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the speech data and identifies the user's emotional state. Emotional states include anger, sadness, impatience, and fear.

[1194] The generated text data and emotion recognition results are then used to classify the user's consultation and, if necessary, automatically notify the appropriate external agency (such as the police or security company). For example, if a user says, "This is an emergency, someone please help," the emotion recognition engine will determine this as "fear," and based on that, a call will be made immediately to the security company or police.

[1195] Furthermore, the server generates an automatic response message for the user to ensure safety. This generated message is converted into voice and returned to the user through the device. For example, an automatic response message such as "The police have been notified. Please evacuate to a safe place immediately" may be generated and notify the user.

[1196] This allows users to receive prompt and appropriate responses, allowing them to deal with emergencies with peace of mind. In addition, if operator assistance is required, the server will summarize the user's consultation and display it on the operator's terminal along with their emotional state, allowing the operator to respond promptly according to the situation.

[1197] Examples of concrete examples and prompts

[1198] For example, if a user suddenly discovers an intruder in their home and speaks into the smart glasses, "This is an emergency, someone please help me," the voice will be sent to the server in real time, and based on the converted text and emotion recognition results, the police will be immediately notified, and a message will be generated stating, "The police have been notified. Please evacuate to a safe place immediately."

[1199] Prompt Sentence Examples

[1200] "When a user speaks into a smart device, saying, 'This is an emergency, someone please help,' the voice data is sent to a cloud server, where a speech recognition engine converts it into text data. The emotion recognition engine then recognizes the user's emotional state as 'fear,' and based on that, calls the police and generates an automatic response to the user saying, 'The police have been called. Please evacuate to a safe place immediately.'"

[1201] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1202] Step 1:

[1203] A user speaks to a smart device, for example, "This is an emergency, please help me." Voice data is acquired as input. This voice data is captured through the device's microphone.

[1204] Step 2:

[1205] The device captures audio data, records it as digital data in real time, and sends it to a cloud server. The captured audio data is the input, and the digital data is sent to the cloud server as the output. Specifically, the device digitizes the audio signal and transmits it over the network.

[1206] Step 3:

[1207] The server passes the received voice data to a voice recognition engine. The input is digital voice data, and the voice recognition engine (e.g., Google Cloud Speech-to-Text API) converts the voice into text data. The output is text data. Specifically, the voice waveform is analyzed and words are identified.

[1208] Step 4:

[1209] The server passes the text data to an emotion recognition engine. The input is the text data and speech features, and the emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's speech tone, speed, and intonation to identify their emotional state. The output is the user's emotional state. Specifically, emotion feature vectors are extracted and classified.

[1210] Step 5:

[1211] The server analyzes the user's consultation content based on the generated text data and emotion recognition results, and classifies it as necessary. The input is text data and emotional state, and the machine learning model analyzes the consultation content and classifies it into the appropriate category. The output is the classification result. Specifically, a text classifier, a natural language processing algorithm, is used.

[1212] Step 6:

[1213] Based on the classified consultation content, the server detects an emergency and notifies the necessary external agencies (e.g., police or security companies). The inputs are the classification result and emotional state, and the urgency is evaluated to generate routing information to external agencies. The output is a report message that is generated and sent. Specifically, a report is sent to an external system via an API.

[1214] Step 7:

[1215] The server generates an automatic response message for the user to ensure safety. The inputs are the classification result and the emotional state, and a template response message is selected from a database and adjusted. The output is the generated automatic response message. Specifically, a condition-based message generation algorithm is executed.

[1216] Step 8:

[1217] The generated automated response message is converted into voice and responded to the user via the terminal. The input is the generated text message, and the voice conversion engine converts the text into voice data. The output is a voice message that is conveyed to the user. Specifically, voice synthesis technology is used.

[1218] Step 9:

[1219] Depending on the situation, the server summarizes the user's consultation content and the emotion recognition results, and displays them on the operator's terminal. The input is text data and emotional state, and a summary is generated using a text summarization algorithm. The output is the summary information displayed on the operator's terminal. Specifically, the summary is generated using natural language processing technology.

[1220] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1221] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1222] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1223] [Fourth embodiment]

[1224] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1225] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1226] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1227] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1228] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1230] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1231] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1232] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1233] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1234] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1235] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1236] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] This invention relates to a system that uses voice recognition to improve the operational efficiency of call centers. This system uses voice recognition technology to appropriately classify customer inquiries, enabling automatic responses and efficient support for operators.

[1238] What the program does

[1239] 1. Capture user voice input

[1240] (User)

[1241] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1242] 2. Acquiring and transmitting audio data

[1243] (Terminal)

[1244] The terminal captures the user's voice and transmits the voice data to the server.

[1245] 3. Speech Recognition and Text Conversion

[1246] (server)

[1247] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text into "Internet connection not possible."

[1248] 4. Text Data Analysis and Classification

[1249] (server)

[1250] The server analyzes the text data and classifies the issue based on pre-defined rules and machine learning models. In this case, it is classified as a "connection problem."

[1251] 5. Routing to specialized departments

[1252] (server)

[1253] Route inquiries to network support departments based on the classified inquiry content.

[1254] 6. Generate Auto-Responses

[1255] (server)

[1256] The server generates an auto-reply message based on the classified inquiry, for example, "Please restart your modem."

[1257] 7. Speech conversion and response to the user

[1258] (server)

[1259] The generated automatic response message is converted into voice and responded to the user via the terminal.

[1260] 8. Summary and Operator Assistance

[1261] (server)

[1262] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. In some cases, the operator will respond to the user directly, but the summary information allows for a quicker response.

[1263] Specific examples

[1264] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the device and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" and routed to the network support department.

[1265] The server generates an automatic response message for a frequently asked question, such as "Please try restarting the modem," and converts it into voice to respond to the user. If the automatic response is not appropriate, the server summarizes the inquiry and displays it on the operator's terminal. The operator can respond quickly based on the displayed summary and related FAQs.

[1266] In this way, the system of the present invention makes full use of voice recognition and automatic response technology to enhance user convenience and significantly improve the operational efficiency of call centers.

[1267] The processing flow will be explained below.

[1268] Step 1:

[1269] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1270] Step 2:

[1271] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[1272] Step 3:

[1273] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[1274] Step 4:

[1275] The server analyzes the text data obtained from the speech recognition engine and classifies the consultation content using pre-set rules and machine learning models. In this example, it is classified as a "connection problem."

[1276] Step 5:

[1277] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[1278] Step 6:

[1279] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[1280] Step 7:

[1281] The server generates the prepared auto-answer message and converts it into voice using speech synthesis technology. Specifically, the voice message generated is "Please try restarting your modem."

[1282] Step 8:

[1283] The server sends a voice-converted automated response message to the terminal, which then relays it to the user, who receives the advice "Try restarting your modem."

[1284] Step 9:

[1285] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[1286] Step 10:

[1287] Based on the summary displayed on the terminal and related FAQs, the operator can provide the user with a quick and specific response, such as giving appropriate advice such as "Check the modem settings."

[1288] Through these steps, the system enables quick and accurate responses to user inquiries, while at the same time significantly improving the operational efficiency of call centers.

[1289] Example 1

[1290] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1291] In conventional call centers, operators manually handle inquiries from users, which takes time and manpower, resulting in low operational efficiency. It was particularly difficult to respond quickly when multiple inquiries occurred simultaneously. Furthermore, there was a lack of technology to efficiently convert voice data into text data and properly classify the content, making it difficult to respond accurately or assign calls to the appropriate department.

[1292] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1293] In this invention, the server includes means for acquiring user voice input, means for transmitting the acquired voice to the server as digital data, speech recognition means for converting the acquired voice data into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automatic response message based on the classified consultation content, means for converting the generated automatic response message into voice and responding to the user, and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to process user inquiries quickly and accurately, and to provide an appropriate automatic response or route to a specialized department, thereby significantly improving business efficiency.

[1294] "Means for acquiring user voice input" refers to devices or software for receiving voice from the user, such as a microphone installed on a telephone or smartphone.

[1295] The "means for transmitting the captured audio as digital data to the server" refers to a communication means for converting analog audio into digital form and transmitting it to the server over a network. Specifically, this includes an audio signal converter and a communication protocol.

[1296] "Speech recognition means for converting acquired voice data into text data" refers to technology for analyzing voice and converting it into text information, and a voice recognition engine is an example of this.

[1297] "Means for analyzing the converted text data and classifying the consultation content" refers to algorithms or software that interpret the text data using natural language processing technology and classify it based on pre-set categories.

[1298] "Means of routing to specialized departments based on classified consultation content" refers to a system that automatically assigns inquiries to the appropriate department or person in charge based on the analysis results.

[1299] "A means for generating automatic response messages based on classified inquiry content" refers to a function that automatically creates responses to user inquiries by referring to an FAQ database or template.

[1300] "Means for converting the generated automated response message into speech and responding to the user" refers to technology for converting text-based messages into speech, such as a speech synthesis engine.

[1301] "Means of summarizing the user's inquiry content and displaying it on the operator's terminal as the case may be" refers to a function for concisely summarizing the user's inquiry data and displaying it in a way that the operator can quickly understand.

[1302] The present invention relates to a system that uses a voice recognition function to improve the operational efficiency of a call center. The system acquires voice input from a user, analyzes the voice, and responds appropriately.

[1303] Hardware and software used

[1304] Hardware

[1305] Devices: Phones, smartphones, PCs with headsets

[1306] Server: A server machine with a powerful processor and sufficient memory

[1307] software

[1308] Speech recognition engine: Technology that converts voice data into text data. For example, a general-purpose speech recognition API or in-house developed speech recognition software.

[1309] Natural language processing engine (NLP): A technology that analyzes text data and classifies the content of inquiries.

[1310] Speech synthesis engine: Technology that converts auto-response messages into speech, such as in-house developed speech synthesis software.

[1311] Database: A database system for storing FAQs and template answers.

[1312] Data processing and calculation

[1313] 1. Audio capture and transmission

[1314] A user calls a call center using a telephone or smartphone. The device picks up the user's voice through a microphone and sends the voice data to a server. The sent voice data is then passed to a speech recognition engine within the server.

[1315] 2. Speech Recognition and Text Conversion

[1316] The server uses a speech recognition engine to convert the voice data into text data. For example, the speech recognition engine converts the voice "I can't connect to the Internet" into text. This conversion process involves noise filtering and language modeling.

[1317] 3. Analysis and Classification of Text Data

[1318] The converted text data is passed to an NLP engine on the server, where its content is analyzed. The NLP engine then classifies the text data based on pre-defined rules and machine learning models. For example, it may be classified as a "connection problem."

[1319] 4. Routing to specialized departments

[1320] The server routes the inquiry to the appropriate department based on the classified inquiry content, and uses an internal routing algorithm to transfer the inquiry to the most appropriate department or person.

[1321] 5. Generate Auto-Response Messages

[1322] The server generates an automatic response message from a FAQ database or template based on the classified inquiry content, such as "Please restart your modem."

[1323] 6. Speech synthesis and response to the user

[1324] The generated auto-answer message is passed to a speech synthesis engine and converted into voice data, which is then sent to the user via the terminal in a voice response. For example, the message tells the user to "restart the modem."

[1325] 7. Summary and Operator Assistance

[1326] The server further analyzes the user's inquiry and provides a concise summary. This summary and related FAQs are displayed on the operator's terminal. The operator can then respond quickly based on the displayed summary information.

[1327] Specific examples

[1328] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert it into text data saying, "I can't connect to the Internet." The converted text data is classified as a "connection problem" by an NLP engine and routed to the network support department. The server generates an automated response message from the FAQ database saying, "Please restart your modem," and responds to the user by converting it into speech using a speech synthesis engine. If the automated response is not appropriate, the server summarizes the consultation and displays it on the operator terminal. The operator can respond based on the displayed summary.

[1329] Example prompts for generative AI models

[1330] A user uses their smartphone and says, "I can't connect to the Internet."

[1331] The server uses a speech recognition engine to convert the speech into text data.

[1332] The text data is classified by the NLP engine as a "connection issue" and routed to the network support department.

[1333] The server generates an automatic response message from the FAQ database and converts "Please restart your modem" into speech using a speech synthesis engine.

[1334] The terminal plays the audio data and responds to the user.

[1335] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1336] Step 1:

[1337] The user begins speaking

[1338] The user calls the call center using a telephone or smartphone and verbally communicates the inquiry through a microphone.

[1339] Input: User speech (e.g. "I can't connect to the Internet")

[1340] Output: Audio data (analog signal)

[1341] Step 2:

[1342] The device converts the voice data into digital data and sends it to the server.

[1343] The device converts the audio picked up by the microphone into a digital format and transmits it to a server using a secure communication protocol (e.g., HTTPS).

[1344] Input: Audio data (analog signal)

[1345] Output: Digital audio data

[1346] Step 3:

[1347] The server receives the voice data and passes it to the voice recognition engine.

[1348] The server passes the received digital voice data to a voice recognition engine (e.g., a general-purpose voice recognition API) and converts the voice into text data. It also performs preprocessing such as noise filtering.

[1349] Input: Digital audio data

[1350] Output: Text data (e.g. "Internet connection not available")

[1351] Step 4:

[1352] The server analyzes the text data and classifies the content of the consultation.

[1353] The server uses an NLP engine to analyze the text data, and based on the analysis, classifies the text data based on pre-defined rules and machine learning models.

[1354] Input: Text data (e.g. "Cannot connect to the Internet")

[1355] Output: Classification result (e.g. "connectivity problem")

[1356] Step 5:

[1357] The server routes the request to a specialist department based on the content of the request.

[1358] The server automatically routes inquiries to the most appropriate department based on the classification results, using a routing algorithm to forward them to the network support department, etc.

[1359] Input: Classification result (e.g. "Connectivity problem")

[1360] Output: Routing instructions (e.g., Network Support Department)

[1361] Step 6:

[1362] The server generates an auto-reply message

[1363] The server generates an appropriate auto-reply message from a FAQ database or template, such as "Please restart your modem."

[1364] Input: Classification result (e.g. "Connectivity problem")

[1365] Output: Auto-answer message (e.g. "Please restart your modem")

[1366] Step 7:

[1367] The server converts the auto-answer message into voice and responds to the user.

[1368] The generated automated response message is passed to a speech synthesis engine (e.g., speech synthesis software) and converted into voice data. This voice data is then sent to the terminal, which then responds to the user.

[1369] Input: Auto-answer message (e.g. "Please restart your modem")

[1370] Output: Audio data

[1371] Step 8:

[1372] The server summarizes the user's consultation and displays it on the operator's terminal.

[1373] The server further analyzes and summarizes the user's inquiry, and displays the summary information, including related FAQs, on the operator's terminal, helping the operator respond quickly.

[1374] Input: User inquiry information, related FAQ

[1375] Output: Summary, operator display information

[1376] By implementing the above steps, the system of the present invention can efficiently process user inquiries and enable automatic responses and rapid routing to specialized departments, thereby significantly improving the operational efficiency of call centers and enhancing user convenience.

[1377] (Application example 1)

[1378] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] Conventional self-driving vehicles lack a system that allows passengers to smoothly respond to their requests and questions. This often results in a lack of convenience and comfort for passengers inside the vehicle. Another problem is that there are insufficient means to quickly obtain emergency or specific information. To solve these issues, a highly functional support system that uses voice input is needed.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1381] In this invention, the server includes means for acquiring a user's voice input, speech recognition means for converting the acquired voice into text data, means for analyzing the converted text data and classifying the consultation content, means for routing to a specialized department based on the classified consultation content, means for generating an automated response message based on the classified consultation content, means for converting the generated automated response message into voice and responding to the user, means for summarizing the user's consultation content and displaying it on an operator's terminal as needed, and means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle. This enables passengers to receive prompt and appropriate responses to their inquiries and requests within the autonomous vehicle.

[1382] "Means for obtaining user voice input" refers to devices or software that recognize voice signals emitted by the user and capture them as digital data.

[1383] "Speech recognition means for converting acquired speech into text data" refers to technology that analyzes speech signals and converts them into corresponding text data, and is generally composed of a microphone and a speech recognition engine.

[1384] The "means for analyzing the converted text data and classifying the consultation content" is a method for analyzing the meaning of the text data using natural language processing technology and classifying the consultation content into predefined categories.

[1385] "Means for routing to specialized departments based on classified consultation content" is a system for automatically distributing inquiries to appropriate specialized departments based on the analyzed consultation content.

[1386] The "means for generating automatic response messages" is a system that automatically creates appropriate response messages in response to user inquiries based on a database or template responses.

[1387] The "means for converting the generated automated response message into speech and responding to the user" refers to a technology for converting the text-based automated response message into speech and providing audible feedback to the user.

[1388] The "means for summarizing the user's consultation content and displaying it on the operator's terminal" is a system that has the function of concisely summarizing the user's consultation content and displaying it on the operator's display device.

[1389] "Means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle" refers to a technology that transmits voice instructions given within the vehicle to an in-vehicle server in real time and provides a comfortable in-vehicle environment based on that.

[1390] This invention is a system for significantly improving passenger convenience in an autonomous vehicle. A specific implementation method of this system will be described below.

[1391] Overall system overview

[1392] The server receives voice input from the user through the infotainment system in the vehicle and generates appropriate responses. The system includes the following main functions:

[1393] Acquiring voice input

[1394] Speech-to-text

[1395] Text Analysis and Classification

[1396] Routing to specialized departments

[1397] Generate auto-response messages

[1398] Speech transcription and response

[1399] Summary of user inquiries and operator support

[1400] Passenger voice commands are sent to an in-vehicle server to provide convenience within the vehicle.

[1401] Hardware and software used

[1402] 1. Hardware:

[1403] Autonomous vehicle infotainment systems

[1404] In-vehicle server

[1405] 2. Software:

[1406] Speech recognition engine: To convert voice input into text data, use services such as Amazon Transcribe or Google Cloud Speech-to-Text.

[1407] Natural Language Processing: To analyze text data and classify its content, use, for example, AWS SageMaker and NLTK.

[1408] Response message generation engine: Refers to FAQs and template answers to automatically create appropriate response messages.

[1409] Text conversion engine: For example, using Amazon Polly or Google Cloud Text-to-Speech to convert the generated text response into speech.

[1410] Example

[1411] 1. Acquiring voice input:

[1412] A passenger issues a voice command such as, "Where is the next service area?" This speech is picked up by the infotainment system.

[1413] 2. Speech to text transcription:

[1414] The acquired voice data is sent to an in-vehicle server and converted into text data such as "Where is the next service area?" by a voice recognition engine (e.g., Amazon Transcribe, Google Cloud Speech-to-Text).

[1415] 3. Text Analysis and Classification:

[1416] The server uses natural language processing technology (e.g., AWS SageMaker, NLTK) to analyze the text data and classify it into categories called "nearby facility information."

[1417] 4. Routing to specialized departments:

[1418] Based on the classification, the call is routed to a specialized department (e.g., vehicle support center) if necessary.

[1419] 5. Generate auto-response messages:

[1420] The server refers to FAQs and template answers in the database and generates an automated response message saying, "The next service area is approximately 15 kilometers away."

[1421] 6. Voice conversion and response:

[1422] The generated automated response message is converted into speech using a text conversion engine (e.g., Amazon Polly, Google Cloud Text-to-Speech) and provided to passengers through the infotainment system.

[1423] 7. Summary of user inquiries and operator support:

[1424] If the consultation is complicated, the server will summarize the content and display it on the operator's terminal along with related information, allowing the operator to respond quickly.

[1425] Examples and prompts

[1426] Example: When a passenger asks, "The car is shaking a little. Is everything okay?", the system automatically responds, "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[1427] Example prompt sentence:

[1428] Where is the next service area?

[1429] The car is shaking a little, is everything okay?

[1430] As a result, the system of the present invention can increase passenger convenience and enable comfortable travel in an autonomous vehicle.

[1431] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1432] Step 1:

[1433] The user provides voice input. The passenger speaks a question or request. For example, "Where is the next service area?" The input is the user's voice, and the infotainment system captures the voice data. The output is the voice data.

[1434] Step 2:

[1435] The infotainment system sends the voice data to the in-vehicle server, which receives it in digital format. The input is the voice data, and the output is the voice data in digital format.

[1436] Step 3:

[1437] The in-vehicle server passes the voice data to a voice recognition engine, which converts the voice into text data. The software used is Amazon Transcribe or Google Cloud Speech-to-Text. The input is voice data, which is processed by voice recognition, and the output is text data. Specifically, the text data obtained is, "Where is the next service area?"

[1438] Step 4:

[1439] The server analyzes the text data and classifies the consultation content based on pre-defined rules and machine learning models. The software used is AWS SageMaker and NLTK. The input is text data, an automatic classification algorithm is applied, and the output is a category such as "information on nearby facilities."

[1440] Step 5:

[1441] The server routes inquiries to specialized departments as needed based on the classified content. The input is the classification result, and the output is routing information. For example, if the inquiry is for "information on nearby facilities," the inquiry will be routed to a specific database or department.

[1442] Step 6:

[1443] The server generates an automatic response message based on the classified inquiry content, referencing FAQs and template answers in the database. The input is the category of the inquiry content and information in the database, and the output is the message "The next service area is approximately 15 kilometers away."

[1444] Step 7:

[1445] The generated automated response message is converted into speech and responded to the user via the infotainment system. The software used is Amazon Polly and Google Cloud Text-to-Speech. The input is a text message, which is converted into speech, and the output is a speech response.

[1446] Step 8:

[1447] The server further analyzes and summarizes the user's inquiry. This summary and related FAQs are displayed on the operator's terminal. The input is text data and analysis results, and the output is summary information and display content. As a specific example, in an emergency, a consultation such as "The car is shaking a little. Are you OK?" is summarized and displayed to the operator as "The current shaking is within the automatic adjustment range, but if the abnormality persists, we will contact the support center."

[1448] Through the above steps, smooth responses and services based on passenger voice input are realized.

[1449] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1450] This invention relates to a system that improves the operational efficiency of call centers by using voice recognition functions, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more appropriate responses. This system uses voice recognition technology and emotion recognition technology to understand the content of the user's inquiry and their emotional state, and provides appropriate automatic responses and support to operators.

[1451] What the program does

[1452] 1. Capture user voice input

[1453] (User)

[1454] The user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1455] 2. Acquiring and transmitting audio data

[1456] (Terminal)

[1457] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[1458] 3. Speech Recognition and Text Conversion

[1459] (server)

[1460] The server passes the received voice data to a voice recognition engine, which converts the voice into text data. Specifically, it converts the text data into "Cannot connect to the Internet."

[1461] 4. Emotion Recognition by Emotion Engine

[1462] (server)

[1463] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[1464] 5. Text Data Analysis and Classification

[1465] (server)

[1466] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[1467] 6. Routing to specialized departments

[1468] (server)

[1469] Based on the classified inquiry, the user's inquiry is routed to the appropriate specialist department. In this case, it is routed to the network support department as a "connection problem."

[1470] 7. Generate Auto-Responses

[1471] (server)

[1472] The server generates an automatic response message based on the classified consultation content and also refers to the emotion recognition results. For example, if the user is angry, the tone will be adjusted to say, "We apologize for the inconvenience. Please try restarting your modem."

[1473] 8. Speech conversion and response to the user

[1474] (server)

[1475] The generated automated answering message is converted into voice and responded to the user via the terminal, who receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[1476] 9. Summary and Operator Assistance

[1477] (server)

[1478] The server further analyzes and summarizes the user's inquiry. This summary, along with the emotion recognition results and related FAQs, is displayed on the operator's terminal. In some cases, the operator will respond directly to the user, but the summary information and emotional state provided enable a quick and appropriate response.

[1479] Specific examples

[1480] A user calls a call center and says, "I can't connect to the Internet." This voice is picked up by the device and sent to the server. The server uses a speech recognition engine to convert the voice into text data saying, "I can't connect to the Internet." At the same time, an emotion engine recognizes the emotion "impatience" from the user's voice data. The converted text data is classified as a "connection problem" and routed to the network support department.

[1481] The server, taking into account the emotion recognition results, generates an automatic response message such as "Please try restarting the modem," converts it into speech, and responds to the user. If the automatic response is not appropriate, the server summarizes the consultation content and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state.

[1482] In this way, the system of the present invention makes full use of voice recognition and emotion recognition technologies to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that correspond to the user's emotions.

[1483] The processing flow will be explained below.

[1484] Step 1:

[1485] A user calls the call center and verbally replies with their inquiry. For example, the user says, "I can't connect to the Internet."

[1486] Step 2:

[1487] The device captures the user's voice and records the voice data as digital data in real time, which is then automatically sent to the server.

[1488] Step 3:

[1489] The server passes the received voice data to the voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. Specifically, it converts it into text data saying "Cannot connect to the Internet."

[1490] Step 4:

[1491] The server uses an emotion engine that analyzes features such as the user's voice tone, speed, and intonation from the voice data to recognize the user's emotional state (e.g., anger, sadness, impatience, etc.).

[1492] Step 5:

[1493] The server analyzes the text data and classifies the issue using pre-defined rules and machine learning models. In this example, the issue is classified as a "connection problem."

[1494] Step 6:

[1495] The server routes the user's inquiry to the appropriate department based on the classified issue. In this case, it routes the inquiry to the network support department as a "connection problem."

[1496] Step 7:

[1497] The server determines whether an automatic response is possible for the classified inquiry content. It refers to FAQs and template answers in the database. In this case, it prepares a template answer saying "Try restarting your modem."

[1498] Step 8:

[1499] The server generates a prepared automated response message while referring to the emotion recognition results. If the user is in a state of impatience, it creates a tone-adjusted message such as "We apologize for the inconvenience. Please try restarting your modem."

[1500] Step 9:

[1501] The server converts the generated automated response message into voice and responds to the user via the terminal. Specifically, the voice message "We apologize for the inconvenience. Please try restarting your modem" is transmitted to the user.

[1502] Step 10:

[1503] If the user is unable to resolve the issue through the automated response message, they will contact the operator again. In this case, the server will summarize the user's original inquiry and display common solutions and related FAQs on the operator's terminal.

[1504] Step 11:

[1505] The server displays the user's emotional state (e.g., impatience, anger, sadness, etc.) obtained by the emotion engine on the operator's terminal, allowing the operator to understand the user's emotional state and respond accordingly.

[1506] Step 12:

[1507] Based on the summary displayed on the terminal, relevant FAQs, and the user's emotional state, the operator can provide the user with a quick and specific response, such as "Check your modem settings."

[1508] In this way, this system makes full use of voice recognition and emotion recognition technology to enhance user convenience and significantly improve the operational efficiency of call centers. It also enables detailed responses that reflect the user's emotions.

[1509] Example 2

[1510] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1511] Modern call center operations require accurate understanding of user inquiries and prompt and appropriate responses. However, current systems lack the ability to recognize the user's emotional state, which can lead to lower user satisfaction. In particular, when a user is anxious or irritated, it is difficult to respond appropriately, increasing the burden on operators.

[1512] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for acquiring a user's voice input; speech recognition means for converting the acquired voice into text data; emotion recognition means for recognizing the user's emotional state from the voice data; means for adjusting the tone of the automatic response message based on the emotion recognition result; means for analyzing the converted text data and classifying the consultation content; means for routing to a specialized department based on the classified consultation content; means for generating an automatic response message based on the classified consultation content; means for converting the generated automatic response message into voice and responding to the user; and means for summarizing the user's consultation content and displaying it on the operator's terminal, as the case may be. This makes it possible to accurately grasp not only the content of the user's inquiry but also their emotional state, enabling a prompt and appropriate response.

[1513] A "user" is a person who contacts the call center to make an inquiry or ask for advice.

[1514] "Audio input" refers to audio signals emitted by a user.

[1515] A "terminal" is a device used by a user, and is a device that acquires voice data and transmits it to a server.

[1516] "Speech recognition" refers to the process of analyzing voice data and converting it into text data.

[1517] "Emotion recognition" is a technology that analyzes characteristics such as a user's voice tone, speed, and intonation to determine their emotional state.

[1518] An "automatic response message" is an automatic response text generated by the server to a user's inquiry.

[1519] "Text-to-speech" refers to the process of converting text data into audio data.

[1520] "Routing" is a method of transferring inquiries to appropriate specialist departments based on the classified content of the inquiries.

[1521] "Categorizing consultation content" is the process of analyzing the content of a user's inquiry and classifying it into pre-defined categories.

[1522] "Operator support" is a function that provides the operator with the user's consultation content and emotional state, and supports a quick and appropriate response.

[1523] This invention relates to a system that combines speech recognition technology and emotion recognition technology to improve the operational efficiency of call centers. When a user makes an inquiry to a call center, the system understands the inquiry content and emotional state of the user, and provides appropriate automated responses and support to operators, thereby achieving a fast and accurate response.

[1524] This system is composed of the following elements: users, terminals, and servers. Specifically, it uses the following hardware and software:

[1525] 1. Acquiring voice input

[1526] (User)

[1527] Users contact the call center using devices such as telephones or smartphones. For example, if a user cannot connect to the Internet at home, they will say, "I can't connect to the Internet."

[1528] 2. Acquiring and sending audio data

[1529] (Terminal)

[1530] The device picks up the user's voice using a built-in microphone or an externally connected headset, and the voice data is recorded as digital data in real time and immediately sent to the server.

[1531] 3. Speech Recognition and Text Conversion

[1532] (server)

[1533] The server passes the received voice data to a speech recognition engine (for example, Google Cloud Speech-to-Text API). This engine converts the voice data into text data. Specifically, it converts it into text data that says, "Internet connection not available."

[1534] 4. Emotion recognition

[1535] (server)

[1536] The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the user's voice characteristics, such as tone, speed, and intonation, from the voice data, thereby determining the user's emotional state (e.g., anger, sadness, impatience).

[1537] 5. Text Data Analysis and Classification

[1538] (server)

[1539] The server analyzes the text data using a natural language processing engine (e.g., OpenAI's GPT-3), which extracts intent and keywords from the text and classifies them into predefined categories (e.g., "connection problem").

[1540] 6. Routing to specialized departments

[1541] (server)

[1542] The server routes the user's inquiry to the appropriate department based on the classification of the problem, for example, forwarding it to the network support department as a "connection problem."

[1543] 7. Generate Auto-Responses

[1544] (server)

[1545] The server generates an automatic response message based on the classified consultation content and the emotion recognition results. For example, if the user is angry, the server generates a message saying, "We apologize for the inconvenience. Please try restarting your modem."

[1546] 8. Speech conversion and response to the user

[1547] (server)

[1548] The generated automated response message is converted into speech using a speech synthesis engine (e.g., Amazon Polly) and responded to the user via the terminal. The user receives the advice "We apologize for the inconvenience. Please try restarting your modem."

[1549] 9. Summary and Operator Assistance

[1550] (server)

[1551] The server further analyzes the user's inquiry and creates a summary. This summary, along with the emotion recognition results and related FAQs, are displayed on the operator's terminal. This allows the operator to respond quickly based on the displayed summary and the user's emotional state.

[1552] Specific examples

[1553] A user calls a call center and says, "I can't connect to the Internet." This speech is captured by the terminal and sent to the server. The server uses a speech recognition engine to convert the speech into text data saying, "I can't connect to the Internet." At the same time, an emotion recognition engine recognizes the emotion "impatience" from the user's speech data. The converted text data is classified as a "connection problem" and routed to the network support department. The server, taking into account the emotion recognition results, generates an automated response message saying, "Please try restarting your modem," converts this into speech, and responds to the user. If the automated response is inappropriate, the server summarizes the consultation and displays the operator's emotional state and related FAQs on the operator's terminal. The operator can respond quickly based on the displayed summary and emotional state. This system increases user convenience and improves call center efficiency.

[1554] Prompt Sentence Examples

[1555] Create a program that recognizes user questions and emotions in a call center system. Explain with a concrete example how the system would respond if a user says, "I can't connect to the Internet."

[1556] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1557] Step 1:

[1558] The user calls the call center and speaks their inquiry, for example, "I can't connect to the Internet." This voice input is sent from the user to the terminal.

[1559] Step 2:

[1560] The device captures the user's voice using a built-in microphone or an externally connected headset. The captured voice data is recorded as digital data in real time. This digital voice data is then sent from the device to the server.

[1561] Step 3:

[1562] The server passes the received voice data to a speech recognition engine. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data. In this process, the voice waveform is analyzed and converted into a string of characters based on a language model. The output text data is, for example, "Internet connection not available."

[1563] Step 4:

[1564] The server passes the received voice data to an emotion recognition engine. Specifically, it uses IBM Watson Tone Analyzer to analyze the voice tone, speed, and intonation characteristics. This analysis determines the user's emotional state. The output data includes emotional states such as "impatience."

[1565] Step 5:

[1566] The server passes the text data to a natural language processing engine for analysis. Specifically, it uses OpenAI's GPT-3 to extract intent and keywords from the text. Based on the input text, it classifies it into categories such as "connection problems." The output of this process is called a "connection problem."

[1567] Step 6:

[1568] The server routes the call to the appropriate department based on the classified text data, for example, "Connection Problems" and forwards it to the network support department. A custom rules-based routing engine is used to determine the route to the department.

[1569] Step 7:

[1570] The server references the classified text data and emotion recognition results to generate an automated response message. This uses FAQs and template answers stored in the database. The inputs are "Connection problem" and "Impatience," and the output is a message that reads, "We apologize for the inconvenience. Please try restarting your modem."

[1571] Step 8:

[1572] The server passes the generated automated response message to a speech synthesis engine and converts it into voice data. Specifically, Amazon Polly is used to convert the text message into voice. The generated voice data is sent to the user via the terminal. The user receives a voice response saying, "We apologize for the inconvenience. Please try restarting your modem."

[1573] Step 9:

[1574] The server further analyzes and summarizes the user's consultation content and the emotion recognition results. The created summary information, emotional state, and related FAQs are displayed on the operator's terminal. This allows the operator to respond quickly based on the summary content and emotional state. Specifically, the operator's screen displays the message, "The user is anxious about an Internet connection problem. We advise you to try restarting the modem."

[1575] (Application example 2)

[1576] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1577] Conventional security systems have had issues with users reporting abnormalities late and being unable to respond appropriately in emergencies. Furthermore, conventional voice recognition systems are unable to take into account the user's emotional state, making it difficult to respond quickly to highly urgent situations. Furthermore, because reporting to appropriate external agencies is not automated, prompt assistance may not be available.

[1578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1579] In this invention, the server includes an emotion recognition means for identifying an emotional state from the user's voice data, a means for determining an optimal response based on the identified emotion, and a means for detecting an emergency and notifying an appropriate external agency, thereby enabling the user's emergency to be quickly detected and an appropriate response to be taken.

[1580] "Voice input" refers to the user speaking into a device.

[1581] "Speech recognition" refers to the technology of analyzing captured speech as digital data and converting it into text data.

[1582] "Text data" refers to text information converted using voice recognition technology.

[1583] "Emotion recognition" refers to the technology of identifying a user's emotional state from voice data.

[1584] "Emotional state" is information that indicates the mental state of the user, and specifically refers to anger, sadness, impatience, and the like.

[1585] "Rational response" refers to the optimal response generated based on the classified consultation content and the user's emotional state.

[1586] A "specialized department" refers to a department that handles specific consultation matters.

[1587] "Routing" refers to the series of processes that send classified consultation content to the appropriate department or external agency.

[1588] An "automatic response message" refers to an automatically generated reply message to a user.

[1589] "Text-to-speech" refers to the process of converting text messages back into audio data.

[1590] An "emergency" is any situation that requires immediate action.

[1591] "External agencies" refer to support organizations outside the system, such as police and security companies.

[1592] This system combines voice recognition technology and emotion recognition technology to enhance user convenience and respond quickly and appropriately to emergency situations.

[1593] First, when a user speaks into a smart device (smartphone, smart glasses, head-mounted display, etc.), the device picks up the user's voice and sends it as digital data to a cloud server. The voice data is collected in real time and sent to the server immediately.

[1594] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the captured speech into text data. Once the text data is generated, an emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the speech data and identifies the user's emotional state. Emotional states include anger, sadness, impatience, and fear.

[1595] The generated text data and emotion recognition results are then used to classify the user's consultation and, if necessary, automatically notify the appropriate external agency (such as the police or security company). For example, if a user says, "This is an emergency, someone please help," the emotion recognition engine will determine this as "fear," and based on that, a call will be made immediately to the security company or police.

[1596] Furthermore, the server generates an automatic response message for the user to ensure safety. This generated message is converted into voice and returned to the user through the device. For example, an automatic response message such as "The police have been notified. Please evacuate to a safe place immediately" may be generated and notify the user.

[1597] This allows users to receive prompt and appropriate responses, allowing them to deal with emergencies with peace of mind. In addition, if operator assistance is required, the server will summarize the user's consultation and display it on the operator's terminal along with their emotional state, allowing the operator to respond promptly according to the situation.

[1598] Examples of concrete examples and prompts

[1599] For example, if a user suddenly discovers an intruder in their home and speaks into the smart glasses, "This is an emergency, someone please help me," the voice will be sent to the server in real time, and based on the converted text and emotion recognition results, the police will be immediately notified, and a message will be generated stating, "The police have been notified. Please evacuate to a safe place immediately."

[1600] Prompt Sentence Examples

[1601] "When a user speaks into a smart device, saying, 'This is an emergency, someone please help,' the voice data is sent to a cloud server, where a speech recognition engine converts it into text data. The emotion recognition engine then recognizes the user's emotional state as 'fear,' and based on that, calls the police and generates an automatic response to the user saying, 'The police have been called. Please evacuate to a safe place immediately.'"

[1602] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1603] Step 1:

[1604] A user speaks to a smart device, for example, "This is an emergency, please help me." Voice data is acquired as input. This voice data is captured through the device's microphone.

[1605] Step 2:

[1606] The device captures audio data, records it as digital data in real time, and sends it to a cloud server. The captured audio data is the input, and the digital data is sent to the cloud server as the output. Specifically, the device digitizes the audio signal and transmits it over the network.

[1607] Step 3:

[1608] The server passes the received voice data to a voice recognition engine. The input is digital voice data, and the voice recognition engine (e.g., Google Cloud Speech-to-Text API) converts the voice into text data. The output is text data. Specifically, the voice waveform is analyzed and words are identified.

[1609] Step 4:

[1610] The server passes the text data to an emotion recognition engine. The input is the text data and speech features, and the emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's speech tone, speed, and intonation to identify their emotional state. The output is the user's emotional state. Specifically, emotion feature vectors are extracted and classified.

[1611] Step 5:

[1612] The server analyzes the user's consultation content based on the generated text data and emotion recognition results, and classifies it as necessary. The input is text data and emotional state, and the machine learning model analyzes the consultation content and classifies it into the appropriate category. The output is the classification result. Specifically, a text classifier, a natural language processing algorithm, is used.

[1613] Step 6:

[1614] Based on the classified consultation content, the server detects an emergency and notifies the necessary external agencies (e.g., police or security companies). The inputs are the classification result and emotional state, and the urgency is evaluated to generate routing information to external agencies. The output is a report message that is generated and sent. Specifically, a report is sent to an external system via an API.

[1615] Step 7:

[1616] The server generates an automatic response message for the user to ensure safety. The inputs are the classification result and the emotional state, and a template response message is selected from a database and adjusted. The output is the generated automatic response message. Specifically, a condition-based message generation algorithm is executed.

[1617] Step 8:

[1618] The generated automated response message is converted into voice and responded to the user via the terminal. The input is the generated text message, and the voice conversion engine converts the text into voice data. The output is a voice message that is conveyed to the user. Specifically, voice synthesis technology is used.

[1619] Step 9:

[1620] Depending on the situation, the server summarizes the user's consultation content and the emotion recognition results, and displays them on the operator's terminal. The input is text data and emotional state, and a summary is generated using a text summarization algorithm. The output is the summary information displayed on the operator's terminal. Specifically, the summary is generated using natural language processing technology.

[1621] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1622] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1623] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1624] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1625] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1626] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1627] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1628] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1629] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1630] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1631] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1632] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1633] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1634] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1635] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1636] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1637] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1638] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1639] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1640] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1641] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1642] The following is further disclosed regarding the above embodiment.

[1643] (Claim 1)

[1644] means for obtaining a user's voice input;

[1645] A speech recognition means for converting the acquired speech into text data;

[1646] A means for analyzing the converted text data and classifying the consultation content;

[1647] A means of routing to specialized departments based on the classified consultation content;

[1648] means for generating an automatic response message based on the classified consultation content;

[1649] means for converting the generated automatic response message into voice and responding to the user;

[1650] A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary;

[1651] A system including:

[1652] (Claim 2)

[1653] 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary or model containing technical terms.

[1654] (Claim 3)

[1655] 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database.

[1656] "Example 1"

[1657] (Claim 1)

[1658] means for obtaining a user's voice input;

[1659] means for transmitting the acquired voice to a server as digital data;

[1660] a speech recognition means for converting the acquired speech data into text data;

[1661] A means for analyzing the converted text data and classifying the consultation content;

[1662] A means of routing to specialized departments based on the classified consultation content;

[1663] means for generating an automatic response message based on the classified consultation content;

[1664] means for converting the generated automatic response message into voice and responding to the user;

[1665] A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary;

[1666] A system including:

[1667] (Claim 2)

[1668] 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary or model containing technical terms.

[1669] (Claim 3)

[1670] 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database.

[1671] "Application Example 1"

[1672] (Claim 1)

[1673] means for obtaining a user's voice input;

[1674] A speech recognition means for converting the acquired speech into text data;

[1675] A means for analyzing the converted text data and classifying the consultation content;

[1676] A means of routing to specialized departments based on the classified consultation content;

[1677] means for generating an automatic response message based on the classified consultation content;

[1678] means for converting the generated automatic response message into voice and responding to the user;

[1679] A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary;

[1680] means for transmitting passenger voice commands to an in-vehicle server to provide convenience within the vehicle;

[1681] A system including:

[1682] (Claim 2)

[1683] 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary or model containing technical terms.

[1684] (Claim 3)

[1685] 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database.

[1686] "Example 2: Combining Emotion Engines"

[1687] (Claim 1)

[1688] means for obtaining a user's voice input;

[1689] A speech recognition means for converting the acquired speech into text data;

[1690] A means for analyzing the converted text data and classifying the consultation content;

[1691] A means of routing to specialized departments based on the classified consultation content;

[1692] means for generating an automatic response message based on the classified consultation content;

[1693] means for converting the generated automatic response message into voice and responding to the user;

[1694] A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary;

[1695] emotion recognition means for recognizing an emotional state of a user from voice data;

[1696] A means for adjusting the tone of an auto-reply message based on the emotion recognition results;

[1697] A system including:

[1698] (Claim 2)

[1699] 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary or model containing technical terms.

[1700] (Claim 3)

[1701] 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database.

[1702] "Application example 2 when combining emotion engines"

[1703] (Claim 1)

[1704] means for obtaining a user's voice input;

[1705] A speech recognition means for converting the acquired speech into text data;

[1706] A means for analyzing the converted text data and classifying the consultation content;

[1707] A means of routing to specialized departments based on the classified consultation content;

[1708] means for generating an automatic response message based on the classified consultation content;

[1709] means for converting the generated automatic response message into voice and responding to the user;

[1710] emotion recognition means for identifying an emotional state of a user from voice data;

[1711] means for determining an optimal response based on the identified emotion;

[1712] A means of detecting and reporting emergencies to appropriate external authorities;

[1713] A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary;

[1714] A system including:

[1715] (Claim 2)

[1716] 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary or model containing technical terms.

[1717] (Claim 3)

[1718] 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database. [Explanation of symbols]

[1719] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for obtaining a user's voice input; A speech recognition means for converting the acquired speech into text data; A means for analyzing the converted text data and classifying the consultation content; A means of routing to specialized departments based on the classified consultation content; means for generating an automatic response message based on the classified consultation content; means for converting the generated automatic response message into voice and responding to the user; A means for summarizing the user's consultation content and displaying it on the operator's terminal, if necessary; A system including:

2. 2. The system of claim 1, wherein the speech recognition means converts speech into text data using a dictionary and models containing technical terms.

3. 2. The system according to claim 1, wherein the means for generating the automatic response message generates the automatic response message by referring to FAQs and template answers in the database.

Images