system

Abstract

Provide a system. [Solution] a voice collecting means for collecting voices during customer service operations; a transcription means for converting the collected voice data into text data; an analysis means for analyzing the converted character data and evaluating the customer service content; visualization means for visualizing the evaluated data; providing means for providing the visualized data to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In customer service operations, there is a need to clearly identify the customer service performance of high performers and improve overall service quality. However, traditional methods have made it difficult to quantify the skill gaps between staff members and provide specific guidance for improvement. Furthermore, there has been a lack of a way for managers to evaluate each staff member's performance in real time and provide feedback. For this reason, a system that can efficiently evaluate and improve customer service operations is needed. [Means for solving the problem]

[0005] The present invention provides a system that includes an audio collection means for collecting audio recorded during customer service operations, a transcription means for converting the collected audio data into text data, an analysis means for analyzing the converted text data to evaluate the customer service content, a visualization means for visualizing the evaluated data, and a provision means for providing the visualized data to a user. This system enables real-time evaluation of customer service performance and clearly identifies specific areas for improvement. Specifically, the system collects audio in real time using an AI microphone, converts the audio into text data using a transcription engine, and then analyzes it using a natural language processing engine to quantify staff performance and visually display it on a dashboard. This system allows managers to provide quick and accurate feedback and improve the customer service skills of the entire staff.

[0006] The "voice collection means" is a device or method for collecting voices during customer service operations in real time.

[0007] A "transcription means" is a technology or device for converting collected audio data into text data.

[0008] The "analysis means" is a technique or method for analyzing the converted text data and evaluating the customer service content.

[0009] A "visualizer" is a system or method for visually displaying evaluated data.

[0010] "Providing means" refers to a technology or system for providing visualized data to users.

[0011] "Customer service" refers to a series of activities involved in providing services to customers.

[0012] "High performers" refer to staff who demonstrate high performance in customer service work.

[0013] "Real-time" means that data is collected and processed immediately, without delay.

[0014] "Natural language processing" is a general term for technology that allows computers to understand and analyze human language. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, and a provision means.

[0037] System configuration and processing flow

[0038] 1. Collection of audio data

[0039] Terminal (AI microphone): When a staff member serving a customer begins serving them, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0040] 2. Transcription of audio data

[0041] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0042] 3. Data Analysis

[0043] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data and performs sentiment analysis and intent detection. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs.

[0044] 4. Performance visualization

[0045] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0046] 5. Feedback

[0047] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0048] Specific examples

[0049] Imagine a scene in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This voice is collected by a device (AI microphone) and sent to a server. The server transcribes the received voice data and analyzes the text using an analysis method (NLP engine). As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0050] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0054] Step 2:

[0055] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0056] Step 3:

[0057] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0058] Step 4:

[0059] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0060] Step 5:

[0061] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0062] Step 6:

[0063] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0064] Step 7:

[0065] The server stores the analysis results obtained from the NLP engine in a database, including extracted keywords, sentiment scores, and category classifications.

[0066] Step 8:

[0067] The server generates a dashboard based on the stored data, which visually displays the analysis results, such as graphs and charts showing high customer satisfaction for a particular staff member.

[0068] Step 9:

[0069] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0070] Step 10:

[0071] Users can provide feedback based on the dashboard data, specifically sharing the customer service methods of high performers with other staff members and providing advice on how to improve overall service quality.

[0072] Through this series of processes, customer service data is collected, analyzed, and visualized, enabling specific feedback and service improvements.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Modern customer service operations require accurate and efficient methods for evaluating employee service quality and customer satisfaction. However, traditional evaluation methods are subjective and time-consuming, resulting in issues with accuracy and efficiency. Furthermore, analysis of customer service content and feedback of evaluation results are not centrally managed, making it difficult to provide appropriate information to improve overall service quality.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization tool for visualizing the analyzed data, a provision means for providing the visualized data to a user, and a means for saving the analysis results in a database and displaying them on a dashboard accessible to a manager. This makes it possible to objectively and efficiently evaluate the quality of customer service operations and visually provide the analysis results to a manager, thereby improving employee performance and overall service quality.

[0078] The "voice collection means" is a means for collecting voice data in real time during customer service operations and transmitting the data to a server.

[0079] The "transcription means" is a means for converting collected voice data into text data. Specifically, it has a function of converting voice data into text using a transcription engine.

[0080] The "analysis means" is a means for analyzing the converted text data using a natural language processing engine, for example, to extract keywords, analyze sentiment, and detect intent.

[0081] A "visualization tool" is a tool for visually displaying the evaluation data obtained by the analysis means. Specifically, it is a tool that has the function of retrieving data from a database and displaying it in the form of a graph, chart, etc.

[0082] The "provision means" is a means for providing visualized data to a user. For example, it has a function of presenting the visualized data in a form accessible to an administrator via a dashboard.

[0083] A "database" is a device or system that stores information to store analyzed results and make them available for retrieval as needed.

[0084] A "dashboard" is an interface for visually displaying analysis results and evaluation data in a form accessible to administrators.

[0085] This invention relates to a system that collects voices during customer service operations, analyzes the data, evaluates the customer service, and provides the results to users. This system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization tool, and a presentation means.

[0086] System configuration

[0087] Audio collection method

[0088] A device (AI microphone) is used to collect voices during customer service. The device records conversations with customers in real time, compresses the data, and sends it to a server. For example, when a restaurant staff member says, "Welcome, what would you like to order today?", this voice data is collected.

[0089] Transcription methods

[0090] The server receives the voice data sent from the device. The received voice data is sent to a transcription engine such as Google® Cloud Speech-to-Text and converted into text data. For example, the voice data "Welcome, what are you looking for today?" is converted into text data.

[0091] Analysis means

[0092] The server then sends the converted text data to a natural language processing (NLP) engine, such as the Google Cloud Natural Language API, for analysis. The analysis extracts keywords from the text data and performs sentiment analysis and intent detection. For example, it recognizes the phrase "Welcome" as a greeting and classifies "What are you looking for?" as a customer need confirmation.

[0093] Visualization Tools

[0094] The server stores the analyzed results in a database. The server then uses a visualization tool (e.g., Tableau or Google Data Studio) to display this data on a dashboard. The dashboard visually displays data on which specific staff members have achieved high customer satisfaction using graphs and charts.

[0095] Providing means

[0096] Users (managers and staff) can access the dashboard and evaluate staff performance based on visualized data. By providing specific feedback to staff, they can improve the quality of customer service. For example, they can point out that a particular greeting phrase leads to high customer satisfaction and encourage other staff to use the same greeting.

[0097] Specific examples

[0098] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" The audio of this exchange is collected by a device (AI microphone) and sent to a server. The server converts the received audio into text data using a transcription engine, and analyzes the text using an NLP engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, making it clear that Staff A's customer service is of high quality. The manager can use this data as a reference to provide feedback to other staff members, encouraging them to provide similar customer service.

[0099] Prompt Sentence Examples

[0100] Here are some examples of prompts for generative AI models:

[0101] Please model a customer service scenario in a clothing store. Assuming a scenario in which staff member A says to customer B, "Welcome, is there anything you're looking for today?", explain the process from collecting voice data to analyzing, visualizing, and providing feedback.

[0102] This allows the system to objectively and efficiently evaluate the quality of customer service operations and provide users with visual analysis results, thereby improving employee performance and overall service quality.

[0103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0104] System program processing flow

[0105] Step 1: Collecting audio data

[0106] Terminal (AI microphone): When the terminal detects a user (staff member) starting customer service, it automatically starts collecting voice data.

[0107] Detailed operation: The terminal uses a microphone to record voice in real time and compresses the recorded voice data. For example, when a staff member says, "Welcome, what would you like to buy today?", this voice data is collected.

[0108] Input: Voice data (user speech)

[0109] Output: Compressed audio data (for transfer)

[0110] Step 2: Sending audio data

[0111] Terminal (AI microphone): The terminal sends compressed audio data to the server.

[0112] Detailed operation: The device uploads the collected and compressed audio data to a server via the Internet.

[0113] Input: Compressed audio data

[0114] Output: Audio data sent to the server

[0115] Step 3: Transcription

[0116] Server: The server receives the voice data sent from the terminal.

[0117] How it works in detail: The server sends the audio data to a transcription engine such as Google Cloud Speech-to-Text, which converts it into text data.

[0118] Input: Audio data

[0119] Output: Text data (speech-to-text conversion)

[0120] Step 4: Data analysis

[0121] Server: The server sends the transcribed text data to a natural language processing (NLP) engine.

[0122] How it works in detail: The server uses an NLP engine such as Google Cloud Natural Language API to perform keyword extraction, sentiment analysis, and intent detection from text data.

[0123] Input: Text data

[0124] Output: Analysis results (keywords, sentiment, intent)

[0125] Step 5: Save your data

[0126] Server: The server stores the analysis results in a database.

[0127] Detailed operation: The server records the analysis results in a database such as MySQL (registered trademark) or PostgreSQL, making the data available for retrieval as needed.

[0128] Input: Analysis results

[0129] Output: Data stored in the database

[0130] Step 6: Visualize in a dashboard

[0131] Server: The server uses visualization tools to display the analysis results on a dashboard.

[0132] How it works in detail: The server uses visualization tools like Tableau or Google Data Studio to extract the necessary data from the database, convert it into graphs and charts, and display it on a dashboard.

[0133] Input: Data stored in a database

[0134] Output: Visualized data on a dashboard

[0135] Step 7: Feedback

[0136] User (Manager): The user accesses the dashboard and evaluates staff performance based on visualized data.

[0137] Detailed Action: The manager reviews the data displayed on the dashboard and provides specific feedback to staff, for example, "A specific greeting phrase leads to high satisfaction."

[0138] Input: Data visualized on the dashboard

[0139] Output: Feedback (instructions for staff)

[0140] This series of processes allows the quality of customer service to be objectively and efficiently evaluated, and the analysis results are presented to the user visually, making it possible to improve employee performance and overall service quality.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] Conventional customer service performance evaluation systems have the problem that it is difficult to analyze customer service content in real time and provide immediate feedback to staff. In addition, there are insufficient means to visually display the evaluation data, making it difficult to effectively support staff performance improvement.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a means for visualizing data collected during customer service operations and having a display device that allows the user to check the data. This enables real-time analysis and visual display of customer service content, provides immediate feedback to staff, and enables improvement of customer service performance.

[0146] "Speech collection means" refers to a device or technology that collects speech generated during customer service operations.

[0147] A "transcription means" is a device or technology that converts collected audio data into text data.

[0148] The "analysis means" refers to a device or technology that analyzes the converted character data and evaluates the customer service content.

[0149] "Visualization means" refers to a device or technology that visually displays the evaluated data.

[0150] "Providing means" refers to the device or technology that provides visualized data to users.

[0151] A "display device" is a device that visualizes data collected during customer service operations and allows users to check it.

[0152] To implement the present invention, the following system configuration and processing flow are used.

[0153] System Configuration

[0154] The system includes the following hardware and software:

[0155] Hardware:

[0156] Audio collection means (e.g., microphone in smart glasses)

[0157] Server (e.g. cloud server)

[0158] Display devices (e.g., smart glasses displays or external monitors)

[0159] software:

[0160] Audio collection library (e.g., Google Cloud Speech-to-Text API)

[0161] Natural language processing engines (e.g., spaCy, Google Cloud Natural Language API)

[0162] Database (e.g. Firebase Realtime Database)

[0163] Data visualization tools (e.g., Google Data Studio)

[0164] Program processing

[0165] The server receives real-time voice data from voice collection devices such as smart glasses during customer service. The voice data is converted into text data using the Google Cloud Speech-to-Text API. The converted text data is then analyzed using the Google Cloud Natural Language API. This analysis involves keyword extraction, sentiment analysis, and intent detection. The analysis results are stored in the Firebase Realtime Database and then visually displayed using Google Data Studio.

[0166] Explanations and examples

[0167] For example, imagine a scenario where a staff member at a clothing store greets a customer through smart glasses, saying, "Welcome, what are you looking for today?" This speech is collected by the smart glasses' microphone and sent to a cloud server. The server converts the speech into text data using the Google Cloud Speech-to-Text API, and then analyzes the data using the Google Cloud Natural Language API. Through this analysis, the phrase "Welcome" is recognized as a greeting, and "What are you looking for today?" is classified as a confirmation of the customer's needs. The analysis results are stored in the Firebase Realtime Database and visualized using Google Data Studio.

[0168] Staff and managers can view the analysis results in real time on the smart glasses display or an external monitor, and receive immediate feedback, which can help improve customer service performance.

[0169] Prompt Sentence Examples

[0170] By inputting the prompt below into the generative AI model, you can check the application overview and processing flow.

[0171] "Based on the following description, please provide a detailed description of an application that analyzes customer service performance in a brick-and-mortar store in real time and provides feedback through smart glasses."

[0172] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0173] Step 1:

[0174] Audio collection method

[0175] The device (the microphone in the smart glasses) collects voice data in real time during customer service. Voice is captured from the environment and stored as a digital audio file within the device. This becomes input data. The device then transmits the collected voice data to a server.

[0176] Step 2:

[0177] Sending audio data

[0178] The collected voice data is sent from the device to the server in real time. Specifically, the voice data is uploaded to a cloud server via a mobile communication network. The input is the voice data collected by the device, and the server receives this data.

[0179] Step 3:

[0180] Transcription of audio data

[0181] The server sends the received voice data to the Google Cloud Speech-to-Text API, which converts it into text data. This API performs speech analysis and outputs text data. The input is voice data, and the output is the corresponding text data.

[0182] Step 4:

[0183] Parsing character data

[0184] The server uses the Google Cloud Natural Language API to analyze text data. Specifically, it uses this API to extract keywords, analyze sentiment, and detect intent from text. The input is text data, and the output is the analysis results (keyword list, sentiment score, and intent label).

[0185] Step 5:

[0186] Saving analysis data

[0187] The server stores the analysis results in the Firebase Realtime Database. The input is the analysis data, and the output is the state stored in the database. This data is later used for visualization and feedback.

[0188] Step 6:

[0189] Data Visualization

[0190] After the analysis results are saved, the server visualizes these data using Google Data Studio. The visualization tool generates a dashboard that displays the analysis results in the form of graphs and charts. The input is the saved analysis data, and the output is the visually displayed information.

[0191] Step 7:

[0192] Providing Feedback

[0193] Users (staff or managers) can access visualized data through the smart glasses display or an external monitor and receive real-time feedback. The input is visual data, and the output is feedback information confirmed by the user. This helps users to immediately improve their customer service performance.

[0194] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0195] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, a presentation means, and an emotion engine.

[0196] System configuration and processing flow

[0197] 1. Collection of audio data

[0198] Terminal (AI microphone): When a wait staff member begins serving a customer, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0199] 2. Transcription of audio data

[0200] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0201] 3. Data Analysis

[0202] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data, and the emotion engine recognizes the user's emotions. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs. In addition, the emotion engine extracts emotional data on whether the customer is satisfied based on the customer's voice response.

[0203] 4. Performance visualization

[0204] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0205] 5. Feedback

[0206] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0207] Specific examples

[0208] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using an analysis method (NLP engine) and an emotion engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and choice of words. These results are stored in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0209] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality. In addition, by incorporating customer sentiment data, more detailed and specific feedback is possible, helping to improve staff customer service skills.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0213] Step 2:

[0214] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0215] Step 3:

[0216] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0217] Step 4:

[0218] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0219] Step 5:

[0220] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0221] Step 6:

[0222] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0223] Step 7:

[0224] The server passes the analysis results to the emotion engine, which recognizes the customer's emotions (such as "satisfaction," "dissatisfaction," or "happiness") from the customer's tone of voice and the way they use words.

[0225] Step 8:

[0226] The server integrates the emotion data from the emotion engine with the analysis results from the NLP engine, including extracted keywords, customer emotion scores, and emotion categories.

[0227] Step 9:

[0228] The server stores the consolidated data in a database, which includes analytical data and emotion data corresponding to each customer interaction event.

[0229] Step 10:

[0230] The server generates a dashboard based on the stored data. The dashboard visually displays each staff member's performance and customer sentiment data. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction.

[0231] Step 11:

[0232] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0233] Step 12:

[0234] Users can provide feedback based on the dashboard data, specifically offering specific advice to other staff members based on the customer service methods of high performers and customer sentiment data.

[0235] This process allows for the collection, analysis, and visualization of customer service data, enabling specific feedback and service improvement. Customer sentiment data is also taken into account, enabling more detailed and accurate evaluations.

[0236] Example 2

[0237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] Effective evaluation and feedback of staff performance in customer service operations is a challenge. In particular, efficient collection and accurate analysis of voice data during customer interactions is essential to improve staff customer service skills. Furthermore, integrating real-time data collection and emotion data analysis is essential to provide more detailed and specific feedback and improve overall service quality.

[0239] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0240] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization means for visualizing the evaluated data, an emotion engine means for analyzing emotion data, and a provision means for providing the visualized data to the user. This makes it possible to collect voice data during customer service operations in real time and provide highly accurate feedback based on the analysis results.

[0241] The "voice collection means" is a device or system for collecting and recording voices during customer service operations in real time.

[0242] A "transcription means" is software or hardware for converting collected audio data into text data.

[0243] "Analysis means" refers to natural language processing engines and other analysis tools that use text data to evaluate customer service content.

[0244] A "visualization means" is a system that includes dashboards and graph generation tools for visually displaying the evaluated data.

[0245] The "emotion engine means" is software or hardware for analyzing emotions from voice data or text data and recognizing the user's emotional state.

[0246] "Providing means" refers to the interface or system for providing visualized data to users.

[0247] The present invention is a system that collects voices during customer service operations, analyzes the data, evaluates the service content and customer emotions, and visualizes them. It mainly includes a voice collection means, a transcription means, an analysis means, a visualization means, an emotion engine means, and a provision means.

[0248] Audio collection method

[0249] The device (e.g., an AI microphone) automatically starts collecting voice data when a waiter begins serving a customer. The device records the conversation in real time and sends the collected voice data to a server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0250] Transcription methods

[0251] The server receives the voice data sent from the device and converts the voice into text data using a transcription engine (e.g., Google Cloud Speech-to-Text API). For example, the collected voice "Welcome, what are you looking for today?" is converted into text data.

[0252] Analysis means

[0253] The server analyzes the transcribed text data using a natural language processing (NLP) engine (e.g., spaCy) and an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer). Through this analysis, keywords are extracted from the text data and the customer service content is classified. Emotional data is also extracted from the customer's voice response. For example, "Welcome" is classified as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Emotional data such as "satisfied" or "dissatisfied" is also extracted from the customer's tone of voice and choice of words.

[0254] Visualization means

[0255] The server stores the analyzed data in a database and displays it visually on a dashboard. This dashboard shows the customer satisfaction achieved by a specific staff member in the form of graphs and charts. For example, it becomes possible to see at a glance that a specific staff member has achieved high customer satisfaction. This dashboard can be accessed by administrators.

[0256] Providing means

[0257] Users (managers or staff) can analyze staff performance based on the visualized data and provide specific feedback. For example, they can refer to the dashboard evaluation data and point out that a particular greeting phrase leads to high customer satisfaction, and recommend similar customer service to other staff members.

[0258] Specific examples

[0259] Imagine a scenario in a clothing store where Staff A greets Customer B with, "Welcome, are you looking for anything today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using analytical methods (NLP engine and emotion engine). As a result, "Welcome" is classified as a greeting, and "Are you looking for anything?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and phrasing. These results are stored in a database and displayed on a dashboard, visualizing the quality of Staff A's customer service. Managers can use this data to provide specific feedback to other staff members and work to improve overall customer service skills.

[0260] Prompt Sentence Examples

[0261] Enter the following prompt into the generative AI model:

[0262] "Please explain the specific process flow of a system that analyzes voice data during customer service operations, visualizes staff performance, and provides feedback. Please provide a detailed description of the specific operations at each step."

[0263] In this way, this system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0265] Step 1:

[0266] The device (AI microphone) automatically starts collecting voice data when a waiter starts interacting with a customer. Specifically, the device records the conversation in real time and sends the voice data to a server.

[0267] Input: Audio of conversation between wait staff and customer.

[0268] Output: The audio data sent to the server.

[0269] How it works: The multi-function AI microphone starts collecting voice data the moment a waiter says, "Welcome, what would you like to order today?" The collected voice data is then compressed and sent to a server in real time via Wi-Fi or Bluetooth.

[0270] Step 2:

[0271] The server receives the voice data sent from the terminal and converts the voice data into text data using a transcription engine.

[0272] Input: Audio data sent from the device.

[0273] Output: Character data.

[0274] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the received audio, "Welcome, what are you looking for today?", into text data in real time.

[0275] Step 3:

[0276] The server analyzes the transcribed text data using a natural language processing (NLP) engine and an emotion engine.

[0277] Input: Character data.

[0278] Output: Analysis results (keywords and sentiment data).

[0279] Specific operation: The server uses spaCy to classify text data, such as "Welcome" as a greeting and "What are you looking for?" as confirmation of customer needs. At the same time, it uses IBM Watson Tone Analyzer to extract emotional data such as "satisfied" or "dissatisfied" from the customer's voice response.

[0280] Step 4:

[0281] The server stores the analyzed results in a database and displays them visually on a dashboard.

[0282] Input: Analysis results (keywords and sentiment data).

[0283] Output: Visualized data (graphs and charts).

[0284] What it does: The server stores the analysis results in a database and uses dashboard software such as Tableau to display the staff member customer satisfaction in the form of graphs and charts. This dashboard can be accessed by managers, allowing them to see at a glance the performance of specific staff members.

[0285] Step 5:

[0286] Users (managers, staff) analyze performance based on the visualized data and provide specific feedback.

[0287] Input: Visualized data (graphs and charts).

[0288] Output: Feedback (specific improvements and enhancements).

[0289] Specific Action: The manager opens the dashboard and checks the customer satisfaction data for a specific staff member. Then, based on the dashboard evaluation data, the manager points out that "a specific greeting phrase leads to high satisfaction" and provides feedback recommending similar customer service to other staff members. The manager also plans regular training sessions to improve overall customer service skills.

[0290] (Application example 2)

[0291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] Efficient evaluation and improvement of customer service operations requires a system that comprehensively covers the entire process of collecting, converting, analyzing, visualizing, and providing feedback on voice data. However, achieving these processes efficiently in a single environment is difficult and time-consuming. In particular, there is a lack of a system for customer service staff and managers to access data in real time and provide prompt feedback, which delays improvements in service quality. In addition, there is a lack of systems that use mobile devices that can be used on-site, which reduces staff work efficiency.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0294] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data to evaluate the customer service content, and a visualization means for visualizing the evaluated data. This enables customer service staff and managers to evaluate the quality of customer service operations in real time and provide quick feedback using smartphones.

[0295] The "voice collection means" is a device or method for collecting voices during customer service operations.

[0296] A "transcription means" is a device or method for converting collected audio data into text data.

[0297] The "analysis means" is a device or method for analyzing the converted character data and evaluating the customer service content.

[0298] A "visualization means" is a device or method for visually displaying evaluated data.

[0299] The "providing means" is a device or method for providing visualized data to a user.

[0300] "Mobile terminal means" refers to a device or method for collecting voice and displaying analysis results using a mobile terminal such as a smartphone.

[0301] The present invention is a system that collects voice data during customer service operations, analyzes the data, and evaluates and visualizes the customer service content. The system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a mobile terminal means.

[0302] System configuration and processing flow

[0303] Audio Collection

[0304] When a customer service staff member begins serving a customer, the server starts collecting voice data in real time. Using a mobile terminal such as a smartphone, the terminal records the voice and sends the data to the server. For example, when a store staff member says to a customer, "Welcome, what are you looking for today?", this voice is collected in real time.

[0305] Transcription

[0306] The collected voice data is converted into text data using a transcription method on the server. For example, the voice "Welcome, what are you looking for today?" is converted into text data "Welcome, what are you looking for today?". This process uses Python and the SpeechRecognition library.

[0307] Data analysis

[0308] The transcribed text data is then analyzed by an analysis tool on the server. During this process, the Transformers library is used as a natural language processing (NLP) engine to evaluate the customer service content and customer sentiment from the text data. For example, "Welcome" is recognized as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Furthermore, sentiment analysis is used to evaluate whether the customer is satisfied.

[0309] visualization

[0310] The analyzed data is visually displayed by the server using visualization means. For this purpose, the data is stored in a database and displayed as graphs and charts on a dashboard or similar. For example, it allows users to see at a glance whether a particular staff member is achieving high customer satisfaction.

[0311] Providing Feedback

[0312] The evaluated data is provided to users (managers and staff) using a delivery method. Mobile devices using smartphones enable immediate feedback. For example, a manager can check the dashboard and provide timely, specific advice to staff, such as "This phrase will increase customer satisfaction."

[0313] Specific examples

[0314] A specific example is a scene in a store where Staff A greets Customer B, saying, "Welcome, what are you looking for today?" This speech is collected by a mobile device (smartphone) and sent to a server. The server converts this speech into text data and analyzes it using a natural language processing engine. As a result of the analysis, "Welcome" is classified as a greeting, and "What are you looking for?" is extracted as confirmation of the customer's needs. Emotional data on whether the customer is satisfied or not can also be obtained from the customer's speech response. These results are stored in a database and visualized on a dashboard. Managers can refer to this and use it to provide customer service training to Staff A.

[0315] Prompt Sentence Examples

[0316] Analyze this conversation and evaluate the effectiveness of your customer service:

[0317] "Welcome, what are you looking for today?"

[0318] The analysis should include elements related to "greetings," "confirming customer needs," and "customer satisfaction."

[0319] This system makes it possible to improve the quality of customer service and enhance staff skills.

[0320] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0321] Step 1:

[0322] When a customer service staff member starts interacting with a customer, the server uses a voice collection means to collect voice data in real time. Specifically, the conversation between the customer service staff member and the customer is recorded from a terminal (smartphone) and the voice data is sent to the server. The input is the voice spoken during the interaction, and the output is the voice data sent to the server.

[0323] Step 2:

[0324] The server converts the audio data obtained from the audio collection means into text data using a transcription means. Specifically, it converts the audio data into text data using the SpeechRecognition library. The input is audio data, and the output is text data.

[0325] Step 3:

[0326] The server performs natural language processing (NLP) on the transcribed text data using analytical methods. Specifically, it analyzes the text data using the sentiment-analysis pipeline of the Transformers library and evaluates the customer service content and emotions. The input is text data, and the output is evaluation data as the analysis result.

[0327] Step 4:

[0328] The server visually displays the evaluation data obtained by the analysis means using visualization means. Specifically, the evaluation data is stored in a database and displayed in graph or chart format on a dashboard. The input is the evaluation data of the analysis results, and the output is visualized data in the form of graphs or charts.

[0329] Step 5:

[0330] The user (manager) checks the visualized data using the means provided. Specifically, they view the dashboard using a mobile device such as a smartphone and evaluate the customer service performance of the staff. The input is the visualized data on the dashboard, and the output is the feedback provided by the user.

[0331] Step 6:

[0332] Users can provide specific guidance to staff through feedback. For example, they can give advice such as "This phrase will increase customer satisfaction" based on dashboard data. This is expected to improve the staff's customer service skills. The input is the feedback content generated by the user, and the output is specific customer service improvement measures that the staff will put into practice.

[0333] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0335] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0336] [Second embodiment]

[0337] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0338] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0339] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0340] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0341] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0343] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0344] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0345] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0346] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0347] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0348] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0349] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, and a provision means.

[0350] System configuration and processing flow

[0351] 1. Collection of audio data

[0352] Terminal (AI microphone): When a staff member serving a customer begins serving them, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0353] 2. Transcription of audio data

[0354] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0355] 3. Data Analysis

[0356] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data and performs sentiment analysis and intent detection. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs.

[0357] 4. Performance visualization

[0358] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0359] 5. Feedback

[0360] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0361] Specific examples

[0362] Imagine a scene in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This voice is collected by a device (AI microphone) and sent to a server. The server transcribes the received voice data and analyzes the text using an analysis method (NLP engine). As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0363] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0364] The processing flow will be explained below.

[0365] Step 1:

[0366] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0367] Step 2:

[0368] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0369] Step 3:

[0370] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0371] Step 4:

[0372] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0373] Step 5:

[0374] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0375] Step 6:

[0376] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0377] Step 7:

[0378] The server stores the analysis results obtained from the NLP engine in a database, including extracted keywords, sentiment scores, and category classifications.

[0379] Step 8:

[0380] The server generates a dashboard based on the stored data, which visually displays the analysis results, such as graphs and charts showing high customer satisfaction for a particular staff member.

[0381] Step 9:

[0382] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0383] Step 10:

[0384] Users can provide feedback based on the dashboard data, specifically sharing the customer service methods of high performers with other staff members and providing advice on how to improve overall service quality.

[0385] Through this series of processes, customer service data is collected, analyzed, and visualized, enabling specific feedback and service improvements.

[0386] Example 1

[0387] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0388] Modern customer service operations require accurate and efficient methods for evaluating employee service quality and customer satisfaction. However, traditional evaluation methods are subjective and time-consuming, resulting in issues with accuracy and efficiency. Furthermore, analysis of customer service content and feedback of evaluation results are not centrally managed, making it difficult to provide appropriate information to improve overall service quality.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0390] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization tool for visualizing the analyzed data, a provision means for providing the visualized data to a user, and a means for saving the analysis results in a database and displaying them on a dashboard accessible to a manager. This makes it possible to objectively and efficiently evaluate the quality of customer service operations and visually provide the analysis results to a manager, thereby improving employee performance and overall service quality.

[0391] The "voice collection means" is a means for collecting voice data in real time during customer service operations and transmitting the data to a server.

[0392] The "transcription means" is a means for converting collected voice data into text data. Specifically, it has a function of converting voice data into text using a transcription engine.

[0393] The "analysis means" is a means for analyzing the converted text data using a natural language processing engine, for example, to extract keywords, analyze sentiment, and detect intent.

[0394] A "visualization tool" is a tool for visually displaying the evaluation data obtained by the analysis means. Specifically, it is a tool that has the function of retrieving data from a database and displaying it in the form of a graph, chart, etc.

[0395] The "provision means" is a means for providing visualized data to a user. For example, it has a function of presenting the visualized data in a form accessible to an administrator via a dashboard.

[0396] A "database" is a device or system that stores information to store analyzed results and make them available for retrieval as needed.

[0397] A "dashboard" is an interface for visually displaying analysis results and evaluation data in a form accessible to administrators.

[0398] This invention relates to a system that collects voices during customer service operations, analyzes the data, evaluates the customer service, and provides the results to users. This system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization tool, and a presentation means.

[0399] System configuration

[0400] Audio collection method

[0401] A device (AI microphone) is used to collect voices during customer service. The device records conversations with customers in real time, compresses the data, and sends it to a server. For example, when a restaurant staff member says, "Welcome, what would you like to order today?", this voice data is collected.

[0402] Transcription methods

[0403] The server receives the voice data sent from the device. The received voice data is sent to a transcription engine such as Google Cloud Speech-to-Text and converted into text data. For example, the voice data "Welcome, what are you looking for today?" is converted into text data.

[0404] Analysis means

[0405] The server then sends the converted text data to a natural language processing (NLP) engine, such as the Google Cloud Natural Language API, for analysis. The analysis extracts keywords from the text data and performs sentiment analysis and intent detection. For example, it recognizes the phrase "Welcome" as a greeting and classifies "What are you looking for?" as a customer need confirmation.

[0406] Visualization Tools

[0407] The server stores the analyzed results in a database. The server then uses a visualization tool (e.g., Tableau or Google Data Studio) to display this data on a dashboard. The dashboard visually displays data on which specific staff members have achieved high customer satisfaction using graphs and charts.

[0408] Providing means

[0409] Users (managers and staff) can access the dashboard and evaluate staff performance based on visualized data. By providing specific feedback to staff, they can improve the quality of customer service. For example, they can point out that a particular greeting phrase leads to high customer satisfaction and encourage other staff to use the same greeting.

[0410] Specific examples

[0411] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" The audio of this exchange is collected by a device (AI microphone) and sent to a server. The server converts the received audio into text data using a transcription engine, and analyzes the text using an NLP engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, making it clear that Staff A's customer service is of high quality. The manager can use this data as a reference to provide feedback to other staff members, encouraging them to provide similar customer service.

[0412] Prompt Sentence Examples

[0413] Here are some examples of prompts for generative AI models:

[0414] Please model a customer service scenario in a clothing store. Assuming a scenario in which staff member A says to customer B, "Welcome, is there anything you're looking for today?", explain the process from collecting voice data to analyzing, visualizing, and providing feedback.

[0415] This allows the system to objectively and efficiently evaluate the quality of customer service operations and provide users with visual analysis results, thereby improving employee performance and overall service quality.

[0416] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0417] System program processing flow

[0418] Step 1: Collecting audio data

[0419] Terminal (AI microphone): When the terminal detects a user (staff member) starting customer service, it automatically starts collecting voice data.

[0420] Detailed operation: The terminal uses a microphone to record voice in real time and compresses the recorded voice data. For example, when a staff member says, "Welcome, what would you like to buy today?", this voice data is collected.

[0421] Input: Voice data (user speech)

[0422] Output: Compressed audio data (for transfer)

[0423] Step 2: Sending audio data

[0424] Terminal (AI microphone): The terminal sends compressed audio data to the server.

[0425] Detailed operation: The device uploads the collected and compressed audio data to a server via the Internet.

[0426] Input: Compressed audio data

[0427] Output: Audio data sent to the server

[0428] Step 3: Transcription

[0429] Server: The server receives the voice data sent from the terminal.

[0430] How it works in detail: The server sends the audio data to a transcription engine such as Google Cloud Speech-to-Text, which converts it into text data.

[0431] Input: Audio data

[0432] Output: Text data (speech-to-text conversion)

[0433] Step 4: Data analysis

[0434] Server: The server sends the transcribed text data to a natural language processing (NLP) engine.

[0435] How it works in detail: The server uses an NLP engine such as Google Cloud Natural Language API to perform keyword extraction, sentiment analysis, and intent detection from text data.

[0436] Input: Text data

[0437] Output: Analysis results (keywords, sentiment, intent)

[0438] Step 5: Save your data

[0439] Server: The server stores the analysis results in a database.

[0440] How it works in detail: The server records the analysis results in a database such as MySQL or PostgreSQL, making the data available for retrieval when needed.

[0441] Input: Analysis results

[0442] Output: Data stored in the database

[0443] Step 6: Visualize in a dashboard

[0444] Server: The server uses visualization tools to display the analysis results on a dashboard.

[0445] How it works in detail: The server uses visualization tools like Tableau or Google Data Studio to extract the necessary data from the database, convert it into graphs and charts, and display it on a dashboard.

[0446] Input: Data stored in a database

[0447] Output: Visualized data on a dashboard

[0448] Step 7: Feedback

[0449] User (Manager): The user accesses the dashboard and evaluates staff performance based on visualized data.

[0450] Detailed Action: The manager reviews the data displayed on the dashboard and provides specific feedback to staff, for example, "A specific greeting phrase leads to high satisfaction."

[0451] Input: Data visualized on the dashboard

[0452] Output: Feedback (instructions for staff)

[0453] This series of processes allows the quality of customer service to be objectively and efficiently evaluated, and the analysis results are presented to the user visually, making it possible to improve employee performance and overall service quality.

[0454] (Application example 1)

[0455] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0456] Conventional customer service performance evaluation systems have the problem that it is difficult to analyze customer service content in real time and provide immediate feedback to staff. In addition, there are insufficient means to visually display the evaluation data, making it difficult to effectively support staff performance improvement.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0458] In this invention, the server includes a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a means for visualizing data collected during customer service operations and having a display device that allows the user to check the data. This enables real-time analysis and visual display of customer service content, provides immediate feedback to staff, and enables improvement of customer service performance.

[0459] "Speech collection means" refers to a device or technology that collects speech generated during customer service operations.

[0460] A "transcription means" is a device or technology that converts collected audio data into text data.

[0461] The "analysis means" refers to a device or technology that analyzes the converted character data and evaluates the customer service content.

[0462] "Visualization means" refers to a device or technology that visually displays the evaluated data.

[0463] "Providing means" refers to the device or technology that provides visualized data to users.

[0464] A "display device" is a device that visualizes data collected during customer service operations and allows users to check it.

[0465] To implement the present invention, the following system configuration and processing flow are used.

[0466] System Configuration

[0467] The system includes the following hardware and software:

[0468] Hardware:

[0469] Audio collection means (e.g., microphone in smart glasses)

[0470] Server (e.g. cloud server)

[0471] Display devices (e.g., smart glasses displays or external monitors)

[0472] software:

[0473] Audio collection library (e.g., Google Cloud Speech-to-Text API)

[0474] Natural language processing engines (e.g., spaCy, Google Cloud Natural Language API)

[0475] Database (e.g. Firebase Realtime Database)

[0476] Data visualization tools (e.g., Google Data Studio)

[0477] Program processing

[0478] The server receives real-time voice data from voice collection devices such as smart glasses during customer service. The voice data is converted into text data using the Google Cloud Speech-to-Text API. The converted text data is then analyzed using the Google Cloud Natural Language API. This analysis involves keyword extraction, sentiment analysis, and intent detection. The analysis results are stored in the Firebase Realtime Database and then visually displayed using Google Data Studio.

[0479] Explanations and examples

[0480] For example, imagine a scenario where a staff member at a clothing store greets a customer through smart glasses, saying, "Welcome, what are you looking for today?" This speech is collected by the smart glasses' microphone and sent to a cloud server. The server converts the speech into text data using the Google Cloud Speech-to-Text API, and then analyzes the data using the Google Cloud Natural Language API. Through this analysis, the phrase "Welcome" is recognized as a greeting, and "What are you looking for today?" is classified as a confirmation of the customer's needs. The analysis results are stored in the Firebase Realtime Database and visualized using Google Data Studio.

[0481] Staff and managers can view the analysis results in real time on the smart glasses display or an external monitor, and receive immediate feedback, which can help improve customer service performance.

[0482] Prompt Sentence Examples

[0483] By inputting the prompt below into the generative AI model, you can check the application overview and processing flow.

[0484] "Based on the following description, please provide a detailed description of an application that analyzes customer service performance in a brick-and-mortar store in real time and provides feedback through smart glasses."

[0485] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0486] Step 1:

[0487] Audio collection method

[0488] The device (the microphone in the smart glasses) collects voice data in real time during customer service. Voice is captured from the environment and stored as a digital audio file within the device. This becomes input data. The device then transmits the collected voice data to a server.

[0489] Step 2:

[0490] Sending audio data

[0491] The collected voice data is sent from the device to the server in real time. Specifically, the voice data is uploaded to a cloud server via a mobile communication network. The input is the voice data collected by the device, and the server receives this data.

[0492] Step 3:

[0493] Transcription of audio data

[0494] The server sends the received voice data to the Google Cloud Speech-to-Text API, which converts it into text data. This API performs speech analysis and outputs text data. The input is voice data, and the output is the corresponding text data.

[0495] Step 4:

[0496] Parsing character data

[0497] The server uses the Google Cloud Natural Language API to analyze text data. Specifically, it uses this API to extract keywords, analyze sentiment, and detect intent from text. The input is text data, and the output is the analysis results (keyword list, sentiment score, and intent label).

[0498] Step 5:

[0499] Saving analysis data

[0500] The server stores the analysis results in the Firebase Realtime Database. The input is the analysis data, and the output is the state stored in the database. This data is later used for visualization and feedback.

[0501] Step 6:

[0502] Data Visualization

[0503] After the analysis results are saved, the server visualizes these data using Google Data Studio. The visualization tool generates a dashboard that displays the analysis results in the form of graphs and charts. The input is the saved analysis data, and the output is the visually displayed information.

[0504] Step 7:

[0505] Providing Feedback

[0506] Users (staff or managers) can access visualized data through the smart glasses display or an external monitor and receive real-time feedback. The input is visual data, and the output is feedback information confirmed by the user. This helps users to immediately improve their customer service performance.

[0507] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0508] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, a presentation means, and an emotion engine.

[0509] System configuration and processing flow

[0510] 1. Collection of audio data

[0511] Terminal (AI microphone): When a wait staff member begins serving a customer, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0512] 2. Transcription of audio data

[0513] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0514] 3. Data Analysis

[0515] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data, and the emotion engine recognizes the user's emotions. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs. In addition, the emotion engine extracts emotional data on whether the customer is satisfied based on the customer's voice response.

[0516] 4. Performance visualization

[0517] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0518] 5. Feedback

[0519] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0520] Specific examples

[0521] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using an analysis method (NLP engine) and an emotion engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and choice of words. These results are stored in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0522] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality. In addition, by incorporating customer sentiment data, more detailed and specific feedback is possible, helping to improve staff customer service skills.

[0523] The processing flow will be explained below.

[0524] Step 1:

[0525] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0526] Step 2:

[0527] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0528] Step 3:

[0529] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0530] Step 4:

[0531] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0532] Step 5:

[0533] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0534] Step 6:

[0535] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0536] Step 7:

[0537] The server passes the analysis results to the emotion engine, which recognizes the customer's emotions (such as "satisfaction," "dissatisfaction," or "happiness") from the customer's tone of voice and the way they use words.

[0538] Step 8:

[0539] The server integrates the emotion data from the emotion engine with the analysis results from the NLP engine, including extracted keywords, customer emotion scores, and emotion categories.

[0540] Step 9:

[0541] The server stores the consolidated data in a database, which includes analytical data and emotion data corresponding to each customer interaction event.

[0542] Step 10:

[0543] The server generates a dashboard based on the stored data. The dashboard visually displays each staff member's performance and customer sentiment data. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction.

[0544] Step 11:

[0545] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0546] Step 12:

[0547] Users can provide feedback based on the dashboard data, specifically offering specific advice to other staff members based on the customer service methods of high performers and customer sentiment data.

[0548] This process allows for the collection, analysis, and visualization of customer service data, enabling specific feedback and service improvement. Customer sentiment data is also taken into account, enabling more detailed and accurate evaluations.

[0549] Example 2

[0550] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] Effective evaluation and feedback of staff performance in customer service operations is a challenge. In particular, efficient collection and accurate analysis of voice data during customer interactions is essential to improve staff customer service skills. Furthermore, integrating real-time data collection and emotion data analysis is essential to provide more detailed and specific feedback and improve overall service quality.

[0552] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0553] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization means for visualizing the evaluated data, an emotion engine means for analyzing emotion data, and a provision means for providing the visualized data to the user. This makes it possible to collect voice data during customer service operations in real time and provide highly accurate feedback based on the analysis results.

[0554] The "voice collection means" is a device or system for collecting and recording voices during customer service operations in real time.

[0555] A "transcription means" is software or hardware for converting collected audio data into text data.

[0556] "Analysis means" refers to natural language processing engines and other analysis tools that use text data to evaluate customer service content.

[0557] A "visualization means" is a system that includes dashboards and graph generation tools for visually displaying the evaluated data.

[0558] The "emotion engine means" is software or hardware for analyzing emotions from voice data or text data and recognizing the user's emotional state.

[0559] "Providing means" refers to the interface or system for providing visualized data to users.

[0560] The present invention is a system that collects voices during customer service operations, analyzes the data, evaluates the service content and customer emotions, and visualizes them. It mainly includes a voice collection means, a transcription means, an analysis means, a visualization means, an emotion engine means, and a provision means.

[0561] Audio collection method

[0562] The device (e.g., an AI microphone) automatically starts collecting voice data when a waiter begins serving a customer. The device records the conversation in real time and sends the collected voice data to a server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0563] Transcription methods

[0564] The server receives the voice data sent from the device and converts the voice into text data using a transcription engine (e.g., Google Cloud Speech-to-Text API). For example, the collected voice "Welcome, what are you looking for today?" is converted into text data.

[0565] Analysis means

[0566] The server analyzes the transcribed text data using a natural language processing (NLP) engine (e.g., spaCy) and an emotion engine (e.g., IBM Watson Tone Analyzer). Through this analysis, keywords are extracted from the text data and the customer service content is classified. Emotional data is also extracted from the customer's voice response. For example, "Welcome" is classified as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Emotional data such as "satisfied" or "dissatisfied" is also extracted from the customer's tone of voice and choice of words.

[0567] Visualization means

[0568] The server stores the analyzed data in a database and displays it visually on a dashboard. This dashboard shows the customer satisfaction achieved by a specific staff member in the form of graphs and charts. For example, it becomes possible to see at a glance that a specific staff member has achieved high customer satisfaction. This dashboard can be accessed by administrators.

[0569] Providing means

[0570] Users (managers or staff) can analyze staff performance based on the visualized data and provide specific feedback. For example, they can refer to the dashboard evaluation data and point out that a particular greeting phrase leads to high customer satisfaction, and recommend similar customer service to other staff members.

[0571] Specific examples

[0572] Imagine a scenario in a clothing store where Staff A greets Customer B with, "Welcome, are you looking for anything today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using analytical methods (NLP engine and emotion engine). As a result, "Welcome" is classified as a greeting, and "Are you looking for anything?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and phrasing. These results are stored in a database and displayed on a dashboard, visualizing the quality of Staff A's customer service. Managers can use this data to provide specific feedback to other staff members and work to improve overall customer service skills.

[0573] Prompt Sentence Examples

[0574] Enter the following prompt into the generative AI model:

[0575] "Please explain the specific process flow of a system that analyzes voice data during customer service operations, visualizes staff performance, and provides feedback. Please provide a detailed description of the specific operations at each step."

[0576] In this way, this system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0577] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0578] Step 1:

[0579] The device (AI microphone) automatically starts collecting voice data when a waiter starts interacting with a customer. Specifically, the device records the conversation in real time and sends the voice data to a server.

[0580] Input: Audio of conversation between wait staff and customer.

[0581] Output: The audio data sent to the server.

[0582] How it works: The multi-function AI microphone starts collecting voice data the moment a waiter says, "Welcome, what would you like to order today?" The collected voice data is then compressed and sent to a server in real time via Wi-Fi or Bluetooth.

[0583] Step 2:

[0584] The server receives the voice data sent from the terminal and converts the voice data into text data using a transcription engine.

[0585] Input: Audio data sent from the device.

[0586] Output: Character data.

[0587] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the received audio, "Welcome, what are you looking for today?", into text data in real time.

[0588] Step 3:

[0589] The server analyzes the transcribed text data using a natural language processing (NLP) engine and an emotion engine.

[0590] Input: Character data.

[0591] Output: Analysis results (keywords and sentiment data).

[0592] Specific operation: The server uses spaCy to classify text data, such as "Welcome" as a greeting and "What are you looking for?" as confirmation of customer needs. At the same time, it uses IBM Watson Tone Analyzer to extract emotional data such as "satisfied" or "dissatisfied" from the customer's voice response.

[0593] Step 4:

[0594] The server stores the analyzed results in a database and displays them visually on a dashboard.

[0595] Input: Analysis results (keywords and sentiment data).

[0596] Output: Visualized data (graphs and charts).

[0597] What it does: The server stores the analysis results in a database and uses dashboard software such as Tableau to display the staff member customer satisfaction in the form of graphs and charts. This dashboard can be accessed by managers, allowing them to see at a glance the performance of specific staff members.

[0598] Step 5:

[0599] Users (managers, staff) analyze performance based on the visualized data and provide specific feedback.

[0600] Input: Visualized data (graphs and charts).

[0601] Output: Feedback (specific improvements and enhancements).

[0602] Specific Action: The manager opens the dashboard and checks the customer satisfaction data for a specific staff member. Then, based on the dashboard evaluation data, the manager points out that "a specific greeting phrase leads to high satisfaction" and provides feedback recommending similar customer service to other staff members. The manager also plans regular training sessions to improve overall customer service skills.

[0603] (Application example 2)

[0604] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Efficient evaluation and improvement of customer service operations requires a system that comprehensively covers the entire process of collecting, converting, analyzing, visualizing, and providing feedback on voice data. However, achieving these processes efficiently in a single environment is difficult and time-consuming. In particular, there is a lack of a system for customer service staff and managers to access data in real time and provide prompt feedback, which delays improvements in service quality. In addition, there is a lack of systems that use mobile devices that can be used on-site, which reduces staff work efficiency.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0607] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data to evaluate the customer service content, and a visualization means for visualizing the evaluated data. This enables customer service staff and managers to evaluate the quality of customer service operations in real time and provide quick feedback using smartphones.

[0608] The "voice collection means" is a device or method for collecting voices during customer service operations.

[0609] A "transcription means" is a device or method for converting collected audio data into text data.

[0610] The "analysis means" is a device or method for analyzing the converted character data and evaluating the customer service content.

[0611] A "visualization means" is a device or method for visually displaying evaluated data.

[0612] The "providing means" is a device or method for providing visualized data to a user.

[0613] "Mobile terminal means" refers to a device or method for collecting voice and displaying analysis results using a mobile terminal such as a smartphone.

[0614] The present invention is a system that collects voice data during customer service operations, analyzes the data, and evaluates and visualizes the customer service content. The system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a mobile terminal means.

[0615] System configuration and processing flow

[0616] Audio Collection

[0617] When a customer service staff member begins serving a customer, the server starts collecting voice data in real time. Using a mobile terminal such as a smartphone, the terminal records the voice and sends the data to the server. For example, when a store staff member says to a customer, "Welcome, what are you looking for today?", this voice is collected in real time.

[0618] Transcription

[0619] The collected voice data is converted into text data using a transcription method on the server. For example, the voice "Welcome, what are you looking for today?" is converted into text data "Welcome, what are you looking for today?". This process uses Python and the SpeechRecognition library.

[0620] Data analysis

[0621] The transcribed text data is then analyzed by an analysis tool on the server. During this process, the Transformers library is used as a natural language processing (NLP) engine to evaluate the customer service content and customer sentiment from the text data. For example, "Welcome" is recognized as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Furthermore, sentiment analysis is used to evaluate whether the customer is satisfied.

[0622] visualization

[0623] The analyzed data is visually displayed by the server using visualization means. For this purpose, the data is stored in a database and displayed as graphs and charts on a dashboard or similar. For example, it allows users to see at a glance whether a particular staff member is achieving high customer satisfaction.

[0624] Providing Feedback

[0625] The evaluated data is provided to users (managers and staff) using a delivery method. Mobile devices using smartphones enable immediate feedback. For example, a manager can check the dashboard and provide timely, specific advice to staff, such as "This phrase will increase customer satisfaction."

[0626] Specific examples

[0627] A specific example is a scene in a store where Staff A greets Customer B, saying, "Welcome, what are you looking for today?" This speech is collected by a mobile device (smartphone) and sent to a server. The server converts this speech into text data and analyzes it using a natural language processing engine. As a result of the analysis, "Welcome" is classified as a greeting, and "What are you looking for?" is extracted as confirmation of the customer's needs. Emotional data on whether the customer is satisfied or not can also be obtained from the customer's speech response. These results are stored in a database and visualized on a dashboard. Managers can refer to this and use it to provide customer service training to Staff A.

[0628] Prompt Sentence Examples

[0629] Analyze this conversation and evaluate the effectiveness of your customer service:

[0630] "Welcome, what are you looking for today?"

[0631] The analysis should include elements related to "greetings," "confirming customer needs," and "customer satisfaction."

[0632] This system makes it possible to improve the quality of customer service and enhance staff skills.

[0633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0634] Step 1:

[0635] When a customer service staff member starts interacting with a customer, the server uses a voice collection means to collect voice data in real time. Specifically, the conversation between the customer service staff member and the customer is recorded from a terminal (smartphone) and the voice data is sent to the server. The input is the voice spoken during the interaction, and the output is the voice data sent to the server.

[0636] Step 2:

[0637] The server converts the audio data obtained from the audio collection means into text data using a transcription means. Specifically, it converts the audio data into text data using the SpeechRecognition library. The input is audio data, and the output is text data.

[0638] Step 3:

[0639] The server performs natural language processing (NLP) on the transcribed text data using analytical methods. Specifically, it analyzes the text data using the sentiment-analysis pipeline of the Transformers library and evaluates the customer service content and emotions. The input is text data, and the output is evaluation data as the analysis result.

[0640] Step 4:

[0641] The server visually displays the evaluation data obtained by the analysis means using visualization means. Specifically, the evaluation data is stored in a database and displayed in graph or chart format on a dashboard. The input is the evaluation data of the analysis results, and the output is visualized data in the form of graphs or charts.

[0642] Step 5:

[0643] The user (manager) checks the visualized data using the means provided. Specifically, they view the dashboard using a mobile device such as a smartphone and evaluate the customer service performance of the staff. The input is the visualized data on the dashboard, and the output is the feedback provided by the user.

[0644] Step 6:

[0645] Users can provide specific guidance to staff through feedback. For example, they can give advice such as "This phrase will increase customer satisfaction" based on dashboard data. This is expected to improve the staff's customer service skills. The input is the feedback content generated by the user, and the output is specific customer service improvement measures that the staff will put into practice.

[0646] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0647] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0648] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0649] [Third embodiment]

[0650] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0651] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0652] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0653] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0654] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0655] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0656] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0657] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0658] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0659] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0660] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0661] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0662] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, and a provision means.

[0663] System configuration and processing flow

[0664] 1. Collection of audio data

[0665] Terminal (AI microphone): When a staff member serving a customer begins serving them, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0666] 2. Transcription of audio data

[0667] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0668] 3. Data Analysis

[0669] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data and performs sentiment analysis and intent detection. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs.

[0670] 4. Performance visualization

[0671] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0672] 5. Feedback

[0673] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0674] Specific examples

[0675] Imagine a scene in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This voice is collected by a device (AI microphone) and sent to a server. The server transcribes the received voice data and analyzes the text using an analysis method (NLP engine). As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0676] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0677] The processing flow will be explained below.

[0678] Step 1:

[0679] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0680] Step 2:

[0681] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0682] Step 3:

[0683] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0684] Step 4:

[0685] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0686] Step 5:

[0687] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0688] Step 6:

[0689] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0690] Step 7:

[0691] The server stores the analysis results obtained from the NLP engine in a database, including extracted keywords, sentiment scores, and category classifications.

[0692] Step 8:

[0693] The server generates a dashboard based on the stored data, which visually displays the analysis results, such as graphs and charts showing high customer satisfaction for a particular staff member.

[0694] Step 9:

[0695] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0696] Step 10:

[0697] Users can provide feedback based on the dashboard data, specifically sharing the customer service methods of high performers with other staff members and providing advice on how to improve overall service quality.

[0698] Through this series of processes, customer service data is collected, analyzed, and visualized, enabling specific feedback and service improvements.

[0699] Example 1

[0700] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0701] Modern customer service operations require accurate and efficient methods for evaluating employee service quality and customer satisfaction. However, traditional evaluation methods are subjective and time-consuming, resulting in issues with accuracy and efficiency. Furthermore, analysis of customer service content and feedback of evaluation results are not centrally managed, making it difficult to provide appropriate information to improve overall service quality.

[0702] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0703] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization tool for visualizing the analyzed data, a provision means for providing the visualized data to a user, and a means for saving the analysis results in a database and displaying them on a dashboard accessible to a manager. This makes it possible to objectively and efficiently evaluate the quality of customer service operations and visually provide the analysis results to a manager, thereby improving employee performance and overall service quality.

[0704] The "voice collection means" is a means for collecting voice data in real time during customer service operations and transmitting the data to a server.

[0705] The "transcription means" is a means for converting collected voice data into text data. Specifically, it has a function of converting voice data into text using a transcription engine.

[0706] The "analysis means" is a means for analyzing the converted text data using a natural language processing engine, for example, to extract keywords, analyze sentiment, and detect intent.

[0707] A "visualization tool" is a tool for visually displaying the evaluation data obtained by the analysis means. Specifically, it is a tool that has the function of retrieving data from a database and displaying it in the form of a graph, chart, etc.

[0708] The "provision means" is a means for providing visualized data to a user. For example, it has a function of presenting the visualized data in a form accessible to an administrator via a dashboard.

[0709] A "database" is a device or system that stores information to store analyzed results and make them available for retrieval as needed.

[0710] A "dashboard" is an interface for visually displaying analysis results and evaluation data in a form accessible to administrators.

[0711] This invention relates to a system that collects voices during customer service operations, analyzes the data, evaluates the customer service, and provides the results to users. This system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization tool, and a presentation means.

[0712] System configuration

[0713] Audio collection method

[0714] A device (AI microphone) is used to collect voices during customer service. The device records conversations with customers in real time, compresses the data, and sends it to a server. For example, when a restaurant staff member says, "Welcome, what would you like to order today?", this voice data is collected.

[0715] Transcription methods

[0716] The server receives the voice data sent from the device. The received voice data is sent to a transcription engine such as Google Cloud Speech-to-Text and converted into text data. For example, the voice data "Welcome, what are you looking for today?" is converted into text data.

[0717] Analysis means

[0718] The server then sends the converted text data to a natural language processing (NLP) engine, such as the Google Cloud Natural Language API, for analysis. The analysis extracts keywords from the text data and performs sentiment analysis and intent detection. For example, it recognizes the phrase "Welcome" as a greeting and classifies "What are you looking for?" as a customer need confirmation.

[0719] Visualization Tools

[0720] The server stores the analyzed results in a database. The server then uses a visualization tool (e.g., Tableau or Google Data Studio) to display this data on a dashboard. The dashboard visually displays data on which specific staff members have achieved high customer satisfaction using graphs and charts.

[0721] Providing means

[0722] Users (managers and staff) can access the dashboard and evaluate staff performance based on visualized data. By providing specific feedback to staff, they can improve the quality of customer service. For example, they can point out that a particular greeting phrase leads to high customer satisfaction and encourage other staff to use the same greeting.

[0723] Specific examples

[0724] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" The audio of this exchange is collected by a device (AI microphone) and sent to a server. The server converts the received audio into text data using a transcription engine, and analyzes the text using an NLP engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, making it clear that Staff A's customer service is of high quality. The manager can use this data as a reference to provide feedback to other staff members, encouraging them to provide similar customer service.

[0725] Prompt Sentence Examples

[0726] Here are some examples of prompts for generative AI models:

[0727] Please model a customer service scenario in a clothing store. Assuming a scenario in which staff member A says to customer B, "Welcome, is there anything you're looking for today?", explain the process from collecting voice data to analyzing, visualizing, and providing feedback.

[0728] This allows the system to objectively and efficiently evaluate the quality of customer service operations and provide users with visual analysis results, thereby improving employee performance and overall service quality.

[0729] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0730] System program processing flow

[0731] Step 1: Collecting audio data

[0732] Terminal (AI microphone): When the terminal detects a user (staff member) starting customer service, it automatically starts collecting voice data.

[0733] Detailed operation: The terminal uses a microphone to record voice in real time and compresses the recorded voice data. For example, when a staff member says, "Welcome, what would you like to buy today?", this voice data is collected.

[0734] Input: Voice data (user speech)

[0735] Output: Compressed audio data (for transfer)

[0736] Step 2: Sending audio data

[0737] Terminal (AI microphone): The terminal sends compressed audio data to the server.

[0738] Detailed operation: The device uploads the collected and compressed audio data to a server via the Internet.

[0739] Input: Compressed audio data

[0740] Output: Audio data sent to the server

[0741] Step 3: Transcription

[0742] Server: The server receives the voice data sent from the terminal.

[0743] How it works in detail: The server sends the audio data to a transcription engine such as Google Cloud Speech-to-Text, which converts it into text data.

[0744] Input: Audio data

[0745] Output: Text data (speech-to-text conversion)

[0746] Step 4: Data analysis

[0747] Server: The server sends the transcribed text data to a natural language processing (NLP) engine.

[0748] How it works in detail: The server uses an NLP engine such as Google Cloud Natural Language API to perform keyword extraction, sentiment analysis, and intent detection from text data.

[0749] Input: Text data

[0750] Output: Analysis results (keywords, sentiment, intent)

[0751] Step 5: Save your data

[0752] Server: The server stores the analysis results in a database.

[0753] How it works in detail: The server records the analysis results in a database such as MySQL or PostgreSQL, making the data available for retrieval when needed.

[0754] Input: Analysis results

[0755] Output: Data stored in the database

[0756] Step 6: Visualize in a dashboard

[0757] Server: The server uses visualization tools to display the analysis results on a dashboard.

[0758] How it works in detail: The server uses visualization tools like Tableau or Google Data Studio to extract the necessary data from the database, convert it into graphs and charts, and display it on a dashboard.

[0759] Input: Data stored in a database

[0760] Output: Visualized data on a dashboard

[0761] Step 7: Feedback

[0762] User (Manager): The user accesses the dashboard and evaluates staff performance based on visualized data.

[0763] Detailed Action: The manager reviews the data displayed on the dashboard and provides specific feedback to staff, for example, "A specific greeting phrase leads to high satisfaction."

[0764] Input: Data visualized on the dashboard

[0765] Output: Feedback (instructions for staff)

[0766] This series of processes allows the quality of customer service to be objectively and efficiently evaluated, and the analysis results are presented to the user visually, making it possible to improve employee performance and overall service quality.

[0767] (Application example 1)

[0768] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0769] Conventional customer service performance evaluation systems have the problem that it is difficult to analyze customer service content in real time and provide immediate feedback to staff. In addition, there are insufficient means to visually display the evaluation data, making it difficult to effectively support staff performance improvement.

[0770] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0771] In this invention, the server includes a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a means for visualizing data collected during customer service operations and having a display device that allows the user to check the data. This enables real-time analysis and visual display of customer service content, provides immediate feedback to staff, and enables improvement of customer service performance.

[0772] "Speech collection means" refers to a device or technology that collects speech generated during customer service operations.

[0773] A "transcription means" is a device or technology that converts collected audio data into text data.

[0774] The "analysis means" refers to a device or technology that analyzes the converted character data and evaluates the customer service content.

[0775] "Visualization means" refers to a device or technology that visually displays the evaluated data.

[0776] "Providing means" refers to the device or technology that provides visualized data to users.

[0777] A "display device" is a device that visualizes data collected during customer service operations and allows users to check it.

[0778] To implement the present invention, the following system configuration and processing flow are used.

[0779] System Configuration

[0780] The system includes the following hardware and software:

[0781] Hardware:

[0782] Audio collection means (e.g., microphone in smart glasses)

[0783] Server (e.g. cloud server)

[0784] Display devices (e.g., smart glasses displays or external monitors)

[0785] software:

[0786] Audio collection library (e.g., Google Cloud Speech-to-Text API)

[0787] Natural language processing engines (e.g., spaCy, Google Cloud Natural Language API)

[0788] Database (e.g. Firebase Realtime Database)

[0789] Data visualization tools (e.g., Google Data Studio)

[0790] Program processing

[0791] The server receives real-time voice data from voice collection devices such as smart glasses during customer service. The voice data is converted into text data using the Google Cloud Speech-to-Text API. The converted text data is then analyzed using the Google Cloud Natural Language API. This analysis involves keyword extraction, sentiment analysis, and intent detection. The analysis results are stored in the Firebase Realtime Database and then visually displayed using Google Data Studio.

[0792] Explanations and examples

[0793] For example, imagine a scenario where a staff member at a clothing store greets a customer through smart glasses, saying, "Welcome, what are you looking for today?" This speech is collected by the smart glasses' microphone and sent to a cloud server. The server converts the speech into text data using the Google Cloud Speech-to-Text API, and then analyzes the data using the Google Cloud Natural Language API. Through this analysis, the phrase "Welcome" is recognized as a greeting, and "What are you looking for today?" is classified as a confirmation of the customer's needs. The analysis results are stored in the Firebase Realtime Database and visualized using Google Data Studio.

[0794] Staff and managers can view the analysis results in real time on the smart glasses display or an external monitor, and receive immediate feedback, which can help improve customer service performance.

[0795] Prompt Sentence Examples

[0796] By inputting the prompt below into the generative AI model, you can check the application overview and processing flow.

[0797] "Based on the following description, please provide a detailed description of an application that analyzes customer service performance in a brick-and-mortar store in real time and provides feedback through smart glasses."

[0798] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0799] Step 1:

[0800] Audio collection method

[0801] The device (the microphone in the smart glasses) collects voice data in real time during customer service. Voice is captured from the environment and stored as a digital audio file within the device. This becomes input data. The device then transmits the collected voice data to a server.

[0802] Step 2:

[0803] Sending audio data

[0804] The collected voice data is sent from the device to the server in real time. Specifically, the voice data is uploaded to a cloud server via a mobile communication network. The input is the voice data collected by the device, and the server receives this data.

[0805] Step 3:

[0806] Transcription of audio data

[0807] The server sends the received voice data to the Google Cloud Speech-to-Text API, which converts it into text data. This API performs speech analysis and outputs text data. The input is voice data, and the output is the corresponding text data.

[0808] Step 4:

[0809] Parsing character data

[0810] The server uses the Google Cloud Natural Language API to analyze text data. Specifically, it uses this API to extract keywords, analyze sentiment, and detect intent from text. The input is text data, and the output is the analysis results (keyword list, sentiment score, and intent label).

[0811] Step 5:

[0812] Saving analysis data

[0813] The server stores the analysis results in the Firebase Realtime Database. The input is the analysis data, and the output is the state stored in the database. This data is later used for visualization and feedback.

[0814] Step 6:

[0815] Data Visualization

[0816] After the analysis results are saved, the server visualizes these data using Google Data Studio. The visualization tool generates a dashboard that displays the analysis results in the form of graphs and charts. The input is the saved analysis data, and the output is the visually displayed information.

[0817] Step 7:

[0818] Providing Feedback

[0819] Users (staff or managers) can access visualized data through the smart glasses display or an external monitor and receive real-time feedback. The input is visual data, and the output is feedback information confirmed by the user. This helps users to immediately improve their customer service performance.

[0820] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0821] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, a presentation means, and an emotion engine.

[0822] System configuration and processing flow

[0823] 1. Collection of audio data

[0824] Terminal (AI microphone): When a wait staff member begins serving a customer, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0825] 2. Transcription of audio data

[0826] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0827] 3. Data Analysis

[0828] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data, and the emotion engine recognizes the user's emotions. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs. In addition, the emotion engine extracts emotional data on whether the customer is satisfied based on the customer's voice response.

[0829] 4. Performance visualization

[0830] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0831] 5. Feedback

[0832] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0833] Specific examples

[0834] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using an analysis method (NLP engine) and an emotion engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and choice of words. These results are stored in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0835] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality. In addition, by incorporating customer sentiment data, more detailed and specific feedback is possible, helping to improve staff customer service skills.

[0836] The processing flow will be explained below.

[0837] Step 1:

[0838] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0839] Step 2:

[0840] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0841] Step 3:

[0842] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0843] Step 4:

[0844] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[0845] Step 5:

[0846] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[0847] Step 6:

[0848] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[0849] Step 7:

[0850] The server passes the analysis results to the emotion engine, which recognizes the customer's emotions (such as "satisfaction," "dissatisfaction," or "happiness") from the customer's tone of voice and the way they use words.

[0851] Step 8:

[0852] The server integrates the emotion data from the emotion engine with the analysis results from the NLP engine, including extracted keywords, customer emotion scores, and emotion categories.

[0853] Step 9:

[0854] The server stores the consolidated data in a database, which includes analytical data and emotion data corresponding to each customer interaction event.

[0855] Step 10:

[0856] The server generates a dashboard based on the stored data. The dashboard visually displays each staff member's performance and customer sentiment data. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction.

[0857] Step 11:

[0858] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[0859] Step 12:

[0860] Users can provide feedback based on the dashboard data, specifically offering specific advice to other staff members based on the customer service methods of high performers and customer sentiment data.

[0861] This process allows for the collection, analysis, and visualization of customer service data, enabling specific feedback and service improvement. Customer sentiment data is also taken into account, enabling more detailed and accurate evaluations.

[0862] Example 2

[0863] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0864] Effective evaluation and feedback of staff performance in customer service operations is a challenge. In particular, efficient collection and accurate analysis of voice data during customer interactions is essential to improve staff customer service skills. Furthermore, integrating real-time data collection and emotion data analysis is essential to provide more detailed and specific feedback and improve overall service quality.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0866] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization means for visualizing the evaluated data, an emotion engine means for analyzing emotion data, and a provision means for providing the visualized data to the user. This makes it possible to collect voice data during customer service operations in real time and provide highly accurate feedback based on the analysis results.

[0867] The "voice collection means" is a device or system for collecting and recording voices during customer service operations in real time.

[0868] A "transcription means" is software or hardware for converting collected audio data into text data.

[0869] "Analysis means" refers to natural language processing engines and other analysis tools that use text data to evaluate customer service content.

[0870] A "visualization means" is a system that includes dashboards and graph generation tools for visually displaying the evaluated data.

[0871] The "emotion engine means" is software or hardware for analyzing emotions from voice data or text data and recognizing the user's emotional state.

[0872] "Providing means" refers to the interface or system for providing visualized data to users.

[0873] The present invention is a system that collects voices during customer service operations, analyzes the data, evaluates the service content and customer emotions, and visualizes them. It mainly includes a voice collection means, a transcription means, an analysis means, a visualization means, an emotion engine means, and a provision means.

[0874] Audio collection method

[0875] The device (e.g., an AI microphone) automatically starts collecting voice data when a waiter begins serving a customer. The device records the conversation in real time and sends the collected voice data to a server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0876] Transcription methods

[0877] The server receives the voice data sent from the device and converts the voice into text data using a transcription engine (e.g., Google Cloud Speech-to-Text API). For example, the collected voice "Welcome, what are you looking for today?" is converted into text data.

[0878] Analysis means

[0879] The server analyzes the transcribed text data using a natural language processing (NLP) engine (e.g., spaCy) and an emotion engine (e.g., IBM Watson Tone Analyzer). Through this analysis, keywords are extracted from the text data and the customer service content is classified. Emotional data is also extracted from the customer's voice response. For example, "Welcome" is classified as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Emotional data such as "satisfied" or "dissatisfied" is also extracted from the customer's tone of voice and choice of words.

[0880] Visualization means

[0881] The server stores the analyzed data in a database and displays it visually on a dashboard. This dashboard shows the customer satisfaction achieved by a specific staff member in the form of graphs and charts. For example, it becomes possible to see at a glance that a specific staff member has achieved high customer satisfaction. This dashboard can be accessed by administrators.

[0882] Providing means

[0883] Users (managers or staff) can analyze staff performance based on the visualized data and provide specific feedback. For example, they can refer to the dashboard evaluation data and point out that a particular greeting phrase leads to high customer satisfaction, and recommend similar customer service to other staff members.

[0884] Specific examples

[0885] Imagine a scenario in a clothing store where Staff A greets Customer B with, "Welcome, are you looking for anything today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using analytical methods (NLP engine and emotion engine). As a result, "Welcome" is classified as a greeting, and "Are you looking for anything?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and phrasing. These results are stored in a database and displayed on a dashboard, visualizing the quality of Staff A's customer service. Managers can use this data to provide specific feedback to other staff members and work to improve overall customer service skills.

[0886] Prompt Sentence Examples

[0887] Enter the following prompt into the generative AI model:

[0888] "Please explain the specific process flow of a system that analyzes voice data during customer service operations, visualizes staff performance, and provides feedback. Please provide a detailed description of the specific operations at each step."

[0889] In this way, this system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0890] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0891] Step 1:

[0892] The device (AI microphone) automatically starts collecting voice data when a waiter starts interacting with a customer. Specifically, the device records the conversation in real time and sends the voice data to a server.

[0893] Input: Audio of conversation between wait staff and customer.

[0894] Output: The audio data sent to the server.

[0895] How it works: The multi-function AI microphone starts collecting voice data the moment a waiter says, "Welcome, what would you like to order today?" The collected voice data is then compressed and sent to a server in real time via Wi-Fi or Bluetooth.

[0896] Step 2:

[0897] The server receives the voice data sent from the terminal and converts the voice data into text data using a transcription engine.

[0898] Input: Audio data sent from the device.

[0899] Output: Character data.

[0900] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the received audio, "Welcome, what are you looking for today?", into text data in real time.

[0901] Step 3:

[0902] The server analyzes the transcribed text data using a natural language processing (NLP) engine and an emotion engine.

[0903] Input: Character data.

[0904] Output: Analysis results (keywords and sentiment data).

[0905] Specific operation: The server uses spaCy to classify text data, such as "Welcome" as a greeting and "What are you looking for?" as confirmation of customer needs. At the same time, it uses IBM Watson Tone Analyzer to extract emotional data such as "satisfied" or "dissatisfied" from the customer's voice response.

[0906] Step 4:

[0907] The server stores the analyzed results in a database and displays them visually on a dashboard.

[0908] Input: Analysis results (keywords and sentiment data).

[0909] Output: Visualized data (graphs and charts).

[0910] What it does: The server stores the analysis results in a database and uses dashboard software such as Tableau to display the staff member customer satisfaction in the form of graphs and charts. This dashboard can be accessed by managers, allowing them to see at a glance the performance of specific staff members.

[0911] Step 5:

[0912] Users (managers, staff) analyze performance based on the visualized data and provide specific feedback.

[0913] Input: Visualized data (graphs and charts).

[0914] Output: Feedback (specific improvements and enhancements).

[0915] Specific Action: The manager opens the dashboard and checks the customer satisfaction data for a specific staff member. Then, based on the dashboard evaluation data, the manager points out that "a specific greeting phrase leads to high satisfaction" and provides feedback recommending similar customer service to other staff members. The manager also plans regular training sessions to improve overall customer service skills.

[0916] (Application example 2)

[0917] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0918] Efficient evaluation and improvement of customer service operations requires a system that comprehensively covers the entire process of collecting, converting, analyzing, visualizing, and providing feedback on voice data. However, achieving these processes efficiently in a single environment is difficult and time-consuming. In particular, there is a lack of a system for customer service staff and managers to access data in real time and provide prompt feedback, which delays improvements in service quality. In addition, there is a lack of systems that use mobile devices that can be used on-site, which reduces staff work efficiency.

[0919] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0920] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data to evaluate the customer service content, and a visualization means for visualizing the evaluated data. This enables customer service staff and managers to evaluate the quality of customer service operations in real time and provide quick feedback using smartphones.

[0921] The "voice collection means" is a device or method for collecting voices during customer service operations.

[0922] A "transcription means" is a device or method for converting collected audio data into text data.

[0923] The "analysis means" is a device or method for analyzing the converted character data and evaluating the customer service content.

[0924] A "visualization means" is a device or method for visually displaying evaluated data.

[0925] The "providing means" is a device or method for providing visualized data to a user.

[0926] "Mobile terminal means" refers to a device or method for collecting voice and displaying analysis results using a mobile terminal such as a smartphone.

[0927] The present invention is a system that collects voice data during customer service operations, analyzes the data, and evaluates and visualizes the customer service content. The system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a mobile terminal means.

[0928] System configuration and processing flow

[0929] Audio Collection

[0930] When a customer service staff member begins serving a customer, the server starts collecting voice data in real time. Using a mobile terminal such as a smartphone, the terminal records the voice and sends the data to the server. For example, when a store staff member says to a customer, "Welcome, what are you looking for today?", this voice is collected in real time.

[0931] Transcription

[0932] The collected voice data is converted into text data using a transcription method on the server. For example, the voice "Welcome, what are you looking for today?" is converted into text data "Welcome, what are you looking for today?". This process uses Python and the SpeechRecognition library.

[0933] Data analysis

[0934] The transcribed text data is then analyzed by an analysis tool on the server. During this process, the Transformers library is used as a natural language processing (NLP) engine to evaluate the customer service content and customer sentiment from the text data. For example, "Welcome" is recognized as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Furthermore, sentiment analysis is used to evaluate whether the customer is satisfied.

[0935] visualization

[0936] The analyzed data is visually displayed by the server using visualization means. For this purpose, the data is stored in a database and displayed as graphs and charts on a dashboard or similar. For example, it allows users to see at a glance whether a particular staff member is achieving high customer satisfaction.

[0937] Providing Feedback

[0938] The evaluated data is provided to users (managers and staff) using a delivery method. Mobile devices using smartphones enable immediate feedback. For example, a manager can check the dashboard and provide timely, specific advice to staff, such as "This phrase will increase customer satisfaction."

[0939] Specific examples

[0940] A specific example is a scene in a store where Staff A greets Customer B, saying, "Welcome, what are you looking for today?" This speech is collected by a mobile device (smartphone) and sent to a server. The server converts this speech into text data and analyzes it using a natural language processing engine. As a result of the analysis, "Welcome" is classified as a greeting, and "What are you looking for?" is extracted as confirmation of the customer's needs. Emotional data on whether the customer is satisfied or not can also be obtained from the customer's speech response. These results are stored in a database and visualized on a dashboard. Managers can refer to this and use it to provide customer service training to Staff A.

[0941] Prompt Sentence Examples

[0942] Analyze this conversation and evaluate the effectiveness of your customer service:

[0943] "Welcome, what are you looking for today?"

[0944] The analysis should include elements related to "greetings," "confirming customer needs," and "customer satisfaction."

[0945] This system makes it possible to improve the quality of customer service and enhance staff skills.

[0946] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0947] Step 1:

[0948] When a customer service staff member starts interacting with a customer, the server uses a voice collection means to collect voice data in real time. Specifically, the conversation between the customer service staff member and the customer is recorded from a terminal (smartphone) and the voice data is sent to the server. The input is the voice spoken during the interaction, and the output is the voice data sent to the server.

[0949] Step 2:

[0950] The server converts the audio data obtained from the audio collection means into text data using a transcription means. Specifically, it converts the audio data into text data using the SpeechRecognition library. The input is audio data, and the output is text data.

[0951] Step 3:

[0952] The server performs natural language processing (NLP) on the transcribed text data using analytical methods. Specifically, it analyzes the text data using the sentiment-analysis pipeline of the Transformers library and evaluates the customer service content and emotions. The input is text data, and the output is evaluation data as the analysis result.

[0953] Step 4:

[0954] The server visually displays the evaluation data obtained by the analysis means using visualization means. Specifically, the evaluation data is stored in a database and displayed in graph or chart format on a dashboard. The input is the evaluation data of the analysis results, and the output is visualized data in the form of graphs or charts.

[0955] Step 5:

[0956] The user (manager) checks the visualized data using the means provided. Specifically, they view the dashboard using a mobile device such as a smartphone and evaluate the customer service performance of the staff. The input is the visualized data on the dashboard, and the output is the feedback provided by the user.

[0957] Step 6:

[0958] Users can provide specific guidance to staff through feedback. For example, they can give advice such as "This phrase will increase customer satisfaction" based on dashboard data. This is expected to improve the staff's customer service skills. The input is the feedback content generated by the user, and the output is specific customer service improvement measures that the staff will put into practice.

[0959] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0960] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0961] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0962] [Fourth embodiment]

[0963] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0964] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0965] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0966] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0967] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0968] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0969] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0970] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0971] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0972] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0973] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0974] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0975] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0976] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, and a provision means.

[0977] System configuration and processing flow

[0978] 1. Collection of audio data

[0979] Terminal (AI microphone): When a staff member serving a customer begins serving them, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[0980] 2. Transcription of audio data

[0981] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[0982] 3. Data Analysis

[0983] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data and performs sentiment analysis and intent detection. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs.

[0984] 4. Performance visualization

[0985] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[0986] 5. Feedback

[0987] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[0988] Specific examples

[0989] Imagine a scene in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This voice is collected by a device (AI microphone) and sent to a server. The server transcribes the received voice data and analyzes the text using an analysis method (NLP engine). As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[0990] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[0991] The processing flow will be explained below.

[0992] Step 1:

[0993] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[0994] Step 2:

[0995] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[0996] Step 3:

[0997] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[0998] Step 4:

[0999] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[1000] Step 5:

[1001] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[1002] Step 6:

[1003] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[1004] Step 7:

[1005] The server stores the analysis results obtained from the NLP engine in a database, including extracted keywords, sentiment scores, and category classifications.

[1006] Step 8:

[1007] The server generates a dashboard based on the stored data, which visually displays the analysis results, such as graphs and charts showing high customer satisfaction for a particular staff member.

[1008] Step 9:

[1009] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[1010] Step 10:

[1011] Users can provide feedback based on the dashboard data, specifically sharing the customer service methods of high performers with other staff members and providing advice on how to improve overall service quality.

[1012] Through this series of processes, customer service data is collected, analyzed, and visualized, enabling specific feedback and service improvements.

[1013] Example 1

[1014] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1015] Modern customer service operations require accurate and efficient methods for evaluating employee service quality and customer satisfaction. However, traditional evaluation methods are subjective and time-consuming, resulting in issues with accuracy and efficiency. Furthermore, analysis of customer service content and feedback of evaluation results are not centrally managed, making it difficult to provide appropriate information to improve overall service quality.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1017] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization tool for visualizing the analyzed data, a provision means for providing the visualized data to a user, and a means for saving the analysis results in a database and displaying them on a dashboard accessible to a manager. This makes it possible to objectively and efficiently evaluate the quality of customer service operations and visually provide the analysis results to a manager, thereby improving employee performance and overall service quality.

[1018] The "voice collection means" is a means for collecting voice data in real time during customer service operations and transmitting the data to a server.

[1019] The "transcription means" is a means for converting collected voice data into text data. Specifically, it has a function of converting voice data into text using a transcription engine.

[1020] The "analysis means" is a means for analyzing the converted text data using a natural language processing engine, for example, to extract keywords, analyze sentiment, and detect intent.

[1021] A "visualization tool" is a tool for visually displaying the evaluation data obtained by the analysis means. Specifically, it is a tool that has the function of retrieving data from a database and displaying it in the form of a graph, chart, etc.

[1022] The "provision means" is a means for providing visualized data to a user. For example, it has a function of presenting the visualized data in a form accessible to an administrator via a dashboard.

[1023] A "database" is a device or system that stores information to store analyzed results and make them available for retrieval as needed.

[1024] A "dashboard" is an interface for visually displaying analysis results and evaluation data in a form accessible to administrators.

[1025] This invention relates to a system that collects voices during customer service operations, analyzes the data, evaluates the customer service, and provides the results to users. This system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization tool, and a presentation means.

[1026] System configuration

[1027] Audio collection method

[1028] A device (AI microphone) is used to collect voices during customer service. The device records conversations with customers in real time, compresses the data, and sends it to a server. For example, when a restaurant staff member says, "Welcome, what would you like to order today?", this voice data is collected.

[1029] Transcription methods

[1030] The server receives the voice data sent from the device. The received voice data is sent to a transcription engine such as Google Cloud Speech-to-Text and converted into text data. For example, the voice data "Welcome, what are you looking for today?" is converted into text data.

[1031] Analysis means

[1032] The server then sends the converted text data to a natural language processing (NLP) engine, such as the Google Cloud Natural Language API, for analysis. The analysis extracts keywords from the text data and performs sentiment analysis and intent detection. For example, it recognizes the phrase "Welcome" as a greeting and classifies "What are you looking for?" as a customer need confirmation.

[1033] Visualization Tools

[1034] The server stores the analyzed results in a database. The server then uses a visualization tool (e.g., Tableau or Google Data Studio) to display this data on a dashboard. The dashboard visually displays data on which specific staff members have achieved high customer satisfaction using graphs and charts.

[1035] Providing means

[1036] Users (managers and staff) can access the dashboard and evaluate staff performance based on visualized data. By providing specific feedback to staff, they can improve the quality of customer service. For example, they can point out that a particular greeting phrase leads to high customer satisfaction and encourage other staff to use the same greeting.

[1037] Specific examples

[1038] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" The audio of this exchange is collected by a device (AI microphone) and sent to a server. The server converts the received audio into text data using a transcription engine, and analyzes the text using an NLP engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" as confirmation of the customer's needs. These results are saved in a database and displayed on a dashboard, making it clear that Staff A's customer service is of high quality. The manager can use this data as a reference to provide feedback to other staff members, encouraging them to provide similar customer service.

[1039] Prompt Sentence Examples

[1040] Here are some examples of prompts for generative AI models:

[1041] Please model a customer service scenario in a clothing store. Assuming a scenario in which staff member A says to customer B, "Welcome, is there anything you're looking for today?", explain the process from collecting voice data to analyzing, visualizing, and providing feedback.

[1042] This allows the system to objectively and efficiently evaluate the quality of customer service operations and provide users with visual analysis results, thereby improving employee performance and overall service quality.

[1043] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1044] System program processing flow

[1045] Step 1: Collecting audio data

[1046] Terminal (AI microphone): When the terminal detects a user (staff member) starting customer service, it automatically starts collecting voice data.

[1047] Detailed operation: The terminal uses a microphone to record voice in real time and compresses the recorded voice data. For example, when a staff member says, "Welcome, what would you like to buy today?", this voice data is collected.

[1048] Input: Voice data (user speech)

[1049] Output: Compressed audio data (for transfer)

[1050] Step 2: Sending audio data

[1051] Terminal (AI microphone): The terminal sends compressed audio data to the server.

[1052] Detailed operation: The device uploads the collected and compressed audio data to a server via the Internet.

[1053] Input: Compressed audio data

[1054] Output: Audio data sent to the server

[1055] Step 3: Transcription

[1056] Server: The server receives the voice data sent from the terminal.

[1057] How it works in detail: The server sends the audio data to a transcription engine such as Google Cloud Speech-to-Text, which converts it into text data.

[1058] Input: Audio data

[1059] Output: Text data (speech-to-text conversion)

[1060] Step 4: Data analysis

[1061] Server: The server sends the transcribed text data to a natural language processing (NLP) engine.

[1062] How it works in detail: The server uses an NLP engine such as Google Cloud Natural Language API to perform keyword extraction, sentiment analysis, and intent detection from text data.

[1063] Input: Text data

[1064] Output: Analysis results (keywords, sentiment, intent)

[1065] Step 5: Save your data

[1066] Server: The server stores the analysis results in a database.

[1067] How it works in detail: The server records the analysis results in a database such as MySQL or PostgreSQL, making the data available for retrieval when needed.

[1068] Input: Analysis results

[1069] Output: Data stored in the database

[1070] Step 6: Visualize in a dashboard

[1071] Server: The server uses visualization tools to display the analysis results on a dashboard.

[1072] How it works in detail: The server uses visualization tools like Tableau or Google Data Studio to extract the necessary data from the database, convert it into graphs and charts, and display it on a dashboard.

[1073] Input: Data stored in a database

[1074] Output: Visualized data on a dashboard

[1075] Step 7: Feedback

[1076] User (Manager): The user accesses the dashboard and evaluates staff performance based on visualized data.

[1077] Detailed Action: The manager reviews the data displayed on the dashboard and provides specific feedback to staff, for example, "A specific greeting phrase leads to high satisfaction."

[1078] Input: Data visualized on the dashboard

[1079] Output: Feedback (instructions for staff)

[1080] This series of processes allows the quality of customer service to be objectively and efficiently evaluated, and the analysis results are presented to the user visually, making it possible to improve employee performance and overall service quality.

[1081] (Application example 1)

[1082] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1083] Conventional customer service performance evaluation systems have the problem that it is difficult to analyze customer service content in real time and provide immediate feedback to staff. In addition, there are insufficient means to visually display the evaluation data, making it difficult to effectively support staff performance improvement.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1085] In this invention, the server includes a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a means for visualizing data collected during customer service operations and having a display device that allows the user to check the data. This enables real-time analysis and visual display of customer service content, provides immediate feedback to staff, and enables improvement of customer service performance.

[1086] "Speech collection means" refers to a device or technology that collects speech generated during customer service operations.

[1087] A "transcription means" is a device or technology that converts collected audio data into text data.

[1088] The "analysis means" refers to a device or technology that analyzes the converted character data and evaluates the customer service content.

[1089] "Visualization means" refers to a device or technology that visually displays the evaluated data.

[1090] "Providing means" refers to the device or technology that provides visualized data to users.

[1091] A "display device" is a device that visualizes data collected during customer service operations and allows users to check it.

[1092] To implement the present invention, the following system configuration and processing flow are used.

[1093] System Configuration

[1094] The system includes the following hardware and software:

[1095] Hardware:

[1096] Audio collection means (e.g., microphone in smart glasses)

[1097] Server (e.g. cloud server)

[1098] Display devices (e.g., smart glasses displays or external monitors)

[1099] software:

[1100] Audio collection library (e.g., Google Cloud Speech-to-Text API)

[1101] Natural language processing engines (e.g., spaCy, Google Cloud Natural Language API)

[1102] Database (e.g. Firebase Realtime Database)

[1103] Data visualization tools (e.g., Google Data Studio)

[1104] Program processing

[1105] The server receives real-time voice data from voice collection devices such as smart glasses during customer service. The voice data is converted into text data using the Google Cloud Speech-to-Text API. The converted text data is then analyzed using the Google Cloud Natural Language API. This analysis involves keyword extraction, sentiment analysis, and intent detection. The analysis results are stored in the Firebase Realtime Database and then visually displayed using Google Data Studio.

[1106] Explanations and examples

[1107] For example, imagine a scenario where a staff member at a clothing store greets a customer through smart glasses, saying, "Welcome, what are you looking for today?" This speech is collected by the smart glasses' microphone and sent to a cloud server. The server converts the speech into text data using the Google Cloud Speech-to-Text API, and then analyzes the data using the Google Cloud Natural Language API. Through this analysis, the phrase "Welcome" is recognized as a greeting, and "What are you looking for today?" is classified as a confirmation of the customer's needs. The analysis results are stored in the Firebase Realtime Database and visualized using Google Data Studio.

[1108] Staff and managers can view the analysis results in real time on the smart glasses display or an external monitor, and receive immediate feedback, which can help improve customer service performance.

[1109] Prompt Sentence Examples

[1110] By inputting the prompt below into the generative AI model, you can check the application overview and processing flow.

[1111] "Based on the following description, please provide a detailed description of an application that analyzes customer service performance in a brick-and-mortar store in real time and provides feedback through smart glasses."

[1112] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1113] Step 1:

[1114] Audio collection method

[1115] The device (the microphone in the smart glasses) collects voice data in real time during customer service. Voice is captured from the environment and stored as a digital audio file within the device. This becomes input data. The device then transmits the collected voice data to a server.

[1116] Step 2:

[1117] Sending audio data

[1118] The collected voice data is sent from the device to the server in real time. Specifically, the voice data is uploaded to a cloud server via a mobile communication network. The input is the voice data collected by the device, and the server receives this data.

[1119] Step 3:

[1120] Transcription of audio data

[1121] The server sends the received voice data to the Google Cloud Speech-to-Text API, which converts it into text data. This API performs speech analysis and outputs text data. The input is voice data, and the output is the corresponding text data.

[1122] Step 4:

[1123] Parsing character data

[1124] The server uses the Google Cloud Natural Language API to analyze text data. Specifically, it uses this API to extract keywords, analyze sentiment, and detect intent from text. The input is text data, and the output is the analysis results (keyword list, sentiment score, and intent label).

[1125] Step 5:

[1126] Saving analysis data

[1127] The server stores the analysis results in the Firebase Realtime Database. The input is the analysis data, and the output is the state stored in the database. This data is later used for visualization and feedback.

[1128] Step 6:

[1129] Data Visualization

[1130] After the analysis results are saved, the server visualizes these data using Google Data Studio. The visualization tool generates a dashboard that displays the analysis results in the form of graphs and charts. The input is the saved analysis data, and the output is the visually displayed information.

[1131] Step 7:

[1132] Providing Feedback

[1133] Users (staff or managers) can access visualized data through the smart glasses display or an external monitor and receive real-time feedback. The input is visual data, and the output is feedback information confirmed by the user. This helps users to immediately improve their customer service performance.

[1134] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1135] This invention relates to a system that collects voices during customer service operations, analyzes the data, and visualizes the customer service content of high performers. This system is mainly composed of a voice collection means, a transcription means, an analysis means, a visualization means, a presentation means, and an emotion engine.

[1136] System configuration and processing flow

[1137] 1. Collection of audio data

[1138] Terminal (AI microphone): When a wait staff member begins serving a customer, it automatically starts collecting voice data. The AI ​​microphone records the conversation in real time and sends it to the server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[1139] 2. Transcription of audio data

[1140] Server: Receives voice data sent from the device and converts it into text data using a transcription engine. For example, the voice collected by the AI ​​microphone, "Welcome, what are you looking for today?", is converted into text data.

[1141] 3. Data Analysis

[1142] Server: The transcribed text data is analyzed using a natural language processing (NLP) engine. Specifically, the NLP engine extracts keywords from the text data, and the emotion engine recognizes the user's emotions. For example, the keyword "Welcome" is classified as a customer service greeting, and "What are you looking for?" is recognized as confirmation of the customer's needs. In addition, the emotion engine extracts emotional data on whether the customer is satisfied based on the customer's voice response.

[1143] 4. Performance visualization

[1144] Server: The analyzed results are stored in a database and displayed visually on a dashboard. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction. This dashboard can be accessed by administrators.

[1145] 5. Feedback

[1146] Users (managers, staff): Based on the visualized data, managers can analyze staff performance and provide specific feedback. For example, referring to the evaluation data on the dashboard, managers can point out that "a particular greeting phrase leads to high satisfaction."

[1147] Specific examples

[1148] Consider a scenario in a clothing store where Staff A greets Customer B, saying, "Welcome, are you looking for something today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using an analysis method (NLP engine) and an emotion engine. As a result of the analysis, "Welcome" is classified as a greeting, and "Are you looking for something?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and choice of words. These results are stored in a database and displayed on a dashboard, visualizing the high quality of Staff A's customer service. The manager uses this data as a reference and provides feedback to other staff members, encouraging them to provide similar customer service.

[1149] This system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality. In addition, by incorporating customer sentiment data, more detailed and specific feedback is possible, helping to improve staff customer service skills.

[1150] The processing flow will be explained below.

[1151] Step 1:

[1152] The terminal (AI microphone) collects the voice of the customer service staff in real time. The microphone automatically records the conversation with the customer and temporarily stores the voice data on the device.

[1153] Step 2:

[1154] The device divides the collected audio data into batches of a fixed duration or size, and once the data is converted into a suitable format, it is uploaded to a server using a secure protocol (e.g., HTTPS).

[1155] Step 3:

[1156] The server receives the voice data sent from the terminal, and stores the received voice data in a database or temporary storage.

[1157] Step 4:

[1158] The server sends the received audio data to a transcription engine. During this process, the audio data is converted into text data. For example, the audio saying "Welcome" becomes the text data "Welcome."

[1159] Step 5:

[1160] The server preprocesses the transcribed text data, which includes removing unnecessary spaces and special characters and normalizing the text.

[1161] Step 6:

[1162] The server passes the preprocessed text data to a natural language processing (NLP) engine for analysis. This analysis involves extracting keywords, analyzing sentiment, and detecting intent. For example, keywords such as "Welcome" and "Are you looking for?" are extracted.

[1163] Step 7:

[1164] The server passes the analysis results to the emotion engine, which recognizes the customer's emotions (such as "satisfaction," "dissatisfaction," or "happiness") from the customer's tone of voice and the way they use words.

[1165] Step 8:

[1166] The server integrates the emotion data from the emotion engine with the analysis results from the NLP engine, including extracted keywords, customer emotion scores, and emotion categories.

[1167] Step 9:

[1168] The server stores the consolidated data in a database, which includes analytical data and emotion data corresponding to each customer interaction event.

[1169] Step 10:

[1170] The server generates a dashboard based on the stored data. The dashboard visually displays each staff member's performance and customer sentiment data. For example, graphs and charts show that a particular staff member has a high level of customer satisfaction.

[1171] Step 11:

[1172] Users (managers and staff) can access the dashboard to check customer service data. Managers can understand the performance of specific staff members and areas for improvement.

[1173] Step 12:

[1174] Users can provide feedback based on the dashboard data, specifically offering specific advice to other staff members based on the customer service methods of high performers and customer sentiment data.

[1175] This process allows for the collection, analysis, and visualization of customer service data, enabling specific feedback and service improvement. Customer sentiment data is also taken into account, enabling more detailed and accurate evaluations.

[1176] Example 2

[1177] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] Effective evaluation and feedback of staff performance in customer service operations is a challenge. In particular, efficient collection and accurate analysis of voice data during customer interactions is essential to improve staff customer service skills. Furthermore, integrating real-time data collection and emotion data analysis is essential to provide more detailed and specific feedback and improve overall service quality.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1180] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data and evaluating the customer service content, a visualization means for visualizing the evaluated data, an emotion engine means for analyzing emotion data, and a provision means for providing the visualized data to the user. This makes it possible to collect voice data during customer service operations in real time and provide highly accurate feedback based on the analysis results.

[1181] The "voice collection means" is a device or system for collecting and recording voices during customer service operations in real time.

[1182] A "transcription means" is software or hardware for converting collected audio data into text data.

[1183] "Analysis means" refers to natural language processing engines and other analysis tools that use text data to evaluate customer service content.

[1184] A "visualization means" is a system that includes dashboards and graph generation tools for visually displaying the evaluated data.

[1185] The "emotion engine means" is software or hardware for analyzing emotions from voice data or text data and recognizing the user's emotional state.

[1186] "Providing means" refers to the interface or system for providing visualized data to users.

[1187] The present invention is a system that collects voices during customer service operations, analyzes the data, evaluates the service content and customer emotions, and visualizes them. It mainly includes a voice collection means, a transcription means, an analysis means, a visualization means, an emotion engine means, and a provision means.

[1188] Audio collection method

[1189] The device (e.g., an AI microphone) automatically starts collecting voice data when a waiter begins serving a customer. The device records the conversation in real time and sends the collected voice data to a server. For example, if a restaurant staff member says, "Welcome, what would you like to order today?", this voice is collected.

[1190] Transcription methods

[1191] The server receives the voice data sent from the device and converts the voice into text data using a transcription engine (e.g., Google Cloud Speech-to-Text API). For example, the collected voice "Welcome, what are you looking for today?" is converted into text data.

[1192] Analysis means

[1193] The server analyzes the transcribed text data using a natural language processing (NLP) engine (e.g., spaCy) and an emotion engine (e.g., IBM Watson Tone Analyzer). Through this analysis, keywords are extracted from the text data and the customer service content is classified. Emotional data is also extracted from the customer's voice response. For example, "Welcome" is classified as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Emotional data such as "satisfied" or "dissatisfied" is also extracted from the customer's tone of voice and choice of words.

[1194] Visualization means

[1195] The server stores the analyzed data in a database and displays it visually on a dashboard. This dashboard shows the customer satisfaction achieved by a specific staff member in the form of graphs and charts. For example, it becomes possible to see at a glance that a specific staff member has achieved high customer satisfaction. This dashboard can be accessed by administrators.

[1196] Providing means

[1197] Users (managers or staff) can analyze staff performance based on the visualized data and provide specific feedback. For example, they can refer to the dashboard evaluation data and point out that a particular greeting phrase leads to high customer satisfaction, and recommend similar customer service to other staff members.

[1198] Specific examples

[1199] Imagine a scenario in a clothing store where Staff A greets Customer B with, "Welcome, are you looking for anything today?" This speech is collected by a device (AI microphone) and sent to a server. The server transcribes the received speech data and analyzes the text using analytical methods (NLP engine and emotion engine). As a result, "Welcome" is classified as a greeting, and "Are you looking for anything?" is extracted as confirmation of the customer's needs. At the same time, the emotion engine extracts emotional data such as "satisfied" or "dissatisfied" from the customer's tone of voice and phrasing. These results are stored in a database and displayed on a dashboard, visualizing the quality of Staff A's customer service. Managers can use this data to provide specific feedback to other staff members and work to improve overall customer service skills.

[1200] Prompt Sentence Examples

[1201] Enter the following prompt into the generative AI model:

[1202] "Please explain the specific process flow of a system that analyzes voice data during customer service operations, visualizes staff performance, and provides feedback. Please provide a detailed description of the specific operations at each step."

[1203] In this way, this system enables efficient evaluation and improvement of customer service operations, contributing to the improvement of overall service quality.

[1204] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1205] Step 1:

[1206] The device (AI microphone) automatically starts collecting voice data when a waiter starts interacting with a customer. Specifically, the device records the conversation in real time and sends the voice data to a server.

[1207] Input: Audio of conversation between wait staff and customer.

[1208] Output: The audio data sent to the server.

[1209] How it works: The multi-function AI microphone starts collecting voice data the moment a waiter says, "Welcome, what would you like to order today?" The collected voice data is then compressed and sent to a server in real time via Wi-Fi or Bluetooth.

[1210] Step 2:

[1211] The server receives the voice data sent from the terminal and converts the voice data into text data using a transcription engine.

[1212] Input: Audio data sent from the device.

[1213] Output: Character data.

[1214] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the received audio, "Welcome, what are you looking for today?", into text data in real time.

[1215] Step 3:

[1216] The server analyzes the transcribed text data using a natural language processing (NLP) engine and an emotion engine.

[1217] Input: Character data.

[1218] Output: Analysis results (keywords and sentiment data).

[1219] Specific operation: The server uses spaCy to classify text data, such as "Welcome" as a greeting and "What are you looking for?" as confirmation of customer needs. At the same time, it uses IBM Watson Tone Analyzer to extract emotional data such as "satisfied" or "dissatisfied" from the customer's voice response.

[1220] Step 4:

[1221] The server stores the analyzed results in a database and displays them visually on a dashboard.

[1222] Input: Analysis results (keywords and sentiment data).

[1223] Output: Visualized data (graphs and charts).

[1224] What it does: The server stores the analysis results in a database and uses dashboard software such as Tableau to display the staff member customer satisfaction in the form of graphs and charts. This dashboard can be accessed by managers, allowing them to see at a glance the performance of specific staff members.

[1225] Step 5:

[1226] Users (managers, staff) analyze performance based on the visualized data and provide specific feedback.

[1227] Input: Visualized data (graphs and charts).

[1228] Output: Feedback (specific improvements and enhancements).

[1229] Specific Action: The manager opens the dashboard and checks the customer satisfaction data for a specific staff member. Then, based on the dashboard evaluation data, the manager points out that "a specific greeting phrase leads to high satisfaction" and provides feedback recommending similar customer service to other staff members. The manager also plans regular training sessions to improve overall customer service skills.

[1230] (Application example 2)

[1231] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1232] Efficient evaluation and improvement of customer service operations requires a system that comprehensively covers the entire process of collecting, converting, analyzing, visualizing, and providing feedback on voice data. However, achieving these processes efficiently in a single environment is difficult and time-consuming. In particular, there is a lack of a system for customer service staff and managers to access data in real time and provide prompt feedback, which delays improvements in service quality. In addition, there is a lack of systems that use mobile devices that can be used on-site, which reduces staff work efficiency.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1234] In this invention, the server includes a voice collection means for collecting voices during customer service operations, a transcription means for converting the collected voice data into text data, an analysis means for analyzing the converted text data to evaluate the customer service content, and a visualization means for visualizing the evaluated data. This enables customer service staff and managers to evaluate the quality of customer service operations in real time and provide quick feedback using smartphones.

[1235] The "voice collection means" is a device or method for collecting voices during customer service operations.

[1236] A "transcription means" is a device or method for converting collected audio data into text data.

[1237] The "analysis means" is a device or method for analyzing the converted character data and evaluating the customer service content.

[1238] A "visualization means" is a device or method for visually displaying evaluated data.

[1239] The "providing means" is a device or method for providing visualized data to a user.

[1240] "Mobile terminal means" refers to a device or method for collecting voice and displaying analysis results using a mobile terminal such as a smartphone.

[1241] The present invention is a system that collects voice data during customer service operations, analyzes the data, and evaluates and visualizes the customer service content. The system mainly consists of a voice collection means, a transcription means, an analysis means, a visualization means, a provision means, and a mobile terminal means.

[1242] System configuration and processing flow

[1243] Audio Collection

[1244] When a customer service staff member begins serving a customer, the server starts collecting voice data in real time. Using a mobile terminal such as a smartphone, the terminal records the voice and sends the data to the server. For example, when a store staff member says to a customer, "Welcome, what are you looking for today?", this voice is collected in real time.

[1245] Transcription

[1246] The collected voice data is converted into text data using a transcription method on the server. For example, the voice "Welcome, what are you looking for today?" is converted into text data "Welcome, what are you looking for today?". This process uses Python and the SpeechRecognition library.

[1247] Data analysis

[1248] The transcribed text data is then analyzed by an analysis tool on the server. During this process, the Transformers library is used as a natural language processing (NLP) engine to evaluate the customer service content and customer sentiment from the text data. For example, "Welcome" is recognized as a greeting, and "What are you looking for?" is classified as confirmation of the customer's needs. Furthermore, sentiment analysis is used to evaluate whether the customer is satisfied.

[1249] visualization

[1250] The analyzed data is visually displayed by the server using visualization means. For this purpose, the data is stored in a database and displayed as graphs and charts on a dashboard or similar. For example, it allows users to see at a glance whether a particular staff member is achieving high customer satisfaction.

[1251] Providing Feedback

[1252] The evaluated data is provided to users (managers and staff) using a delivery method. Mobile devices using smartphones enable immediate feedback. For example, a manager can check the dashboard and provide timely, specific advice to staff, such as "This phrase will increase customer satisfaction."

[1253] Specific examples

[1254] A specific example is a scene in a store where Staff A greets Customer B, saying, "Welcome, what are you looking for today?" This speech is collected by a mobile device (smartphone) and sent to a server. The server converts this speech into text data and analyzes it using a natural language processing engine. As a result of the analysis, "Welcome" is classified as a greeting, and "What are you looking for?" is extracted as confirmation of the customer's needs. Emotional data on whether the customer is satisfied or not can also be obtained from the customer's speech response. These results are stored in a database and visualized on a dashboard. Managers can refer to this and use it to provide customer service training to Staff A.

[1255] Prompt Sentence Examples

[1256] Analyze this conversation and evaluate the effectiveness of your customer service:

[1257] "Welcome, what are you looking for today?"

[1258] The analysis should include elements related to "greetings," "confirming customer needs," and "customer satisfaction."

[1259] This system makes it possible to improve the quality of customer service and enhance staff skills.

[1260] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1261] Step 1:

[1262] When a customer service staff member starts interacting with a customer, the server uses a voice collection means to collect voice data in real time. Specifically, the conversation between the customer service staff member and the customer is recorded from a terminal (smartphone) and the voice data is sent to the server. The input is the voice spoken during the interaction, and the output is the voice data sent to the server.

[1263] Step 2:

[1264] The server converts the audio data obtained from the audio collection means into text data using a transcription means. Specifically, it converts the audio data into text data using the SpeechRecognition library. The input is audio data, and the output is text data.

[1265] Step 3:

[1266] The server performs natural language processing (NLP) on the transcribed text data using analytical methods. Specifically, it analyzes the text data using the sentiment-analysis pipeline of the Transformers library and evaluates the customer service content and emotions. The input is text data, and the output is evaluation data as the analysis result.

[1267] Step 4:

[1268] The server visually displays the evaluation data obtained by the analysis means using visualization means. Specifically, the evaluation data is stored in a database and displayed in graph or chart format on a dashboard. The input is the evaluation data of the analysis results, and the output is visualized data in the form of graphs or charts.

[1269] Step 5:

[1270] The user (manager) checks the visualized data using the means provided. Specifically, they view the dashboard using a mobile device such as a smartphone and evaluate the customer service performance of the staff. The input is the visualized data on the dashboard, and the output is the feedback provided by the user.

[1271] Step 6:

[1272] Users can provide specific guidance to staff through feedback. For example, they can give advice such as "This phrase will increase customer satisfaction" based on dashboard data. This is expected to improve the staff's customer service skills. The input is the feedback content generated by the user, and the output is specific customer service improvement measures that the staff will put into practice.

[1273] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1276] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1277] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1278] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1279] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1280] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1281] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1282] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1283] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1284] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1285] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1286] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1287] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1288] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1289] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1290] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1291] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1292] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1293] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1294] The following is further disclosed regarding the above embodiment.

[1295] (Claim 1)

[1296] a voice collecting means for collecting voices during customer service operations;

[1297] a transcription means for converting the collected voice data into text data;

[1298] an analysis means for analyzing the converted character data and evaluating the customer service content;

[1299] visualization means for visualizing the evaluated data;

[1300] providing means for providing the visualized data to a user;

[1301] A system including:

[1302] (Claim 2)

[1303] The system according to claim 1, further comprising means for collecting voices during customer service in real time.

[1304] (Claim 3)

[1305] 10. The system of claim 1, further comprising means for analyzing the character data using a natural language processing engine.

[1306] "Example 1"

[1307] (Claim 1)

[1308] a voice collecting means for collecting voices during customer service operations;

[1309] a transcription means for converting the collected voice data into text data;

[1310] an analysis means for analyzing the converted character data and evaluating the customer service content;

[1311] a visualization tool for visualizing the evaluated data;

[1312] providing means for providing the visualized data to a user;

[1313] A means to store the analyzed results in a database and display them on an administrator-accessible dashboard;

[1314] A system including:

[1315] (Claim 2)

[1316] A means for collecting voices during customer service operations in real time and transmitting compressed voice data to a server;

[1317] The server has a means for transmitting the voice data to a transcription engine and converting the voice data into text while performing noise reduction.

[1318] 10. The system of claim 1.

[1319] (Claim 3)

[1320] A means for transmitting character data to a natural language processing engine and performing keyword extraction, sentiment analysis, and intent detection;

[1321] 10. The system of claim 1.

[1322] "Application Example 1"

[1323] (Claim 1)

[1324] a voice collecting means for collecting voices during customer service operations;

[1325] a transcription means for converting the collected voice data into text data;

[1326] an analysis means for analyzing the converted character data and evaluating the customer service content;

[1327] visualization means for visualizing the evaluated data;

[1328] providing means for providing the visualized data to a user;

[1329] It is equipped with a display device that visualizes data collected during customer service operations and allows users to check it.

[1330] A system including:

[1331] (Claim 2)

[1332] The system according to claim 1, further comprising means for collecting voices during customer service in real time.

[1333] (Claim 3)

[1334] 10. The system of claim 1, further comprising means for analyzing the character data using a natural language processing engine.

[1335] "Example 2: Combining Emotion Engines"

[1336] (Claim 1)

[1337] a voice collecting means for collecting voices during customer service operations;

[1338] a transcription means for converting the collected voice data into text data;

[1339] an analysis means for analyzing the converted character data and evaluating the customer service content;

[1340] visualization means for visualizing the evaluated data;

[1341] emotion engine means for analyzing emotion data;

[1342] providing means for providing the visualized data to a user;

[1343] A system including:

[1344] (Claim 2)

[1345] The system according to claim 1, further comprising means for collecting voices during customer service in real time.

[1346] (Claim 3)

[1347] 10. The system of claim 1, further comprising means for analyzing the character data using a natural language processing engine.

[1348] "Application example 2 when combining emotion engines"

[1349] (Claim 1)

[1350] a voice collecting means for collecting voices during customer service operations;

[1351] a transcription means for converting the collected voice data into text data;

[1352] an analysis means for analyzing the converted character data and evaluating the customer service content;

[1353] visualization means for visualizing the evaluated data;

[1354] providing means for providing the visualized data to a user;

[1355] a mobile terminal means for collecting voice and displaying analysis results using a smartphone;

[1356] A system including:

[1357] (Claim 2)

[1358] The system according to claim 1, further comprising means for collecting voices during customer service in real time.

[1359] (Claim 3)

[1360] 10. The system of claim 1, further comprising means for analyzing the character data using a natural language processing engine. [Explanation of symbols]

[1361] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice collecting means for collecting voices during customer service operations; a transcription means for converting the collected voice data into text data; an analysis means for analyzing the converted character data and evaluating the customer service content; visualization means for visualizing the evaluated data; providing means for providing the visualized data to a user; A system including:

2. The system according to claim 1, further comprising means for collecting voices during customer service in real time.

3. 2. The system according to claim 1, further comprising means for analyzing the character data using a natural language processing engine.

Images