System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting audio and video during a AI with a customer; means for using a generative dialogue model to analyze the collected audio and video to identify an emotion of the customer; and means for calculating an emotional score of the customer based on the analysis and performing a correction to eliminate an answer central tendency to calculate an accurate customer satisfaction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When accurately calculating customer satisfaction (NPS) in Japan, there is a problem that true customer satisfaction is difficult to reflect due to the tendency of Japanese people to center their responses. To solve this problem, a system is needed that does not rely on the customer's subjective responses, but accurately grasps customer emotions based on objective data, and can calculate an accurate NPS. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by including a means for collecting audio and video data during a conversation with a customer, a means for using a generative AI model to analyze the collected audio and video data to identify the customer's emotions, a means for calculating a customer's emotion score based on the analysis results, a correction for eliminating response centralization, and a means for calculating an accurate customer satisfaction rating. Such a system further includes a means for noise reduction and audio feature extraction processing of the audio data, and a means for processing the video data for face recognition and emotion classification for each frame, thereby enabling highly accurate emotion analysis.

[0006] "Audio data" refers to audio signals collected by a microphone, recording device, or the like.

[0007] "Video data" refers to video signals collected by cameras, video equipment, and the like.

[0008] "Means for collecting" refers to hardware and software for recording audio and video data in real time.

[0009] A "generative AI model" refers to an algorithm that uses machine learning and artificial intelligence techniques to analyze data and extract specific patterns and features.

[0010] "Means for identifying emotions" includes processing power and algorithms for analyzing collected audio and video data to infer the emotional state of the customer.

[0011] The "emotion score" represents a numerical index of the intensity and type of the analyzed emotion.

[0012] "Response centralization" refers to the tendency of respondents to choose a middle answer, and is a phenomenon that is particularly influenced by the cultural background of Japanese people.

[0013] "Noise reduction" refers to a process for removing unwanted background sounds and noise from audio data and extracting a clean audio signal.

[0014] "Audio feature extraction processing" refers to a computational process for extracting meaningful features (for example, pitch or tone) from audio data.

[0015] "Facial recognition" refers to an algorithm for identifying a person's face from video data and extracting its location and features.

[0016] "Emotion classification" refers to the process of analyzing facial features extracted by face recognition and classifying emotions such as smile, surprise, and anger.

[0017] "Accurate customer satisfaction" refers to a value that objectively and accurately reflects the actual level of customer satisfaction. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[0040] System Overview

[0041] Terminal: A front-end device that handles customer service and collects audio and video data.

[0042] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[0043] User: Operates the system, understands customer satisfaction, and improves services.

[0044] Program processing explanation

[0045] 1. Data Collection Phase

[0046] During a conversation with a customer, the device uses a built-in microphone and camera to collect audio and video data in real time, for example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[0047] 2. Data transmission phase

[0048] The collected audio and video data is sent from the device to a server, and data transmission is always via a secure protocol to protect the integrity and privacy of the data.

[0049] 3. Data analysis phase

[0050] The server applies a noise reduction filter to the received audio data to make it clearer, for example, by removing background speech and other noise.

[0051] Next, a facial recognition algorithm is used to detect the customer's face from the video data and extract facial features, which can detect minute changes such as eye and mouth movements and wrinkles.

[0052] 4. Emotion Identification Phase

[0053] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained based on a large amount of past data sets, enabling highly accurate emotion identification.

[0054] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "excited" or "calm."

[0055] The system evaluates changes in facial expressions from video data and classifies emotions such as smiles, surprise, and anger with high accuracy.

[0056] 5. Sentiment score and NPS calculation phase

[0057] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[0058] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[0059] 6. Reporting Phase

[0060] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the history of sentiment scores and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[0061] Specific examples

[0062] Example 1: Calculating NPS in a call center

[0063] Data collection: The device collects the voices of both the operator and the customer during the call, and records the operator's facial expressions with a camera.

[0064] Data transmission: These data are transmitted to the server in real time.

[0065] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[0066] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0067] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[0068] Example 2: Customer service at the hotel front desk

[0069] Data collection: The device records audio and video of the customer and staff interaction during check-in.

[0070] Data transmission: Send the recorded data to the server.

[0071] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[0072] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[0073] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[0074] This is a groundbreaking system that improves service quality by precisely understanding customer emotions based on audio and video analysis and more accurately reflecting NPS. In this way, it is possible to evaluate customer satisfaction more specifically and rationally and accurately derive areas for improvement.

[0075] The processing flow will be explained below.

[0076] Step 1: Data collection

[0077] While the device is interacting with the customer, it uses its built-in microphone and camera to collect audio and video data in real time.

[0078] Specifically, a microphone captures the conversational audio, and a camera records the customer's facial expressions frame by frame.

[0079] Step 2: Send data

[0080] The terminal converts the collected audio data into a digital signal and compresses and encodes the video data frame by frame.

[0081] The encoded data is divided into data packets and sent to the server using a secure protocol.

[0082] Step 3: Data reception and preprocessing

[0083] The server receives the data packets sent from the terminal.

[0084] The integrity of the data packets is verified, and the data is consolidated and restored to its original form.

[0085] The server applies a noise reduction filter to the audio data to remove unwanted background sounds.

[0086] Video data is analyzed frame by frame to perform face detection and tracking.

[0087] Step 4: Audio data analysis

[0088] The server inputs the voice data into a speech recognition model and converts it into text.

[0089] Extract speech features such as tone, pitch, and speed.

[0090] Based on the extracted features, a generative AI model identifies emotions.

[0091] Step 5: Video data analysis

[0092] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[0093] It analyzes features such as eye movements, mouth movements, and facial expressions to classify emotions.

[0094] Step 6: Integrating sentiment scores

[0095] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[0096] Each feature is weighted and an overall evaluation is calculated.

[0097] Step 7: Calculate your NPS

[0098] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the emotional scores.

[0099] Calculate an accurate NPS based on sentiment scores and apply a correction to eliminate response centralization bias.

[0100] Step 8: Report your findings

[0101] The server sends the calculated NPS value and related analytical data to the device.

[0102] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Accurately calculating customer satisfaction (NPS) is essential for improving a company's services. However, due to the tendency toward response-centering that is unique to Japan, it has been difficult to accurately calculate NPS using conventional methods. Furthermore, even when identifying customer emotions using audio and video data, noise and unclear data can reduce the accuracy of the analysis, which has been a problem. Therefore, there is a strong demand for the development of a system that can analyze customer audio and video data to accurately identify emotions and calculate an adjusted NPS based on the results.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for transmitting collected audio and video data using an Internet Protocol, means for applying a noise reduction filter to the received audio data, means for detecting a customer's face from the video data using a facial recognition algorithm and extracting facial expression features, means for inputting the analyzed audio and video data into a generative AI model and identifying the customer's emotion using a prompt sentence, means for quantifying the identified emotion and expressing it as an emotion score, means for calculating customer satisfaction based on the emotion score and correcting it to eliminate answer centralization tendency, and means for generating a dashboard for transmitting the calculated customer satisfaction to a terminal and visualizing it. This improves the accuracy of analysis of audio and video data, making it possible to accurately identify customer emotion and calculate a corrected NPS.

[0108] "Voice data" refers to recordings of speech collected during customer interactions and is a digital audio signal captured using a microphone.

[0109] "Video Data" refers to video recordings collected during interactions with customers and is a digital video signal captured using a camera.

[0110] The "Internet Protocol" is a set of protocols for sending and receiving data over a network, and defines how data is packetized and transmitted.

[0111] A "noise reduction filter" is an algorithm or software process used to remove unwanted noise and interference from audio data.

[0112] A "face recognition algorithm" is a computational method for detecting a person's face from video data and extracting its features.

[0113] A "generative AI model" is an artificial intelligence algorithm that is trained on large datasets to generate specific information from audio or video data.

[0114] A "prompt" is an instruction or question that is input into a generative AI model to enable it to perform a specific task.

[0115] An "emotion score" is a numerical representation of the emotion identified from the analyzed audio and video data.

[0116] "Customer satisfaction" is an indicator of how customers rate their satisfaction with their experiences and services, and includes the Net Promoter Score (NPS).

[0117] "Response centralization" is the tendency of respondents in a particular cultural context to choose the middle or average answer.

[0118] A "dashboard" is an interface for centrally displaying data visualization and analysis results.

[0119] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[0120] System Overview

[0121] Terminal: A front-end device that handles customer service and collects audio and video data. Specific hardware features include high-sensitivity microphones and HD cameras. Examples include the Bose NC700 and Logitech C920.

[0122] Server: A backend device that analyzes data, calculates sentiment scores, and calculates NPS. The server has advanced computing power and large data storage capacity.

[0123] Users: Operate the system, understand customer satisfaction, and improve services. Users can use the dashboard to visually check the analysis results.

[0124] Program processing explanation

[0125] 1. Data Collection Phase

[0126] While the terminal is interacting with the customer, it uses a built-in microphone (e.g., a high-sensitivity microphone) and a camera (e.g., an HD camera) to collect audio and video data in real time. For example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[0127] 2. Data transmission phase

[0128] The collected audio and video data is sent from the device to the server. Data transmission is via a secure protocol (e.g., SSL / TLS) to protect the integrity and privacy of the data.

[0129] 3. Data analysis phase

[0130] The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data to remove background noise and unwanted audio components. Specifically, the server analyzes the audio spectrum using an FFT (Fast Fourier Transform) and attenuates the noise components.

[0131] Next, the server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and creates a list of feature points.

[0132] 4. Emotion Identification Phase

[0133] The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) to identify the customer's emotions. For example, for audio data, a request is sent along with the prompt, "Please identify the customer's emotions from this audio data." The returned output includes emotions such as "elated," "calm," and "anxious."

[0134] For video data, the prompt is, "Analyze the customer's facial expressions in this video and identify their emotions." The model's output includes emotions such as "smile," "surprise," and "anger."

[0135] 5. Sentiment score and NPS calculation phase

[0136] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience. The NPS is calculated based on the emotion score, and by adjusting for the bias of intermediate responses using statistical methods, the tendency to center responses, which is unique to Japan, can be eliminated.

[0137] 6. Reporting Phase

[0138] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the sentiment score history and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[0139] Specific examples

[0140] Example 1: Calculating NPS in a call center

[0141] Data collection: The device collects the voices of both the operator and the customer during the call using a built-in microphone, and records the operator's facial expressions using a camera.

[0142] Data transmission: Collected data is transmitted to the server in real time.

[0143] Emotion analysis: The server analyzes the operator's tone and pitch from the voice data and extracts changes in the operator's facial expressions from the video data.

[0144] Emotion Identification: The server identifies emotions using a generative AI model.

[0145] NPS calculation: Calculate the adjusted NPS based on the sentiment score and send the result to the device.

[0146] Displaying results: The device displays the NPS on the dashboard, which administrators can use to improve services.

[0147] Example 2: Customer service at the hotel front desk

[0148] Data collection: The device records the customer and staff interaction during check-in using its built-in microphone and camera.

[0149] Data transmission: The recorded data is securely transmitted to the server.

[0150] Sentiment Analysis: The server analyzes the audio and video data and uses a generative AI model to identify the customer's sentiment score.

[0151] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0152] Results display: The device displays the NPS on a dashboard, which the hotel manager can use as feedback.

[0153] This invention makes it possible to precisely understand customer emotions based on audio and video analysis, more accurately reflecting NPS and improving service quality. Customer satisfaction can be specifically and rationally evaluated, and areas for improvement can be accurately identified.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Program processing steps

[0156] Step 1: Data collection phase

[0157] Input: Customer audio and video data

[0158] Output: Collected audio and video data

[0159] Description: As a front-end device for customer service, the terminal uses a built-in microphone (e.g., a high-sensitivity microphone) and camera (e.g., an HD camera) to collect audio and video data in real time during conversations with customers. When the customer starts speaking, the terminal automatically starts recording and saving the audio data as a digital audio signal and the video data as a digital video signal.

[0160] Step 2: Data transmission phase

[0161] Input: Collected audio and video data

[0162] Output: Audio and video data sent to the server

[0163] Description: The device packetizes collected audio and video data and sends it to the server via a secure protocol (e.g., SSL / TLS). Specifically, it buffers and timestams the data, consolidates it, encodes it, and transmits it using Internet Protocol.

[0164] Step 3: Data Analysis Phase (Audio)

[0165] Input: Audio data sent to the server

[0166] Output: Clear audio data after noise reduction

[0167] Description: The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data. Specifically, it analyzes the audio spectrum using an FFT (Fast Fourier Transform), attenuates noise components, and generates audio data with improved voice clarity.

[0168] Step 4: Data analysis phase (video)

[0169] Input: Video data sent to the server

[0170] Output: Facial video data with extracted features

[0171] Description: The server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and lists the obtained feature points.

[0172] Step 5: Emotion Identification Phase

[0173] Input: Noise-reduced audio data and facial video data with extracted features

[0174] Output: Customer sentiment data

[0175] Description: The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) and identifies the customer's emotions using prompts. Specific prompts include, "Please identify the customer's emotions from this audio data" or "Please analyze the customer's facial expressions from this video and identify their emotions." The generative AI model identifies emotions based on the prompts and outputs emotional data such as "excited," "calm," "anxious," "smiling," "surprised," and "anger."

[0176] Step 6: Sentiment score and NPS calculation phase

[0177] Input: Customer sentiment data

[0178] Output: Sentiment score and adjusted NPS

[0179] Description: The server quantifies the identified emotions and expresses them as an emotion score. Based on this emotion score, the NPS is calculated and statistical methods (e.g., Z-score or standard deviation) are used to adjust for bias in intermediate responses and eliminate the tendency for responses to be centered, which is unique to Japan. As a result, a corrected NPS is calculated.

[0180] Step 7: Debriefing Phase

[0181] Inputs: Adjusted NPS, Sentiment Score

[0182] Output: A dashboard display of visualized NPS and sentiment scores

[0183] Description: The server sends the calculated NPS to the terminal and displays it to the user in the form of a dashboard or report. Specifically, the web server dynamically generates HTML and JavaScript to display it in the form of graphs and charts, allowing the user to intuitively understand the information. Based on this, the user can consider ways to improve the service and take specific actions.

[0184] (Application example 1)

[0185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0186] Conventional systems for evaluating customer satisfaction (NPS) have difficulty accurately grasping customer satisfaction levels because they are unable to properly eliminate the tendency for Japanese customers to center their responses. Furthermore, the lack of a system for analyzing conversations in physical stores in real time and providing immediate feedback makes it difficult to quickly improve services.

[0187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0188] In this invention, the server includes a means for collecting voice data and video data during a conversation with a customer, a means for using a generative AI model to analyze the collected voice data and video data to identify the customer's emotions, a means for calculating a customer's emotion score based on the analysis results, correcting the score to eliminate response centralization, and calculating an accurate customer satisfaction level, and a data collection means for evaluating the conversation between the customer and an employee in a physical store. This allows for an accurate NPS to be calculated immediately based on the customer's emotion evaluation, enabling prompt service improvements in the physical store.

[0189] "Customer" means a person to whom a service is provided, who purchases or uses goods or services.

[0190] "Auditory data" is information collected as sound, and is digital data of sound acquired by a sensor such as a microphone.

[0191] "Video data" refers to image data collected by visual sensors such as video cameras and cameras, and is composed of a series of still images divided into frames.

[0192] A "generative AI model" is an artificial intelligence model that has been trained using a large amount of learning data and has the ability to generate responses based on specific input data.

[0193] The "emotion score" is an index that numerically expresses the emotional state of the customer based on the analysis results of the audio data and video data.

[0194] "Response centralization" refers to the tendency for responses to be biased toward the middle of the evaluation scale in certain cultures and response patterns.

[0195] A "brick and mortar store" is a location that provides goods or services in person, either within a physical building or other location.

[0196] A system for implementing the present invention has a function for identifying customer emotions using audio and video data in order to accurately calculate customer satisfaction (NPS). The system includes the following components.

[0197] Hardware and Software Configuration

[0198] 1. Device:

[0199] To collect audio and video data, devices such as smartphones and smart glasses are used, which have built-in microphones and cameras and can collect data in real time.

[0200] 2. Server:

[0201] To analyze the collected audio and video data, a high-performance computer server is used. The server is equipped with the voice recognition library "Vosk" and the image recognition library "OpenCV," and these are used to analyze the data.

[0202] Emotion identification uses a generative AI model: a RoBERTa-based emotion classification model using the Hugging Face transforms library.

[0203] Data collection and transmission

[0204] The device collects audio and video data in real time during customer interactions, and the collected data is sent to the server via a secure protocol, such as Transport Layer Security (TLS), to protect the integrity and privacy of the data.

[0205] Data analysis

[0206] The server parses the data as follows:

[0207] Audio data analysis:

[0208] Apply noise reduction filters to get clear audio data. Filter to remove background noise.

[0209] It uses a speech recognition algorithm to convert the voice data into text, analyzes what the customer is saying, extracts features such as tone, pitch, and speed, and evaluates the sentiment.

[0210] Video data analysis:

[0211] A facial recognition algorithm is used to detect the customer's face from the video data, extracting facial features (e.g., eye and mouth movements, wrinkles, etc.) to identify emotions.

[0212] Sentiment score and NPS calculation

[0213] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The identified emotions are quantified and expressed as an emotion score. The NPS is calculated based on the emotion score and adjusted using statistical methods to eliminate the tendency for response centering, which is unique to Japan.

[0214] Reporting the results

[0215] The calculated NPS is returned to the device and displayed in the form of a dashboard and report. The dashboard visually displays the history of sentiment scores and detailed analysis results, allowing users to immediately identify problems and use the information to improve their services.

[0216] Specific examples

[0217] Example 1: Evaluation of cashier service at a physical store

[0218] Data collection: Smart glasses record the conversations and facial expressions between store staff and customers.

[0219] Data transmission: The recorded data is transmitted to the server via a secure protocol.

[0220] Sentiment analysis: The server analyzes voice tone and facial expressions to identify the customer's emotions.

[0221] NPS calculation: Calculates the adjusted NPS from the sentiment score and displays the results in real time.

[0222] View the results: See your NPS score displayed on the dashboard and take action to improve your customer experience.

[0223] Example prompts to input to the generative AI model

[0224] "Please identify customer emotions and calculate NPS based on audio and video data recorded from conversations between customers and store staff in-store. Specifically, I would like to analyze the tone of the voice and changes in facial expressions in the video to identify emotions such as 'excited' or 'calm'."

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] Step 1:

[0227] The device collects audio and video data during interactions with customers. The input is the interaction between the customer and the store clerk in the physical store, and the output is the collected audio file (e.g., customer_interaction.wav) and video file (e.g., customer_interaction.mp4). Specifically, the smart glasses' built-in microphone and camera are used to record the conversation between the customer and the store clerk, as well as their facial expressions.

[0228] Step 2:

[0229] The audio and video data collected by the device is sent to the server via a secure protocol such as TLS (Transport Layer Security). The input is the audio and video files collected in step 1, and the output is these data transferred to the server. Specifically, when sending data from the device to the server, an encrypted connection is maintained to protect the integrity and privacy of the data.

[0230] Step 3:

[0231] The server applies a noise reduction filter to the received audio data to generate clear audio data. The input is the transferred audio file, and the output is the audio data with reduced noise. Specifically, it uses the "Vosk" speech recognition library and applies a specific filtering algorithm to remove background noise.

[0232] Step 4:

[0233] The server analyzes the noise-reduced speech data and extracts features such as speech tone, pitch, and speed. The input is the noise-reduced speech data, and the output is the extracted speech features. Specifically, a speech signal processing library is used to calculate speech parameters that indicate customer emotions.

[0234] Step 5:

[0235] The server uses a facial recognition algorithm to detect customer faces for each frame of the video data it receives and extracts facial expression features. The input is the video file, and the output is the detected face image and its facial expression features. Specifically, it uses OpenCV to extract face regions from the video frames and classify each facial expression (e.g., smile, surprise, anger).

[0236] Step 6:

[0237] The server inputs the analyzed audio and video data into a generative AI model to identify the customer's emotions. The inputs are audio features and video features, and the output is an identified emotion score. Specifically, a RoBERTa-based emotion classification model using the Hugging Face transformers library is used to predict the customer's emotional state from each feature.

[0238] Step 7:

[0239] The server quantifies the identified emotion scores and performs statistical corrections to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the identified emotion score, and the output is the corrected NPS score. Specifically, statistical methods are used to adjust responses that are biased toward the median value, resulting in a more accurate NPS.

[0240] Step 8:

[0241] The server sends the calculated NPS score to the terminal and displays the results in the form of a dashboard or report. The input is the corrected NPS score, and the output is the NPS score displayed on the terminal and its detailed analysis results. Specifically, the server provides a GUI (graphical user interface) to visually display the NPS score and its analysis results.

[0242] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0243] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the response-centered tendency that is unique to Japanese people. Furthermore, by incorporating an emotion engine that recognizes user emotions in real time, this system evaluates customer satisfaction with even greater precision. This system collects audio and video data during customer interactions, analyzes the collected data using a generative AI model to identify customer emotions, and calculates NPS based on the results. Furthermore, by recognizing user emotions in real time through the emotion engine and storing them as emotion data, further analysis and service improvement are possible.

[0244] System Overview

[0245] Terminal: A front-end device that handles customer service, it collects audio and video data and has a built-in emotion engine that recognizes user emotion data in real time.

[0246] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[0247] User: Operates the system, understands customer satisfaction, and improves services.

[0248] Program processing explanation

[0249] 1. Data Collection Phase

[0250] During a conversation with a customer, the device uses a built-in microphone and camera to collect real-time audio and video data, and the camera records the customer's facial expressions frame by frame in high resolution.

[0251] The emotion engine recognizes the user's emotions in real time and stores them as emotion data, which is based on information obtained from voice tone and facial expressions.

[0252] 2. Data transmission phase

[0253] The collected audio, video, and emotion data is sent from the device to a server, and data transmission is always via a secure protocol to protect data integrity and privacy.

[0254] 3. Data analysis phase

[0255] The server applies a noise reduction filter to the received audio data to make the audio clearer, for example by removing background noise to make the conversation clearer.

[0256] For video data, a facial recognition algorithm is used to detect the customer's face in each frame and extract facial features, capturing subtle facial features such as eye movements and changes in the mouth.

[0257] 4. Emotion Identification Phase

[0258] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained using a large amount of past data sets, enabling highly accurate emotion identification.

[0259] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "joy" or "anger."

[0260] The video data is used to evaluate changes in facial expressions and classify emotions such as smiles, surprise, and anger.

[0261] The accuracy of emotion identification is improved by taking into account user emotion data obtained from the emotion engine.

[0262] 5. Sentiment score and NPS calculation phase

[0263] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[0264] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[0265] 6. Reporting Phase

[0266] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard also displays the sentiment score history and detailed analysis results, allowing users to quickly identify problems and take measures to improve.

[0267] Specific examples

[0268] Example 1: Calculating NPS in a call center

[0269] Data collection: The terminal collects the voices of the operator and the customer during the call, records the operator's facial expressions with a camera, and the emotion engine recognizes the operator's emotional data in real time.

[0270] Data transmission: These data are transmitted to the server in real time.

[0271] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[0272] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0273] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[0274] Example 2: Customer service at the hotel front desk

[0275] Data collection: The terminal records audio and video of the interaction between the customer and staff during check-in. The emotion engine also recognizes the staff's emotions in real time and stores them as emotion data.

[0276] Data transmission: Send the recorded data to the server.

[0277] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[0278] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[0279] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[0280] This is a groundbreaking system that precisely understands customer emotions based on audio and video analysis and accurately reflects NPS to improve service quality. Furthermore, by grasping user emotions in real time through the emotion engine, even more advanced analysis and feedback become possible. In this way, it is possible to specifically and rationally evaluate customer satisfaction and accurately identify areas for improvement.

[0281] The processing flow will be explained below.

[0282] Step 1: Collect audio and video data

[0283] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with the customer.

[0284] Specifically, a microphone collects the conversational audio, while a camera records the customer's face frame by frame in high resolution.

[0285] Step 2: Real-time emotion recognition

[0286] The device's built-in emotion engine analyzes collected audio and video data in real time to recognize the user's emotions.

[0287] The system analyzes voice characteristics such as tone, pitch, and speed from audio data, and detects facial expressions and subtle movements from video data.

[0288] Step 3: Storing emotion data

[0289] The emotion engine recognizes emotion data in real time and stores it in internal storage, which is later used for detailed analysis.

[0290] The data stored includes the type of emotion recognized (e.g., joy, surprise, anger), intensity, and time stamp.

[0291] Step 4: Sending data to the server

[0292] The terminal transmits the collected audio data, video data, and stored emotion data all at once to the server.

[0293] The data is divided into data packets and transferred via a secure protocol.

[0294] Step 5: Receiving and Preprocessing Data

[0295] The server receives the data packets sent from the terminal.

[0296] It checks the integrity of the data packets and merges the audio and video data into their original format.

[0297] For audio data, a noise reduction filter is applied to remove background noise, and for video data, face detection and tracking is performed for each frame.

[0298] Step 6: Audio data analysis

[0299] The server inputs the voice data into a speech recognition model and converts it into text.

[0300] It extracts features such as voice tone, pitch, and speed, and uses a generative AI model to identify emotions.

[0301] Step 7: Video data analysis

[0302] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[0303] Emotions are classified by analyzing features such as eye movements, changes in the mouth, and facial expressions.

[0304] Step 8: Aggregating and calculating sentiment scores

[0305] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[0306] Each feature is weighted and an overall evaluation is performed.

[0307] Step 9: Calculate your NPS

[0308] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the sentiment score.

[0309] Based on the sentiment score, we calculate an accurate NPS and apply corrections to eliminate the tendency for responses to be centered, which is unique to Japan.

[0310] Step 10: Reporting results

[0311] The server sends the calculated NPS value and related analytical data to the device.

[0312] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[0313] Specific examples

[0314] Example 1: Calculating NPS in a call center

[0315] Step 1: The device collects the voice of the operator and the customer during the call, and records the operator's facial expressions with a camera.

[0316] Step 2: The emotion engine analyzes the operator's voice and facial expressions in real time to recognize emotions.

[0317] Step 3: The device stores the recognized emotion data in its internal storage.

[0318] Step 4: Send the collected data to the server.

[0319] Step 5: The server receives the data and performs audio and video preprocessing.

[0320] Step 6: Analyze the audio data and identify emotions.

[0321] Step 7: Analyze the video data and identify emotions from facial expressions.

[0322] Step 8: Combine the sentiment scores to generate a single overall sentiment score.

[0323] Step 9: Calculate the adjusted NPS based on the sentiment score.

[0324] Step 10: The device displays the NPS on the dashboard, which administrators can use to improve services.

[0325] Example 2: Customer service at the hotel front desk

[0326] Step 1: The device records audio and video of the customer and staff interaction during check-in.

[0327] Step 2: The emotion engine recognizes staff emotions in real time and stores the emotion data.

[0328] Step 3: Send the collected data to the server.

[0329] Step 4: The server receives the data and performs preprocessing.

[0330] Step 5: Noise reduction and analysis of the audio data.

[0331] Step 6: Analyze video data to identify customer sentiment.

[0332] Step 7: Combine the emotion scores to make an overall rating.

[0333] Step 8: Calculate the adjusted NPS from the sentiment score.

[0334] Step 9: The device displays the NPS, which the hotel manager uses as feedback.

[0335] Example 2

[0336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Previous systems have made it difficult to accurately assess customer satisfaction (NPS). In particular, it was difficult to quickly and accurately analyze audio and video data, and the tendency for response centering, which is unique to Japan, impacted NPS. Furthermore, it was inadequate to recognize customer sentiment in real time and immediately incorporate that data into analysis. This made it difficult to accurately grasp true customer satisfaction and link it to service improvements.

[0338] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for encrypting collected audio data and video data and transmitting them to the server using a secure protocol, means for performing noise reduction and audio feature extraction processing on the audio data in the server, means for performing facial recognition and extracting facial expression features on the video data in the server, and means for recognizing user emotions in real time using an emotion engine and storing them as emotion data. This enables secure processing of audio data and video data in real time and highly accurate identification of customer emotions. Furthermore, it enables accurate calculation of NPS, eliminating the tendency toward response centering that is unique to Japan, enabling rapid and accurate service improvements.

[0339] "Customer Satisfaction (NPS)" is an index that measures a customer's willingness to recommend a product or service to others, and is also known as the Net Promoter Score.

[0340] "Voice Data" refers to sound information collected using a microphone during a customer interaction.

[0341] "Video Data" refers to image information collected using a camera during an interaction with a customer.

[0342] A "generative AI model" refers to an artificial intelligence algorithm that is trained on large datasets and uses audio and video data to analyze customer sentiment.

[0343] An "emotion engine" refers to a software component that recognizes emotions in real time from voice tone and facial expressions.

[0344] "Emotion score" refers to an indicator that quantifies and objectively evaluates customer emotions.

[0345] "Correction to eliminate the tendency to center responses" refers to the process of using statistical methods to correct responses that tend to be biased toward intermediate evaluations, which are unique to Japan.

[0346] "Noise reduction" refers to the process of removing unnecessary background sounds from audio data and processing it into clear audio.

[0347] "Audio features" refers to attribute information necessary for analysis, such as tone, pitch, and speed, extracted from audio data.

[0348] "Facial recognition" refers to the technology of detecting and identifying human faces from video data frame by frame.

[0349] "Facial expression features" refer to attribute information such as eye movements and changes in the mouth that are extracted from video data to capture changes in facial expressions.

[0350] A "secure protocol" refers to a communication protocol that implements security measures such as encryption during data communication to ensure data protection.

[0351] The present invention relates to a system for calculating customer satisfaction (NPS) more accurately and efficiently. Specific embodiments of this system will be described below.

[0352] Hardware and software used

[0353] The device is equipped with a built-in microphone and camera to collect audio and video data during interactions with customers. It also has an emotion engine for real-time emotion recognition. The device encrypts the collected data and transmits it to a server using a secure protocol.

[0354] The server runs a generative AI model to analyze the received audio and video data. The analysis process involves noise reduction of the audio data, extraction of audio features, and facial recognition and extraction of facial features of the video data. Based on the generated emotion data, calculations are performed to calculate the emotion score and NPS.

[0355] Specific processing of the system

[0356] 1. Data Collection

[0357] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with a customer.

[0358] The device's emotion engine analyzes voice tone and facial expressions to generate and store emotion data in real time.

[0359] 2. Data Transmission

[0360] The data collected by the device is AES encrypted and sent securely to the server using the HTTPS protocol.

[0361] 3. Data Analysis

[0362] The server applies a noise reduction filter to the audio data to process it into clearer audio, and extracts audio features (e.g., tone, pitch, and speed).

[0363] The server applies a facial recognition algorithm to the video data and extracts facial features (e.g., eye movements and changes in the mouth).

[0364] 4. Emotion identification

[0365] The server uses the generative AI model to input the analyzed audio and video data to accurately identify the customer's emotions, and also integrates real-time emotional data to determine the most accurate emotional state.

[0366] 5. NPS calculation

[0367] The server quantifies the identified emotions and generates an emotion score. Based on the emotion score, the NPS is calculated and a statistical method is used to correct for the tendency for responses to be centered, which is unique to Japan.

[0368] 6.Result report

[0369] The server sends the calculated NPS to the device, which displays the results in the form of a dashboard or report.

[0370] Specific examples

[0371] Example 1: Calculating NPS in a call center

[0372] Data collection: The device collects the voices of both the operator and the customer during the call using a high-quality microphone, and records the operator's facial expressions frame by frame using a built-in camera. The emotion engine stores the operator's emotional data in real time.

[0373] Data transmission: Encrypted audio and video data is sent to the server using the HTTPS protocol.

[0374] Emotion analysis: The server inputs noise-reduced audio data into a generative AI model to identify emotions. Facial features are extracted and classified from video data.

[0375] NPS calculation: Calculate the corrected NPS based on the sentiment score and send the result to the device.

[0376] Displaying results: The device displays NPS and detailed analysis results on a dashboard, which administrators can use to improve services.

[0377] Example prompt sentence:

[0378] Please explain in detail the processing flow of a system that collects conversation data between agents and customers in a call center, analyzes agent emotions in real time using an emotion engine, and calculates NPS, including the specific operation of each processing step.

[0379] Example 2: Customer service at the hotel front desk

[0380] Data collection: The device records the interaction between the customer and the staff at check-in. The microphone collects audio data, and the camera collects high-resolution video data. The emotion engine recognizes and stores the staff's emotions in real time.

[0381] Data transmission: Encrypted data is sent to the server using a secure protocol.

[0382] Sentiment analysis: The server analyzes the audio and video data using a generative AI model to determine an emotion score.

[0383] NPS calculation: Calculate the corrected NPS based on the sentiment score and send it to the device.

[0384] Results display: The device displays a detailed NPS report on a dashboard for hotel managers to use for feedback.

[0385] Example prompt sentence:

[0386] Please explain the system that collects real-time audio and video data of customer interactions at hotel front desks, analyzes staff emotions using an emotion engine, and calculates emotion scores and NPS, including the specific operation of each processing step.

[0387] In this way, the present invention improves service quality by precisely understanding customer emotions based on audio and video analysis and calculating an accurate NPS. Real-time emotion analysis is possible, which allows for rapid feedback and the implementation of appropriate improvement measures.

[0388] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0389] Step 1: Data collection

[0390] The device uses a built-in microphone to collect voice data in real time while interacting with the customer.

[0391] The device uses a built-in high-resolution camera to record the customer's facial expressions frame by frame.

[0392] The device's emotion engine analyzes voice tone and facial expressions in real time and stores them as emotional data.

[0393] Input: Voice recording of conversation between customer and staff, and video of customer's facial expressions.

[0394] Output: Audio and video data, real-time emotion data.

[0395] Step 2: Send data

[0396] The audio and video data collected by the device, as well as real-time emotion data, are encrypted using AES.

[0397] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[0398] Input: Audio data, video data, and emotion data before encryption.

[0399] Output: The encrypted data sent to the server.

[0400] Step 3: Noise reduction of audio data

[0401] The server applies a noise reduction filter to the audio data it receives, removing background noise and producing clear audio.

[0402] Input: Encrypted audio data.

[0403] Output: Clear audio data after noise reduction.

[0404] Step 4: Extract audio features

[0405] The server extracts speech features such as tone, pitch, and speed from the noise-reduced speech data.

[0406] Input: Noise-reduced audio data.

[0407] Output: Audio feature data.

[0408] Step 5: Facial recognition of video data

[0409] The server applies a facial recognition algorithm to the video data it receives and detects the customer's face in each frame.

[0410] Input: Encrypted video data.

[0411] Output: Face recognition results and facial expression information per frame.

[0412] Step 6: Extraction of facial features

[0413] The server extracts facial expression changes (for example, eye movements or changes in the mouth) as features from the video data that has undergone facial recognition.

[0414] Input: Face-recognized video data.

[0415] Output: Facial expression feature data.

[0416] Step 7: Identify emotions

[0417] The server inputs the analyzed audio and video data using the generative AI model to identify customer emotions.

[0418] The server integrates this with real-time emotional data to determine the most accurate emotional state.

[0419] Input: Audio feature data, facial expression feature data, real-time emotion data.

[0420] Output: Identified emotion data.

[0421] Step 8: Sentiment score and NPS calculation

[0422] The server quantifies the identified emotion and calculates it as an emotion score.

[0423] The server calculates the NPS based on this emotional score and corrects for the tendency for responses to be centered, which is unique to Japan, using statistical methods.

[0424] Input: Identified emotion data.

[0425] Output: Sentiment score and adjusted NPS.

[0426] Step 9: Report your findings

[0427] The server sends the calculated NPS to the device.

[0428] The NPS sent by the device is displayed to the user in the form of a dashboard or report.

[0429] Users use this data to evaluate customer satisfaction and consider ways to improve services.

[0430] Input: Calculated NPS.

[0431] Output: NPS and detailed analysis results displayed in dashboards and reports.

[0432] (Application example 2)

[0433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0434] Conventional systems for calculating customer satisfaction (NPS) have had difficulty eliminating the tendency for Japanese people to center their responses and accurately measure customer satisfaction. There was also a lack of means to recognize customer emotions in real time and immediately improve services. Furthermore, there were issues with the accuracy and practicality of systems that effectively collect and analyze customer voice and video data during interactions.

[0435] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data and video data during a conversation with a customer, means for analyzing the collected voice data and video data and using a generative AI model to identify the customer's emotions, means for calculating a customer's emotion score based on the analysis results and correcting it to eliminate response centralization to accurately calculate customer satisfaction, and means for performing the collection and analysis in real time via smart glasses. This makes it possible to recognize customer emotions with high accuracy in real time and accurately measure customer satisfaction. Furthermore, staff and managers can receive instant feedback and respond quickly to service improvements.

[0436] "Voice Data" refers to the audio information exchanged between a customer and a staff member during an interaction.

[0437] "Video data" refers to video information such as the faces and expressions of customers and staff during a conversation.

[0438] "Generative AI model" refers to an artificial intelligence model trained to analyze collected audio and video data and identify customer sentiment.

[0439] "Emotion score" refers to the customer's emotional state, expressed numerically as a result of analysis using a generative AI model.

[0440] "Correction to eliminate the tendency to center responses" refers to a statistical method for more accurately calculating customer satisfaction by eliminating the intermediate response tendency that is unique to Japanese people.

[0441] "Smart glasses" refers to a wearable eyeglass-type device that has the ability to collect audio and video data and analyze it in real time.

[0442] The present invention relates to a system for measuring customer satisfaction (NPS) in real time using smart glasses in a brick-and-mortar store. Specific embodiments thereof will be described below.

[0443] Hardware Configuration

[0444] The system includes smart glasses for collecting audio and video data, and a back-end server for analyzing the data. The smart glasses can be wearable devices such as Google Glass or Vuzix Blade. The back-end server has a high-performance processor and ample memory for processing and analyzing the collected data. It uses MySQL as the database and GPT-3 as the generative AI model.

[0445] Software Configuration

[0446] The following software modules are installed on the server:

[0447] Data collection module: Receives audio and video data transmitted from the smart glasses.

[0448] Noise reduction filter: Removes noise from audio data.

[0449] Audio feature extraction module: Extracts features from the audio data after noise removal.

[0450] Facial recognition module: Recognizes the customer's face for each frame of video data and extracts facial features.

[0451] Generative AI model module: A generative AI model to analyze collected data and identify customer sentiment.

[0452] Emotion score calculation module: Calculates the emotion score based on the analysis results.

[0453] NPS calculation module: Eliminates the tendency to center responses and calculates accurate customer satisfaction.

[0454] Data processing and calculation

[0455] 1. Data collection: Store staff wearing smart glasses collect audio and video data while interacting with customers. The data is sent to a server in real time.

[0456] 2. Noise reduction: Remove background noise from the audio data on the server to make the conversation clearer.

[0457] 3. Feature extraction: Extract audio features from the noise-removed audio data, and extract facial features for each frame from the video data.

[0458] 4. Emotion Identification: Using a generative AI model, the extracted feature data is analyzed to identify the customer's emotions.

[0459] 5. Emotion score calculation: The identified emotions are quantified and an emotion score is calculated.

[0460] 6. NPS calculation: NPS is calculated based on the emotional score, and the tendency for responses to be centered, which is unique to Japan, is corrected using statistical methods to calculate accurate customer satisfaction.

[0461] 7. Displaying results: The NPS is displayed on the smart glasses screen or on the store's management terminal, allowing staff and managers to receive feedback in real time.

[0462] Examples and prompts

[0463] As a concrete example, when a store staff member explains a product to a customer, audio and video data is collected and emotional data is generated in real time using an emotion engine. This data is sent from the smart glasses to a server, where it is analyzed by a generative AI model. An NPS is calculated based on the emotional score, and the results are displayed on the smart glasses. This allows staff to instantly grasp customer satisfaction levels and use the information to improve service.

[0464] Example prompt sentence:

[0465] "Using the audio and video data collected during the conversation, identify the customer's emotions and calculate the NPS. What adjustments are necessary to eliminate the tendency for responses to be centered, which is unique to Japan? Explain in detail how to calculate the emotion score and NPS."

[0466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0467] Step 1:

[0468] The terminal (smart glasses) collects audio and video data during a conversation with a customer. Specifically, audio and video information is captured in real time using the smart glasses' built-in microphone and camera. The input is the customer's audio and video during the conversation, and the output is digital audio and video files.

[0469] Step 2:

[0470] The audio and video data collected by the device is sent to the server via wireless communication (Bluetooth or Wi-Fi). The input is the collected audio and video files, and the output is the audio and video files stored on the server.

[0471] Step 3:

[0472] The server applies a noise reduction filter to the audio data it receives to process it into clear audio. Specifically, it applies an algorithm to remove background noise from the audio data. The input is the original audio data, and the output is clear audio data with noise removed.

[0473] Step 4:

[0474] The server extracts speech features from the noise-removed speech data. Specifically, it analyzes information such as tone, pitch, and speed of the speech. The input is the noise-removed speech data, and the output is the extracted speech features.

[0475] Step 5:

[0476] The server performs facial recognition and emotion classification for each frame of the video data it receives. Specifically, it uses a facial recognition algorithm to detect the customer's face and analyze their facial features. The input is the original video data, and the output is the extracted facial feature values.

[0477] Step 6:

[0478] The server inputs the extracted voice and facial features into a generative AI model to identify the customer's emotion. The input is the voice and facial features, and the output is the determined customer emotion.

[0479] Step 7:

[0480] The server quantifies the identified customer emotions and calculates an emotion score. Specifically, it assigns a certain score to each emotion based on the prediction results of the generative AI model. The input is the identified emotion, and the output is the emotion score.

[0481] Step 8:

[0482] The server calculates the NPS based on the sentiment score and corrects it using a statistical method to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the sentiment score, and the output is the corrected NPS.

[0483] Step 9:

[0484] The server transmits the calculated NPS to the device (smart glasses) via wireless communication. The input is the corrected NPS, and the output is the NPS displayed on the smart glasses' display.

[0485] Step 10:

[0486] The user (store staff) checks the NPS displayed on the smart glasses display, instantly understanding customer satisfaction and helping to improve services. The input is the NPS displayed on the display, and the output is customer satisfaction information recognized by the user.

[0487] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0488] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0489] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0490] [Second embodiment]

[0491] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0492] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0493] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0494] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0495] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0496] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0497] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0498] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0499] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0500] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0501] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0502] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0503] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[0504] System Overview

[0505] Terminal: A front-end device that handles customer service and collects audio and video data.

[0506] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[0507] User: Operates the system, understands customer satisfaction, and improves services.

[0508] Program processing explanation

[0509] 1. Data Collection Phase

[0510] During a conversation with a customer, the device uses a built-in microphone and camera to collect audio and video data in real time, for example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[0511] 2. Data transmission phase

[0512] The collected audio and video data is sent from the device to a server, and data transmission is always via a secure protocol to protect the integrity and privacy of the data.

[0513] 3. Data analysis phase

[0514] The server applies a noise reduction filter to the received audio data to make it clearer, for example, by removing background speech and other noise.

[0515] Next, a facial recognition algorithm is used to detect the customer's face from the video data and extract facial features, which can detect minute changes such as eye and mouth movements and wrinkles.

[0516] 4. Emotion Identification Phase

[0517] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained based on a large amount of past data sets, enabling highly accurate emotion identification.

[0518] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "excited" or "calm."

[0519] The system evaluates changes in facial expressions from video data and classifies emotions such as smiles, surprise, and anger with high accuracy.

[0520] 5. Sentiment score and NPS calculation phase

[0521] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[0522] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[0523] 6. Reporting Phase

[0524] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the history of sentiment scores and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[0525] Specific examples

[0526] Example 1: Calculating NPS in a call center

[0527] Data collection: The device collects the voices of both the operator and the customer during the call, and records the operator's facial expressions with a camera.

[0528] Data transmission: These data are transmitted to the server in real time.

[0529] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[0530] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0531] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[0532] Example 2: Customer service at the hotel front desk

[0533] Data collection: The device records audio and video of the customer and staff interaction during check-in.

[0534] Data transmission: Send the recorded data to the server.

[0535] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[0536] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[0537] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[0538] This is a groundbreaking system that improves service quality by precisely understanding customer emotions based on audio and video analysis and more accurately reflecting NPS. In this way, it is possible to evaluate customer satisfaction more specifically and rationally and accurately derive areas for improvement.

[0539] The processing flow will be explained below.

[0540] Step 1: Data collection

[0541] While the device is interacting with the customer, it uses its built-in microphone and camera to collect audio and video data in real time.

[0542] Specifically, a microphone captures the conversational audio, and a camera records the customer's facial expressions frame by frame.

[0543] Step 2: Send data

[0544] The terminal converts the collected audio data into a digital signal and compresses and encodes the video data frame by frame.

[0545] The encoded data is divided into data packets and sent to the server using a secure protocol.

[0546] Step 3: Data reception and preprocessing

[0547] The server receives the data packets sent from the terminal.

[0548] The integrity of the data packets is verified, and the data is consolidated and restored to its original form.

[0549] The server applies a noise reduction filter to the audio data to remove unwanted background sounds.

[0550] Video data is analyzed frame by frame to perform face detection and tracking.

[0551] Step 4: Audio data analysis

[0552] The server inputs the voice data into a speech recognition model and converts it into text.

[0553] Extract speech features such as tone, pitch, and speed.

[0554] Based on the extracted features, a generative AI model identifies emotions.

[0555] Step 5: Video data analysis

[0556] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[0557] It analyzes features such as eye movements, mouth movements, and facial expressions to classify emotions.

[0558] Step 6: Integrating sentiment scores

[0559] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[0560] Each feature is weighted and an overall evaluation is calculated.

[0561] Step 7: Calculate your NPS

[0562] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the emotional scores.

[0563] Calculate an accurate NPS based on sentiment scores and apply a correction to eliminate response centralization bias.

[0564] Step 8: Report your findings

[0565] The server sends the calculated NPS value and related analytical data to the device.

[0566] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[0567] Example 1

[0568] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Accurately calculating customer satisfaction (NPS) is essential for improving a company's services. However, due to the tendency toward response-centering that is unique to Japan, it has been difficult to accurately calculate NPS using conventional methods. Furthermore, even when identifying customer emotions using audio and video data, noise and unclear data can reduce the accuracy of the analysis, which has been a problem. Therefore, there is a strong demand for the development of a system that can analyze customer audio and video data to accurately identify emotions and calculate an adjusted NPS based on the results.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0571] In this invention, the server includes means for transmitting collected audio and video data using an Internet Protocol, means for applying a noise reduction filter to the received audio data, means for detecting a customer's face from the video data using a facial recognition algorithm and extracting facial expression features, means for inputting the analyzed audio and video data into a generative AI model and identifying the customer's emotion using a prompt sentence, means for quantifying the identified emotion and expressing it as an emotion score, means for calculating customer satisfaction based on the emotion score and correcting it to eliminate answer centralization tendency, and means for generating a dashboard for transmitting the calculated customer satisfaction to a terminal and visualizing it. This improves the accuracy of analysis of audio and video data, making it possible to accurately identify customer emotion and calculate a corrected NPS.

[0572] "Voice data" refers to recordings of speech collected during customer interactions and is a digital audio signal captured using a microphone.

[0573] "Video Data" refers to video recordings collected during interactions with customers and is a digital video signal captured using a camera.

[0574] The "Internet Protocol" is a set of protocols for sending and receiving data over a network, and defines how data is packetized and transmitted.

[0575] A "noise reduction filter" is an algorithm or software process used to remove unwanted noise and interference from audio data.

[0576] A "face recognition algorithm" is a computational method for detecting a person's face from video data and extracting its features.

[0577] A "generative AI model" is an artificial intelligence algorithm that is trained on large datasets to generate specific information from audio or video data.

[0578] A "prompt" is an instruction or question that is input into a generative AI model to enable it to perform a specific task.

[0579] An "emotion score" is a numerical representation of the emotion identified from the analyzed audio and video data.

[0580] "Customer satisfaction" is an indicator of how customers rate their satisfaction with their experiences and services, and includes the Net Promoter Score (NPS).

[0581] "Response centralization" is the tendency of respondents in a particular cultural context to choose the middle or average answer.

[0582] A "dashboard" is an interface for centrally displaying data visualization and analysis results.

[0583] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[0584] System Overview

[0585] Terminal: A front-end device that handles customer service and collects audio and video data. Specific hardware features include high-sensitivity microphones and HD cameras. Examples include the Bose NC700 and Logitech C920.

[0586] Server: A backend device that analyzes data, calculates sentiment scores, and calculates NPS. The server has advanced computing power and large data storage capacity.

[0587] Users: Operate the system, understand customer satisfaction, and improve services. Users can use the dashboard to visually check the analysis results.

[0588] Program processing explanation

[0589] 1. Data Collection Phase

[0590] While the terminal is interacting with the customer, it uses a built-in microphone (e.g., a high-sensitivity microphone) and a camera (e.g., an HD camera) to collect audio and video data in real time. For example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[0591] 2. Data transmission phase

[0592] The collected audio and video data is sent from the device to the server. Data transmission is via a secure protocol (e.g., SSL / TLS) to protect the integrity and privacy of the data.

[0593] 3. Data analysis phase

[0594] The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data to remove background noise and unwanted audio components. Specifically, the server analyzes the audio spectrum using an FFT (Fast Fourier Transform) and attenuates the noise components.

[0595] Next, the server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and creates a list of feature points.

[0596] 4. Emotion Identification Phase

[0597] The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) to identify the customer's emotions. For example, for audio data, a request is sent along with the prompt, "Please identify the customer's emotions from this audio data." The returned output includes emotions such as "elated," "calm," and "anxious."

[0598] For video data, the prompt is, "Analyze the customer's facial expressions in this video and identify their emotions." The model's output includes emotions such as "smile," "surprise," and "anger."

[0599] 5. Sentiment score and NPS calculation phase

[0600] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience. The NPS is calculated based on the emotion score, and by adjusting for the bias of intermediate responses using statistical methods, the tendency to center responses, which is unique to Japan, can be eliminated.

[0601] 6. Reporting Phase

[0602] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the sentiment score history and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[0603] Specific examples

[0604] Example 1: Calculating NPS in a call center

[0605] Data collection: The device collects the voices of both the operator and the customer during the call using a built-in microphone, and records the operator's facial expressions using a camera.

[0606] Data transmission: Collected data is transmitted to the server in real time.

[0607] Emotion analysis: The server analyzes the operator's tone and pitch from the voice data and extracts changes in the operator's facial expressions from the video data.

[0608] Emotion Identification: The server identifies emotions using a generative AI model.

[0609] NPS calculation: Calculate the adjusted NPS based on the sentiment score and send the result to the device.

[0610] Displaying results: The device displays the NPS on the dashboard, which administrators can use to improve services.

[0611] Example 2: Customer service at the hotel front desk

[0612] Data collection: The device records the customer and staff interaction during check-in using its built-in microphone and camera.

[0613] Data transmission: The recorded data is securely transmitted to the server.

[0614] Sentiment Analysis: The server analyzes the audio and video data and uses a generative AI model to identify the customer's sentiment score.

[0615] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0616] Results display: The device displays the NPS on a dashboard, which the hotel manager can use as feedback.

[0617] This invention makes it possible to precisely understand customer emotions based on audio and video analysis, more accurately reflecting NPS and improving service quality. Customer satisfaction can be specifically and rationally evaluated, and areas for improvement can be accurately identified.

[0618] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0619] Program processing steps

[0620] Step 1: Data collection phase

[0621] Input: Customer audio and video data

[0622] Output: Collected audio and video data

[0623] Description: As a front-end device for customer service, the terminal uses a built-in microphone (e.g., a high-sensitivity microphone) and camera (e.g., an HD camera) to collect audio and video data in real time during conversations with customers. When the customer starts speaking, the terminal automatically starts recording and saving the audio data as a digital audio signal and the video data as a digital video signal.

[0624] Step 2: Data transmission phase

[0625] Input: Collected audio and video data

[0626] Output: Audio and video data sent to the server

[0627] Description: The device packetizes collected audio and video data and sends it to the server via a secure protocol (e.g., SSL / TLS). Specifically, it buffers and timestams the data, consolidates it, encodes it, and transmits it using Internet Protocol.

[0628] Step 3: Data Analysis Phase (Audio)

[0629] Input: Audio data sent to the server

[0630] Output: Clear audio data after noise reduction

[0631] Description: The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data. Specifically, it analyzes the audio spectrum using an FFT (Fast Fourier Transform), attenuates noise components, and generates audio data with improved voice clarity.

[0632] Step 4: Data analysis phase (video)

[0633] Input: Video data sent to the server

[0634] Output: Facial video data with extracted features

[0635] Description: The server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and lists the obtained feature points.

[0636] Step 5: Emotion Identification Phase

[0637] Input: Noise-reduced audio data and facial video data with extracted features

[0638] Output: Customer sentiment data

[0639] Description: The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) and identifies the customer's emotions using prompts. Specific prompts include, "Please identify the customer's emotions from this audio data" or "Please analyze the customer's facial expressions from this video and identify their emotions." The generative AI model identifies emotions based on the prompts and outputs emotional data such as "excited," "calm," "anxious," "smiling," "surprised," and "anger."

[0640] Step 6: Sentiment score and NPS calculation phase

[0641] Input: Customer sentiment data

[0642] Output: Sentiment score and adjusted NPS

[0643] Description: The server quantifies the identified emotions and expresses them as an emotion score. Based on this emotion score, the NPS is calculated and statistical methods (e.g., Z-score or standard deviation) are used to adjust for bias in intermediate responses and eliminate the tendency for responses to be centered, which is unique to Japan. As a result, a corrected NPS is calculated.

[0644] Step 7: Debriefing Phase

[0645] Inputs: Adjusted NPS, Sentiment Score

[0646] Output: A dashboard display of visualized NPS and sentiment scores

[0647] Description: The server sends the calculated NPS to the terminal and displays it to the user in the form of a dashboard or report. Specifically, the web server dynamically generates HTML and JavaScript to display it in the form of graphs and charts, allowing the user to intuitively understand the information. Based on this, the user can consider ways to improve the service and take specific actions.

[0648] (Application example 1)

[0649] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Conventional systems for evaluating customer satisfaction (NPS) have difficulty accurately grasping customer satisfaction levels because they are unable to properly eliminate the tendency for Japanese customers to center their responses. Furthermore, the lack of a system for analyzing conversations in physical stores in real time and providing immediate feedback makes it difficult to quickly improve services.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0652] In this invention, the server includes a means for collecting voice data and video data during a conversation with a customer, a means for using a generative AI model to analyze the collected voice data and video data to identify the customer's emotions, a means for calculating a customer's emotion score based on the analysis results, correcting the score to eliminate response centralization, and calculating an accurate customer satisfaction level, and a data collection means for evaluating the conversation between the customer and an employee in a physical store. This allows for an accurate NPS to be calculated immediately based on the customer's emotion evaluation, enabling prompt service improvements in the physical store.

[0653] "Customer" means a person to whom a service is provided, who purchases or uses goods or services.

[0654] "Auditory data" is information collected as sound, and is digital data of sound acquired by a sensor such as a microphone.

[0655] "Video data" refers to image data collected by visual sensors such as video cameras and cameras, and is composed of a series of still images divided into frames.

[0656] A "generative AI model" is an artificial intelligence model that has been trained using a large amount of learning data and has the ability to generate responses based on specific input data.

[0657] The "emotion score" is an index that numerically expresses the emotional state of the customer based on the analysis results of the audio data and video data.

[0658] "Response centralization" refers to the tendency for responses to be biased toward the middle of the evaluation scale in certain cultures and response patterns.

[0659] A "brick and mortar store" is a location that provides goods or services in person, either within a physical building or other location.

[0660] A system for implementing the present invention has a function for identifying customer emotions using audio and video data in order to accurately calculate customer satisfaction (NPS). The system includes the following components.

[0661] Hardware and Software Configuration

[0662] 1. Device:

[0663] To collect audio and video data, devices such as smartphones and smart glasses are used, which have built-in microphones and cameras and can collect data in real time.

[0664] 2. Server:

[0665] To analyze the collected audio and video data, a high-performance computer server is used. The server is equipped with the voice recognition library "Vosk" and the image recognition library "OpenCV," and these are used to analyze the data.

[0666] Emotion identification uses a generative AI model: a RoBERTa-based emotion classification model using the Hugging Face transforms library.

[0667] Data collection and transmission

[0668] The device collects audio and video data in real time during customer interactions, and the collected data is sent to the server via a secure protocol, such as Transport Layer Security (TLS), to protect the integrity and privacy of the data.

[0669] Data analysis

[0670] The server parses the data as follows:

[0671] Audio data analysis:

[0672] Apply noise reduction filters to get clear audio data. Filter to remove background noise.

[0673] It uses a speech recognition algorithm to convert the voice data into text, analyzes what the customer is saying, extracts features such as tone, pitch, and speed, and evaluates the sentiment.

[0674] Video data analysis:

[0675] A facial recognition algorithm is used to detect the customer's face from the video data, extracting facial features (e.g., eye and mouth movements, wrinkles, etc.) to identify emotions.

[0676] Sentiment score and NPS calculation

[0677] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The identified emotions are quantified and expressed as an emotion score. The NPS is calculated based on the emotion score and adjusted using statistical methods to eliminate the tendency for response centering, which is unique to Japan.

[0678] Reporting the results

[0679] The calculated NPS is returned to the device and displayed in the form of a dashboard and report. The dashboard visually displays the history of sentiment scores and detailed analysis results, allowing users to immediately identify problems and use the information to improve their services.

[0680] Specific examples

[0681] Example 1: Evaluation of cashier service at a physical store

[0682] Data collection: Smart glasses record the conversations and facial expressions between store staff and customers.

[0683] Data transmission: The recorded data is transmitted to the server via a secure protocol.

[0684] Sentiment analysis: The server analyzes voice tone and facial expressions to identify the customer's emotions.

[0685] NPS calculation: Calculates the adjusted NPS from the sentiment score and displays the results in real time.

[0686] View the results: See your NPS score displayed on the dashboard and take action to improve your customer experience.

[0687] Example prompts to input to the generative AI model

[0688] "Please identify customer emotions and calculate NPS based on audio and video data recorded from conversations between customers and store staff in-store. Specifically, I would like to analyze the tone of the voice and changes in facial expressions in the video to identify emotions such as 'excited' or 'calm'."

[0689] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0690] Step 1:

[0691] The device collects audio and video data during interactions with customers. The input is the interaction between the customer and the store clerk in the physical store, and the output is the collected audio file (e.g., customer_interaction.wav) and video file (e.g., customer_interaction.mp4). Specifically, the smart glasses' built-in microphone and camera are used to record the conversation between the customer and the store clerk, as well as their facial expressions.

[0692] Step 2:

[0693] The audio and video data collected by the device is sent to the server via a secure protocol such as TLS (Transport Layer Security). The input is the audio and video files collected in step 1, and the output is these data transferred to the server. Specifically, when sending data from the device to the server, an encrypted connection is maintained to protect the integrity and privacy of the data.

[0694] Step 3:

[0695] The server applies a noise reduction filter to the received audio data to generate clear audio data. The input is the transferred audio file, and the output is the audio data with reduced noise. Specifically, it uses the "Vosk" speech recognition library and applies a specific filtering algorithm to remove background noise.

[0696] Step 4:

[0697] The server analyzes the noise-reduced speech data and extracts features such as speech tone, pitch, and speed. The input is the noise-reduced speech data, and the output is the extracted speech features. Specifically, a speech signal processing library is used to calculate speech parameters that indicate customer emotions.

[0698] Step 5:

[0699] The server uses a facial recognition algorithm to detect customer faces for each frame of the video data it receives and extracts facial expression features. The input is the video file, and the output is the detected face image and its facial expression features. Specifically, it uses OpenCV to extract face regions from the video frames and classify each facial expression (e.g., smile, surprise, anger).

[0700] Step 6:

[0701] The server inputs the analyzed audio and video data into a generative AI model to identify the customer's emotions. The inputs are audio features and video features, and the output is an identified emotion score. Specifically, a RoBERTa-based emotion classification model using the Hugging Face transformers library is used to predict the customer's emotional state from each feature.

[0702] Step 7:

[0703] The server quantifies the identified emotion scores and performs statistical corrections to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the identified emotion score, and the output is the corrected NPS score. Specifically, statistical methods are used to adjust responses that are biased toward the median value, resulting in a more accurate NPS.

[0704] Step 8:

[0705] The server sends the calculated NPS score to the terminal and displays the results in the form of a dashboard or report. The input is the corrected NPS score, and the output is the NPS score displayed on the terminal and its detailed analysis results. Specifically, the server provides a GUI (graphical user interface) to visually display the NPS score and its analysis results.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the response-centered tendency that is unique to Japanese people. Furthermore, by incorporating an emotion engine that recognizes user emotions in real time, this system evaluates customer satisfaction with even greater precision. This system collects audio and video data during customer interactions, analyzes the collected data using a generative AI model to identify customer emotions, and calculates NPS based on the results. Furthermore, by recognizing user emotions in real time through the emotion engine and storing them as emotion data, further analysis and service improvement are possible.

[0708] System Overview

[0709] Terminal: A front-end device that handles customer service, it collects audio and video data and has a built-in emotion engine that recognizes user emotion data in real time.

[0710] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[0711] User: Operates the system, understands customer satisfaction, and improves services.

[0712] Program processing explanation

[0713] 1. Data Collection Phase

[0714] During a conversation with a customer, the device uses a built-in microphone and camera to collect real-time audio and video data, and the camera records the customer's facial expressions frame by frame in high resolution.

[0715] The emotion engine recognizes the user's emotions in real time and stores them as emotion data, which is based on information obtained from voice tone and facial expressions.

[0716] 2. Data transmission phase

[0717] The collected audio, video, and emotion data is sent from the device to a server, and data transmission is always via a secure protocol to protect data integrity and privacy.

[0718] 3. Data analysis phase

[0719] The server applies a noise reduction filter to the received audio data to make the audio clearer, for example by removing background noise to make the conversation clearer.

[0720] For video data, a facial recognition algorithm is used to detect the customer's face in each frame and extract facial features, capturing subtle facial features such as eye movements and changes in the mouth.

[0721] 4. Emotion Identification Phase

[0722] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained using a large amount of past data sets, enabling highly accurate emotion identification.

[0723] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "joy" or "anger."

[0724] The video data is used to evaluate changes in facial expressions and classify emotions such as smiles, surprise, and anger.

[0725] The accuracy of emotion identification is improved by taking into account user emotion data obtained from the emotion engine.

[0726] 5. Sentiment score and NPS calculation phase

[0727] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[0728] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[0729] 6. Reporting Phase

[0730] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard also displays the sentiment score history and detailed analysis results, allowing users to quickly identify problems and take measures to improve.

[0731] Specific examples

[0732] Example 1: Calculating NPS in a call center

[0733] Data collection: The terminal collects the voices of the operator and the customer during the call, records the operator's facial expressions with a camera, and the emotion engine recognizes the operator's emotional data in real time.

[0734] Data transmission: These data are transmitted to the server in real time.

[0735] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[0736] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0737] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[0738] Example 2: Customer service at the hotel front desk

[0739] Data collection: The terminal records audio and video of the interaction between the customer and staff during check-in. The emotion engine also recognizes the staff's emotions in real time and stores them as emotion data.

[0740] Data transmission: Send the recorded data to the server.

[0741] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[0742] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[0743] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[0744] This is a groundbreaking system that precisely understands customer emotions based on audio and video analysis and accurately reflects NPS to improve service quality. Furthermore, by grasping user emotions in real time through the emotion engine, even more advanced analysis and feedback become possible. In this way, it is possible to specifically and rationally evaluate customer satisfaction and accurately identify areas for improvement.

[0745] The processing flow will be explained below.

[0746] Step 1: Collect audio and video data

[0747] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with the customer.

[0748] Specifically, a microphone collects the conversational audio, while a camera records the customer's face frame by frame in high resolution.

[0749] Step 2: Real-time emotion recognition

[0750] The device's built-in emotion engine analyzes collected audio and video data in real time to recognize the user's emotions.

[0751] The system analyzes voice characteristics such as tone, pitch, and speed from audio data, and detects facial expressions and subtle movements from video data.

[0752] Step 3: Storing emotion data

[0753] The emotion engine recognizes emotion data in real time and stores it in internal storage, which is later used for detailed analysis.

[0754] The data stored includes the type of emotion recognized (e.g., joy, surprise, anger), intensity, and time stamp.

[0755] Step 4: Sending data to the server

[0756] The terminal transmits the collected audio data, video data, and stored emotion data all at once to the server.

[0757] The data is divided into data packets and transferred via a secure protocol.

[0758] Step 5: Receiving and Preprocessing Data

[0759] The server receives the data packets sent from the terminal.

[0760] It checks the integrity of the data packets and merges the audio and video data into their original format.

[0761] For audio data, a noise reduction filter is applied to remove background noise, and for video data, face detection and tracking is performed for each frame.

[0762] Step 6: Audio data analysis

[0763] The server inputs the voice data into a speech recognition model and converts it into text.

[0764] It extracts features such as voice tone, pitch, and speed, and uses a generative AI model to identify emotions.

[0765] Step 7: Video data analysis

[0766] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[0767] Emotions are classified by analyzing features such as eye movements, changes in the mouth, and facial expressions.

[0768] Step 8: Aggregating and calculating sentiment scores

[0769] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[0770] Each feature is weighted and an overall evaluation is performed.

[0771] Step 9: Calculate your NPS

[0772] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the sentiment score.

[0773] Based on the sentiment score, we calculate an accurate NPS and apply corrections to eliminate the tendency for responses to be centered, which is unique to Japan.

[0774] Step 10: Reporting results

[0775] The server sends the calculated NPS value and related analytical data to the device.

[0776] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[0777] Specific examples

[0778] Example 1: Calculating NPS in a call center

[0779] Step 1: The device collects the voice of the operator and the customer during the call, and records the operator's facial expressions with a camera.

[0780] Step 2: The emotion engine analyzes the operator's voice and facial expressions in real time to recognize emotions.

[0781] Step 3: The device stores the recognized emotion data in its internal storage.

[0782] Step 4: Send the collected data to the server.

[0783] Step 5: The server receives the data and performs audio and video preprocessing.

[0784] Step 6: Analyze the audio data and identify emotions.

[0785] Step 7: Analyze the video data and identify emotions from facial expressions.

[0786] Step 8: Combine the sentiment scores to generate a single overall sentiment score.

[0787] Step 9: Calculate the adjusted NPS based on the sentiment score.

[0788] Step 10: The device displays the NPS on the dashboard, which administrators can use to improve services.

[0789] Example 2: Customer service at the hotel front desk

[0790] Step 1: The device records audio and video of the customer and staff interaction during check-in.

[0791] Step 2: The emotion engine recognizes staff emotions in real time and stores the emotion data.

[0792] Step 3: Send the collected data to the server.

[0793] Step 4: The server receives the data and performs preprocessing.

[0794] Step 5: Noise reduction and analysis of the audio data.

[0795] Step 6: Analyze video data to identify customer sentiment.

[0796] Step 7: Combine the emotion scores to make an overall rating.

[0797] Step 8: Calculate the adjusted NPS from the sentiment score.

[0798] Step 9: The device displays the NPS, which the hotel manager uses as feedback.

[0799] Example 2

[0800] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] Previous systems have made it difficult to accurately assess customer satisfaction (NPS). In particular, it was difficult to quickly and accurately analyze audio and video data, and the tendency for response centering, which is unique to Japan, impacted NPS. Furthermore, it was inadequate to recognize customer sentiment in real time and immediately incorporate that data into analysis. This made it difficult to accurately grasp true customer satisfaction and link it to service improvements.

[0802] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for encrypting collected audio data and video data and transmitting them to the server using a secure protocol, means for performing noise reduction and audio feature extraction processing on the audio data in the server, means for performing facial recognition and extracting facial expression features on the video data in the server, and means for recognizing user emotions in real time using an emotion engine and storing them as emotion data. This enables secure processing of audio data and video data in real time and highly accurate identification of customer emotions. Furthermore, it enables accurate calculation of NPS, eliminating the tendency toward response centering that is unique to Japan, enabling rapid and accurate service improvements.

[0803] "Customer Satisfaction (NPS)" is an index that measures a customer's willingness to recommend a product or service to others, and is also known as the Net Promoter Score.

[0804] "Voice Data" refers to sound information collected using a microphone during a customer interaction.

[0805] "Video Data" refers to image information collected using a camera during an interaction with a customer.

[0806] A "generative AI model" refers to an artificial intelligence algorithm that is trained on large datasets and uses audio and video data to analyze customer sentiment.

[0807] An "emotion engine" refers to a software component that recognizes emotions in real time from voice tone and facial expressions.

[0808] "Emotion score" refers to an indicator that quantifies and objectively evaluates customer emotions.

[0809] "Correction to eliminate the tendency to center responses" refers to the process of using statistical methods to correct responses that tend to be biased toward intermediate evaluations, which are unique to Japan.

[0810] "Noise reduction" refers to the process of removing unnecessary background sounds from audio data and processing it into clear audio.

[0811] "Audio features" refers to attribute information necessary for analysis, such as tone, pitch, and speed, extracted from audio data.

[0812] "Facial recognition" refers to the technology of detecting and identifying human faces from video data frame by frame.

[0813] "Facial expression features" refer to attribute information such as eye movements and changes in the mouth that are extracted from video data to capture changes in facial expressions.

[0814] A "secure protocol" refers to a communication protocol that implements security measures such as encryption during data communication to ensure data protection.

[0815] The present invention relates to a system for calculating customer satisfaction (NPS) more accurately and efficiently. Specific embodiments of this system will be described below.

[0816] Hardware and software used

[0817] The device is equipped with a built-in microphone and camera to collect audio and video data during interactions with customers. It also has an emotion engine for real-time emotion recognition. The device encrypts the collected data and transmits it to a server using a secure protocol.

[0818] The server runs a generative AI model to analyze the received audio and video data. The analysis process involves noise reduction of the audio data, extraction of audio features, and facial recognition and extraction of facial features of the video data. Based on the generated emotion data, calculations are performed to calculate the emotion score and NPS.

[0819] Specific processing of the system

[0820] 1. Data Collection

[0821] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with a customer.

[0822] The device's emotion engine analyzes voice tone and facial expressions to generate and store emotion data in real time.

[0823] 2. Data Transmission

[0824] The data collected by the device is AES encrypted and sent securely to the server using the HTTPS protocol.

[0825] 3. Data Analysis

[0826] The server applies a noise reduction filter to the audio data to process it into clearer audio, and extracts audio features (e.g., tone, pitch, and speed).

[0827] The server applies a facial recognition algorithm to the video data and extracts facial features (e.g., eye movements and changes in the mouth).

[0828] 4. Emotion identification

[0829] The server uses the generative AI model to input the analyzed audio and video data to accurately identify the customer's emotions, and also integrates real-time emotional data to determine the most accurate emotional state.

[0830] 5. NPS calculation

[0831] The server quantifies the identified emotions and generates an emotion score. Based on the emotion score, the NPS is calculated and a statistical method is used to correct for the tendency for responses to be centered, which is unique to Japan.

[0832] 6.Result report

[0833] The server sends the calculated NPS to the device, which displays the results in the form of a dashboard or report.

[0834] Specific examples

[0835] Example 1: Calculating NPS in a call center

[0836] Data collection: The device collects the voices of both the operator and the customer during the call using a high-quality microphone, and records the operator's facial expressions frame by frame using a built-in camera. The emotion engine stores the operator's emotional data in real time.

[0837] Data transmission: Encrypted audio and video data is sent to the server using the HTTPS protocol.

[0838] Emotion analysis: The server inputs noise-reduced audio data into a generative AI model to identify emotions. Facial features are extracted and classified from video data.

[0839] NPS calculation: Calculate the corrected NPS based on the sentiment score and send the result to the device.

[0840] Displaying results: The device displays NPS and detailed analysis results on a dashboard, which administrators can use to improve services.

[0841] Example prompt sentence:

[0842] Please explain in detail the processing flow of a system that collects conversation data between agents and customers in a call center, analyzes agent emotions in real time using an emotion engine, and calculates NPS, including the specific operation of each processing step.

[0843] Example 2: Customer service at the hotel front desk

[0844] Data collection: The device records the interaction between the customer and the staff at check-in. The microphone collects audio data, and the camera collects high-resolution video data. The emotion engine recognizes and stores the staff's emotions in real time.

[0845] Data transmission: Encrypted data is sent to the server using a secure protocol.

[0846] Sentiment analysis: The server analyzes the audio and video data using a generative AI model to determine an emotion score.

[0847] NPS calculation: Calculate the corrected NPS based on the sentiment score and send it to the device.

[0848] Results display: The device displays a detailed NPS report on a dashboard for hotel managers to use for feedback.

[0849] Example prompt sentence:

[0850] Please explain the system that collects real-time audio and video data of customer interactions at hotel front desks, analyzes staff emotions using an emotion engine, and calculates emotion scores and NPS, including the specific operation of each processing step.

[0851] In this way, the present invention improves service quality by precisely understanding customer emotions based on audio and video analysis and calculating an accurate NPS. Real-time emotion analysis is possible, which allows for rapid feedback and the implementation of appropriate improvement measures.

[0852] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0853] Step 1: Data collection

[0854] The device uses a built-in microphone to collect voice data in real time while interacting with the customer.

[0855] The device uses a built-in high-resolution camera to record the customer's facial expressions frame by frame.

[0856] The device's emotion engine analyzes voice tone and facial expressions in real time and stores them as emotional data.

[0857] Input: Voice recording of conversation between customer and staff, and video of customer's facial expressions.

[0858] Output: Audio and video data, real-time emotion data.

[0859] Step 2: Send data

[0860] The audio and video data collected by the device, as well as real-time emotion data, are encrypted using AES.

[0861] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[0862] Input: Audio data, video data, and emotion data before encryption.

[0863] Output: The encrypted data sent to the server.

[0864] Step 3: Noise reduction of audio data

[0865] The server applies a noise reduction filter to the audio data it receives, removing background noise and producing clear audio.

[0866] Input: Encrypted audio data.

[0867] Output: Clear audio data after noise reduction.

[0868] Step 4: Extract audio features

[0869] The server extracts speech features such as tone, pitch, and speed from the noise-reduced speech data.

[0870] Input: Noise-reduced audio data.

[0871] Output: Audio feature data.

[0872] Step 5: Facial recognition of video data

[0873] The server applies a facial recognition algorithm to the video data it receives and detects the customer's face in each frame.

[0874] Input: Encrypted video data.

[0875] Output: Face recognition results and facial expression information per frame.

[0876] Step 6: Extraction of facial features

[0877] The server extracts facial expression changes (for example, eye movements or changes in the mouth) as features from the video data that has undergone facial recognition.

[0878] Input: Face-recognized video data.

[0879] Output: Facial expression feature data.

[0880] Step 7: Identify emotions

[0881] The server inputs the analyzed audio and video data using the generative AI model to identify customer emotions.

[0882] The server integrates this with real-time emotional data to determine the most accurate emotional state.

[0883] Input: Audio feature data, facial expression feature data, real-time emotion data.

[0884] Output: Identified emotion data.

[0885] Step 8: Sentiment score and NPS calculation

[0886] The server quantifies the identified emotion and calculates it as an emotion score.

[0887] The server calculates the NPS based on this emotional score and corrects for the tendency for responses to be centered, which is unique to Japan, using statistical methods.

[0888] Input: Identified emotion data.

[0889] Output: Sentiment score and adjusted NPS.

[0890] Step 9: Report your findings

[0891] The server sends the calculated NPS to the device.

[0892] The NPS sent by the device is displayed to the user in the form of a dashboard or report.

[0893] Users use this data to evaluate customer satisfaction and consider ways to improve services.

[0894] Input: Calculated NPS.

[0895] Output: NPS and detailed analysis results displayed in dashboards and reports.

[0896] (Application example 2)

[0897] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0898] Conventional systems for calculating customer satisfaction (NPS) have had difficulty eliminating the tendency for Japanese people to center their responses and accurately measure customer satisfaction. There was also a lack of means to recognize customer emotions in real time and immediately improve services. Furthermore, there were issues with the accuracy and practicality of systems that effectively collect and analyze customer voice and video data during interactions.

[0899] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data and video data during a conversation with a customer, means for analyzing the collected voice data and video data and using a generative AI model to identify the customer's emotions, means for calculating a customer's emotion score based on the analysis results and correcting it to eliminate response centralization to accurately calculate customer satisfaction, and means for performing the collection and analysis in real time via smart glasses. This makes it possible to recognize customer emotions with high accuracy in real time and accurately measure customer satisfaction. Furthermore, staff and managers can receive instant feedback and respond quickly to service improvements.

[0900] "Voice Data" refers to the audio information exchanged between a customer and a staff member during an interaction.

[0901] "Video data" refers to video information such as the faces and expressions of customers and staff during a conversation.

[0902] "Generative AI model" refers to an artificial intelligence model trained to analyze collected audio and video data and identify customer sentiment.

[0903] "Emotion score" refers to the customer's emotional state, expressed numerically as a result of analysis using a generative AI model.

[0904] "Correction to eliminate the tendency to center responses" refers to a statistical method for more accurately calculating customer satisfaction by eliminating the intermediate response tendency that is unique to Japanese people.

[0905] "Smart glasses" refers to a wearable eyeglass-type device that has the ability to collect audio and video data and analyze it in real time.

[0906] The present invention relates to a system for measuring customer satisfaction (NPS) in real time using smart glasses in a brick-and-mortar store. Specific embodiments thereof will be described below.

[0907] Hardware Configuration

[0908] The system includes smart glasses for collecting audio and video data, and a back-end server for analyzing the data. The smart glasses can be wearable devices such as Google Glass or Vuzix Blade. The back-end server has a high-performance processor and ample memory for processing and analyzing the collected data. It uses MySQL as the database and GPT-3 as the generative AI model.

[0909] Software Configuration

[0910] The following software modules are installed on the server:

[0911] Data collection module: Receives audio and video data transmitted from the smart glasses.

[0912] Noise reduction filter: Removes noise from audio data.

[0913] Audio feature extraction module: Extracts features from the audio data after noise removal.

[0914] Facial recognition module: Recognizes the customer's face for each frame of video data and extracts facial features.

[0915] Generative AI model module: A generative AI model to analyze collected data and identify customer sentiment.

[0916] Emotion score calculation module: Calculates the emotion score based on the analysis results.

[0917] NPS calculation module: Eliminates the tendency to center responses and calculates accurate customer satisfaction.

[0918] Data processing and calculation

[0919] 1. Data collection: Store staff wearing smart glasses collect audio and video data while interacting with customers. The data is sent to a server in real time.

[0920] 2. Noise reduction: Remove background noise from the audio data on the server to make the conversation clearer.

[0921] 3. Feature extraction: Extract audio features from the noise-removed audio data, and extract facial features for each frame from the video data.

[0922] 4. Emotion Identification: Using a generative AI model, the extracted feature data is analyzed to identify the customer's emotions.

[0923] 5. Emotion score calculation: The identified emotions are quantified and an emotion score is calculated.

[0924] 6. NPS calculation: NPS is calculated based on the emotional score, and the tendency for responses to be centered, which is unique to Japan, is corrected using statistical methods to calculate accurate customer satisfaction.

[0925] 7. Displaying results: The NPS is displayed on the smart glasses screen or on the store's management terminal, allowing staff and managers to receive feedback in real time.

[0926] Examples and prompts

[0927] As a concrete example, when a store staff member explains a product to a customer, audio and video data is collected and emotional data is generated in real time using an emotion engine. This data is sent from the smart glasses to a server, where it is analyzed by a generative AI model. An NPS is calculated based on the emotional score, and the results are displayed on the smart glasses. This allows staff to instantly grasp customer satisfaction levels and use the information to improve service.

[0928] Example prompt sentence:

[0929] "Using the audio and video data collected during the conversation, identify the customer's emotions and calculate the NPS. What adjustments are necessary to eliminate the tendency for responses to be centered, which is unique to Japan? Explain in detail how to calculate the emotion score and NPS."

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The terminal (smart glasses) collects audio and video data during a conversation with a customer. Specifically, audio and video information is captured in real time using the smart glasses' built-in microphone and camera. The input is the customer's audio and video during the conversation, and the output is digital audio and video files.

[0933] Step 2:

[0934] The audio and video data collected by the device is sent to the server via wireless communication (Bluetooth or Wi-Fi). The input is the collected audio and video files, and the output is the audio and video files stored on the server.

[0935] Step 3:

[0936] The server applies a noise reduction filter to the audio data it receives to process it into clear audio. Specifically, it applies an algorithm to remove background noise from the audio data. The input is the original audio data, and the output is clear audio data with noise removed.

[0937] Step 4:

[0938] The server extracts speech features from the noise-removed speech data. Specifically, it analyzes information such as tone, pitch, and speed of the speech. The input is the noise-removed speech data, and the output is the extracted speech features.

[0939] Step 5:

[0940] The server performs facial recognition and emotion classification for each frame of the video data it receives. Specifically, it uses a facial recognition algorithm to detect the customer's face and analyze their facial features. The input is the original video data, and the output is the extracted facial feature values.

[0941] Step 6:

[0942] The server inputs the extracted voice and facial features into a generative AI model to identify the customer's emotion. The input is the voice and facial features, and the output is the determined customer emotion.

[0943] Step 7:

[0944] The server quantifies the identified customer emotions and calculates an emotion score. Specifically, it assigns a certain score to each emotion based on the prediction results of the generative AI model. The input is the identified emotion, and the output is the emotion score.

[0945] Step 8:

[0946] The server calculates the NPS based on the sentiment score and corrects it using a statistical method to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the sentiment score, and the output is the corrected NPS.

[0947] Step 9:

[0948] The server transmits the calculated NPS to the device (smart glasses) via wireless communication. The input is the corrected NPS, and the output is the NPS displayed on the smart glasses' display.

[0949] Step 10:

[0950] The user (store staff) checks the NPS displayed on the smart glasses display, instantly understanding customer satisfaction and helping to improve services. The input is the NPS displayed on the display, and the output is customer satisfaction information recognized by the user.

[0951] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0952] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0953] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0954] [Third embodiment]

[0955] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0956] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0957] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0958] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0959] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0960] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0961] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0962] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0963] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0964] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0965] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0966] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0967] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[0968] System Overview

[0969] Terminal: A front-end device that handles customer service and collects audio and video data.

[0970] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[0971] User: Operates the system, understands customer satisfaction, and improves services.

[0972] Program processing explanation

[0973] 1. Data Collection Phase

[0974] During a conversation with a customer, the device uses a built-in microphone and camera to collect audio and video data in real time, for example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[0975] 2. Data transmission phase

[0976] The collected audio and video data is sent from the device to a server, and data transmission is always via a secure protocol to protect the integrity and privacy of the data.

[0977] 3. Data analysis phase

[0978] The server applies a noise reduction filter to the received audio data to make it clearer, for example, by removing background speech and other noise.

[0979] Next, a facial recognition algorithm is used to detect the customer's face from the video data and extract facial features, which can detect minute changes such as eye and mouth movements and wrinkles.

[0980] 4. Emotion Identification Phase

[0981] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained based on a large amount of past data sets, enabling highly accurate emotion identification.

[0982] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "excited" or "calm."

[0983] The system evaluates changes in facial expressions from video data and classifies emotions such as smiles, surprise, and anger with high accuracy.

[0984] 5. Sentiment score and NPS calculation phase

[0985] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[0986] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[0987] 6. Reporting Phase

[0988] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the history of sentiment scores and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[0989] Specific examples

[0990] Example 1: Calculating NPS in a call center

[0991] Data collection: The device collects the voices of both the operator and the customer during the call, and records the operator's facial expressions with a camera.

[0992] Data transmission: These data are transmitted to the server in real time.

[0993] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[0994] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[0995] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[0996] Example 2: Customer service at the hotel front desk

[0997] Data collection: The device records audio and video of the customer and staff interaction during check-in.

[0998] Data transmission: Send the recorded data to the server.

[0999] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[1000] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[1001] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[1002] This is a groundbreaking system that improves service quality by precisely understanding customer emotions based on audio and video analysis and more accurately reflecting NPS. In this way, it is possible to evaluate customer satisfaction more specifically and rationally and accurately derive areas for improvement.

[1003] The processing flow will be explained below.

[1004] Step 1: Data collection

[1005] While the device is interacting with the customer, it uses its built-in microphone and camera to collect audio and video data in real time.

[1006] Specifically, a microphone captures the conversational audio, and a camera records the customer's facial expressions frame by frame.

[1007] Step 2: Send data

[1008] The terminal converts the collected audio data into a digital signal and compresses and encodes the video data frame by frame.

[1009] The encoded data is divided into data packets and sent to the server using a secure protocol.

[1010] Step 3: Data reception and preprocessing

[1011] The server receives the data packets sent from the terminal.

[1012] The integrity of the data packets is verified, and the data is consolidated and restored to its original form.

[1013] The server applies a noise reduction filter to the audio data to remove unwanted background sounds.

[1014] Video data is analyzed frame by frame to perform face detection and tracking.

[1015] Step 4: Audio data analysis

[1016] The server inputs the voice data into a speech recognition model and converts it into text.

[1017] Extract speech features such as tone, pitch, and speed.

[1018] Based on the extracted features, a generative AI model identifies emotions.

[1019] Step 5: Video data analysis

[1020] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[1021] It analyzes features such as eye movements, mouth movements, and facial expressions to classify emotions.

[1022] Step 6: Integrating sentiment scores

[1023] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[1024] Each feature is weighted and an overall evaluation is calculated.

[1025] Step 7: Calculate your NPS

[1026] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the emotional scores.

[1027] Calculate an accurate NPS based on sentiment scores and apply a correction to eliminate response centralization bias.

[1028] Step 8: Report your findings

[1029] The server sends the calculated NPS value and related analytical data to the device.

[1030] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[1031] Example 1

[1032] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Accurately calculating customer satisfaction (NPS) is essential for improving a company's services. However, due to the tendency toward response-centering that is unique to Japan, it has been difficult to accurately calculate NPS using conventional methods. Furthermore, even when identifying customer emotions using audio and video data, noise and unclear data can reduce the accuracy of the analysis, which has been a problem. Therefore, there is a strong demand for the development of a system that can analyze customer audio and video data to accurately identify emotions and calculate an adjusted NPS based on the results.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1035] In this invention, the server includes means for transmitting collected audio and video data using an Internet Protocol, means for applying a noise reduction filter to the received audio data, means for detecting a customer's face from the video data using a facial recognition algorithm and extracting facial expression features, means for inputting the analyzed audio and video data into a generative AI model and identifying the customer's emotion using a prompt sentence, means for quantifying the identified emotion and expressing it as an emotion score, means for calculating customer satisfaction based on the emotion score and correcting it to eliminate answer centralization tendency, and means for generating a dashboard for transmitting the calculated customer satisfaction to a terminal and visualizing it. This improves the accuracy of analysis of audio and video data, making it possible to accurately identify customer emotion and calculate a corrected NPS.

[1036] "Voice data" refers to recordings of speech collected during customer interactions and is a digital audio signal captured using a microphone.

[1037] "Video Data" refers to video recordings collected during interactions with customers and is a digital video signal captured using a camera.

[1038] The "Internet Protocol" is a set of protocols for sending and receiving data over a network, and defines how data is packetized and transmitted.

[1039] A "noise reduction filter" is an algorithm or software process used to remove unwanted noise and interference from audio data.

[1040] A "face recognition algorithm" is a computational method for detecting a person's face from video data and extracting its features.

[1041] A "generative AI model" is an artificial intelligence algorithm that is trained on large datasets to generate specific information from audio or video data.

[1042] A "prompt" is an instruction or question that is input into a generative AI model to enable it to perform a specific task.

[1043] An "emotion score" is a numerical representation of the emotion identified from the analyzed audio and video data.

[1044] "Customer satisfaction" is an indicator of how customers rate their satisfaction with their experiences and services, and includes the Net Promoter Score (NPS).

[1045] "Response centralization" is the tendency of respondents in a particular cultural context to choose the middle or average answer.

[1046] A "dashboard" is an interface for centrally displaying data visualization and analysis results.

[1047] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[1048] System Overview

[1049] Terminal: A front-end device that handles customer service and collects audio and video data. Specific hardware features include high-sensitivity microphones and HD cameras. Examples include the Bose NC700 and Logitech C920.

[1050] Server: A backend device that analyzes data, calculates sentiment scores, and calculates NPS. The server has advanced computing power and large data storage capacity.

[1051] Users: Operate the system, understand customer satisfaction, and improve services. Users can use the dashboard to visually check the analysis results.

[1052] Program processing explanation

[1053] 1. Data Collection Phase

[1054] While the terminal is interacting with the customer, it uses a built-in microphone (e.g., a high-sensitivity microphone) and a camera (e.g., an HD camera) to collect audio and video data in real time. For example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[1055] 2. Data transmission phase

[1056] The collected audio and video data is sent from the device to the server. Data transmission is via a secure protocol (e.g., SSL / TLS) to protect the integrity and privacy of the data.

[1057] 3. Data analysis phase

[1058] The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data to remove background noise and unwanted audio components. Specifically, the server analyzes the audio spectrum using an FFT (Fast Fourier Transform) and attenuates the noise components.

[1059] Next, the server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and creates a list of feature points.

[1060] 4. Emotion Identification Phase

[1061] The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) to identify the customer's emotions. For example, for audio data, a request is sent along with the prompt, "Please identify the customer's emotions from this audio data." The returned output includes emotions such as "elated," "calm," and "anxious."

[1062] For video data, the prompt is, "Analyze the customer's facial expressions in this video and identify their emotions." The model's output includes emotions such as "smile," "surprise," and "anger."

[1063] 5. Sentiment score and NPS calculation phase

[1064] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience. The NPS is calculated based on the emotion score, and by adjusting for the bias of intermediate responses using statistical methods, the tendency to center responses, which is unique to Japan, can be eliminated.

[1065] 6. Reporting Phase

[1066] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the sentiment score history and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[1067] Specific examples

[1068] Example 1: Calculating NPS in a call center

[1069] Data collection: The device collects the voices of both the operator and the customer during the call using a built-in microphone, and records the operator's facial expressions using a camera.

[1070] Data transmission: Collected data is transmitted to the server in real time.

[1071] Emotion analysis: The server analyzes the operator's tone and pitch from the voice data and extracts changes in the operator's facial expressions from the video data.

[1072] Emotion Identification: The server identifies emotions using a generative AI model.

[1073] NPS calculation: Calculate the adjusted NPS based on the sentiment score and send the result to the device.

[1074] Displaying results: The device displays the NPS on the dashboard, which administrators can use to improve services.

[1075] Example 2: Customer service at the hotel front desk

[1076] Data collection: The device records the customer and staff interaction during check-in using its built-in microphone and camera.

[1077] Data transmission: The recorded data is securely transmitted to the server.

[1078] Sentiment Analysis: The server analyzes the audio and video data and uses a generative AI model to identify the customer's sentiment score.

[1079] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[1080] Results display: The device displays the NPS on a dashboard, which the hotel manager can use as feedback.

[1081] This invention makes it possible to precisely understand customer emotions based on audio and video analysis, more accurately reflecting NPS and improving service quality. Customer satisfaction can be specifically and rationally evaluated, and areas for improvement can be accurately identified.

[1082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1083] Program processing steps

[1084] Step 1: Data collection phase

[1085] Input: Customer audio and video data

[1086] Output: Collected audio and video data

[1087] Description: As a front-end device for customer service, the terminal uses a built-in microphone (e.g., a high-sensitivity microphone) and camera (e.g., an HD camera) to collect audio and video data in real time during conversations with customers. When the customer starts speaking, the terminal automatically starts recording and saving the audio data as a digital audio signal and the video data as a digital video signal.

[1088] Step 2: Data transmission phase

[1089] Input: Collected audio and video data

[1090] Output: Audio and video data sent to the server

[1091] Description: The device packetizes collected audio and video data and sends it to the server via a secure protocol (e.g., SSL / TLS). Specifically, it buffers and timestams the data, consolidates it, encodes it, and transmits it using Internet Protocol.

[1092] Step 3: Data Analysis Phase (Audio)

[1093] Input: Audio data sent to the server

[1094] Output: Clear audio data after noise reduction

[1095] Description: The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data. Specifically, it analyzes the audio spectrum using an FFT (Fast Fourier Transform), attenuates noise components, and generates audio data with improved voice clarity.

[1096] Step 4: Data analysis phase (video)

[1097] Input: Video data sent to the server

[1098] Output: Facial video data with extracted features

[1099] Description: The server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and lists the obtained feature points.

[1100] Step 5: Emotion Identification Phase

[1101] Input: Noise-reduced audio data and facial video data with extracted features

[1102] Output: Customer sentiment data

[1103] Description: The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) and identifies the customer's emotions using prompts. Specific prompts include, "Please identify the customer's emotions from this audio data" or "Please analyze the customer's facial expressions from this video and identify their emotions." The generative AI model identifies emotions based on the prompts and outputs emotional data such as "excited," "calm," "anxious," "smiling," "surprised," and "anger."

[1104] Step 6: Sentiment score and NPS calculation phase

[1105] Input: Customer sentiment data

[1106] Output: Sentiment score and adjusted NPS

[1107] Description: The server quantifies the identified emotions and expresses them as an emotion score. Based on this emotion score, the NPS is calculated and statistical methods (e.g., Z-score or standard deviation) are used to adjust for bias in intermediate responses and eliminate the tendency for responses to be centered, which is unique to Japan. As a result, a corrected NPS is calculated.

[1108] Step 7: Debriefing Phase

[1109] Inputs: Adjusted NPS, Sentiment Score

[1110] Output: A dashboard display of visualized NPS and sentiment scores

[1111] Description: The server sends the calculated NPS to the terminal and displays it to the user in the form of a dashboard or report. Specifically, the web server dynamically generates HTML and JavaScript to display it in the form of graphs and charts, allowing the user to intuitively understand the information. Based on this, the user can consider ways to improve the service and take specific actions.

[1112] (Application example 1)

[1113] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Conventional systems for evaluating customer satisfaction (NPS) have difficulty accurately grasping customer satisfaction levels because they are unable to properly eliminate the tendency for Japanese customers to center their responses. Furthermore, the lack of a system for analyzing conversations in physical stores in real time and providing immediate feedback makes it difficult to quickly improve services.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1116] In this invention, the server includes a means for collecting voice data and video data during a conversation with a customer, a means for using a generative AI model to analyze the collected voice data and video data to identify the customer's emotions, a means for calculating a customer's emotion score based on the analysis results, correcting the score to eliminate response centralization, and calculating an accurate customer satisfaction level, and a data collection means for evaluating the conversation between the customer and an employee in a physical store. This allows for an accurate NPS to be calculated immediately based on the customer's emotion evaluation, enabling prompt service improvements in the physical store.

[1117] "Customer" means a person to whom a service is provided, who purchases or uses goods or services.

[1118] "Auditory data" is information collected as sound, and is digital data of sound acquired by a sensor such as a microphone.

[1119] "Video data" refers to image data collected by visual sensors such as video cameras and cameras, and is composed of a series of still images divided into frames.

[1120] A "generative AI model" is an artificial intelligence model that has been trained using a large amount of learning data and has the ability to generate responses based on specific input data.

[1121] The "emotion score" is an index that numerically expresses the emotional state of the customer based on the analysis results of the audio data and video data.

[1122] "Response centralization" refers to the tendency for responses to be biased toward the middle of the evaluation scale in certain cultures and response patterns.

[1123] A "brick and mortar store" is a location that provides goods or services in person, either within a physical building or other location.

[1124] A system for implementing the present invention has a function for identifying customer emotions using audio and video data in order to accurately calculate customer satisfaction (NPS). The system includes the following components.

[1125] Hardware and Software Configuration

[1126] 1. Device:

[1127] To collect audio and video data, devices such as smartphones and smart glasses are used, which have built-in microphones and cameras and can collect data in real time.

[1128] 2. Server:

[1129] To analyze the collected audio and video data, a high-performance computer server is used. The server is equipped with the voice recognition library "Vosk" and the image recognition library "OpenCV," and these are used to analyze the data.

[1130] Emotion identification uses a generative AI model: a RoBERTa-based emotion classification model using the Hugging Face transforms library.

[1131] Data collection and transmission

[1132] The device collects audio and video data in real time during customer interactions, and the collected data is sent to the server via a secure protocol, such as Transport Layer Security (TLS), to protect the integrity and privacy of the data.

[1133] Data analysis

[1134] The server parses the data as follows:

[1135] Audio data analysis:

[1136] Apply noise reduction filters to get clear audio data. Filter to remove background noise.

[1137] It uses a speech recognition algorithm to convert the voice data into text, analyzes what the customer is saying, extracts features such as tone, pitch, and speed, and evaluates the sentiment.

[1138] Video data analysis:

[1139] A facial recognition algorithm is used to detect the customer's face from the video data, extracting facial features (e.g., eye and mouth movements, wrinkles, etc.) to identify emotions.

[1140] Sentiment score and NPS calculation

[1141] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The identified emotions are quantified and expressed as an emotion score. The NPS is calculated based on the emotion score and adjusted using statistical methods to eliminate the tendency for response centering, which is unique to Japan.

[1142] Reporting the results

[1143] The calculated NPS is returned to the device and displayed in the form of a dashboard and report. The dashboard visually displays the history of sentiment scores and detailed analysis results, allowing users to immediately identify problems and use the information to improve their services.

[1144] Specific examples

[1145] Example 1: Evaluation of cashier service at a physical store

[1146] Data collection: Smart glasses record the conversations and facial expressions between store staff and customers.

[1147] Data transmission: The recorded data is transmitted to the server via a secure protocol.

[1148] Sentiment analysis: The server analyzes voice tone and facial expressions to identify the customer's emotions.

[1149] NPS calculation: Calculates the adjusted NPS from the sentiment score and displays the results in real time.

[1150] View the results: See your NPS score displayed on the dashboard and take action to improve your customer experience.

[1151] Example prompts to input to the generative AI model

[1152] "Please identify customer emotions and calculate NPS based on audio and video data recorded from conversations between customers and store staff in-store. Specifically, I would like to analyze the tone of the voice and changes in facial expressions in the video to identify emotions such as 'excited' or 'calm'."

[1153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1154] Step 1:

[1155] The device collects audio and video data during interactions with customers. The input is the interaction between the customer and the store clerk in the physical store, and the output is the collected audio file (e.g., customer_interaction.wav) and video file (e.g., customer_interaction.mp4). Specifically, the smart glasses' built-in microphone and camera are used to record the conversation between the customer and the store clerk, as well as their facial expressions.

[1156] Step 2:

[1157] The audio and video data collected by the device is sent to the server via a secure protocol such as TLS (Transport Layer Security). The input is the audio and video files collected in step 1, and the output is these data transferred to the server. Specifically, when sending data from the device to the server, an encrypted connection is maintained to protect the integrity and privacy of the data.

[1158] Step 3:

[1159] The server applies a noise reduction filter to the received audio data to generate clear audio data. The input is the transferred audio file, and the output is the audio data with reduced noise. Specifically, it uses the "Vosk" speech recognition library and applies a specific filtering algorithm to remove background noise.

[1160] Step 4:

[1161] The server analyzes the noise-reduced speech data and extracts features such as speech tone, pitch, and speed. The input is the noise-reduced speech data, and the output is the extracted speech features. Specifically, a speech signal processing library is used to calculate speech parameters that indicate customer emotions.

[1162] Step 5:

[1163] The server uses a facial recognition algorithm to detect customer faces for each frame of the video data it receives and extracts facial expression features. The input is the video file, and the output is the detected face image and its facial expression features. Specifically, it uses OpenCV to extract face regions from the video frames and classify each facial expression (e.g., smile, surprise, anger).

[1164] Step 6:

[1165] The server inputs the analyzed audio and video data into a generative AI model to identify the customer's emotions. The inputs are audio features and video features, and the output is an identified emotion score. Specifically, a RoBERTa-based emotion classification model using the Hugging Face transformers library is used to predict the customer's emotional state from each feature.

[1166] Step 7:

[1167] The server quantifies the identified emotion scores and performs statistical corrections to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the identified emotion score, and the output is the corrected NPS score. Specifically, statistical methods are used to adjust responses that are biased toward the median value, resulting in a more accurate NPS.

[1168] Step 8:

[1169] The server sends the calculated NPS score to the terminal and displays the results in the form of a dashboard or report. The input is the corrected NPS score, and the output is the NPS score displayed on the terminal and its detailed analysis results. Specifically, the server provides a GUI (graphical user interface) to visually display the NPS score and its analysis results.

[1170] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1171] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the response-centered tendency that is unique to Japanese people. Furthermore, by incorporating an emotion engine that recognizes user emotions in real time, this system evaluates customer satisfaction with even greater precision. This system collects audio and video data during customer interactions, analyzes the collected data using a generative AI model to identify customer emotions, and calculates NPS based on the results. Furthermore, by recognizing user emotions in real time through the emotion engine and storing them as emotion data, further analysis and service improvement are possible.

[1172] System Overview

[1173] Terminal: A front-end device that handles customer service, it collects audio and video data and has a built-in emotion engine that recognizes user emotion data in real time.

[1174] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[1175] User: Operates the system, understands customer satisfaction, and improves services.

[1176] Program processing explanation

[1177] 1. Data Collection Phase

[1178] During a conversation with a customer, the device uses a built-in microphone and camera to collect real-time audio and video data, and the camera records the customer's facial expressions frame by frame in high resolution.

[1179] The emotion engine recognizes the user's emotions in real time and stores them as emotion data, which is based on information obtained from voice tone and facial expressions.

[1180] 2. Data transmission phase

[1181] The collected audio, video, and emotion data is sent from the device to a server, and data transmission is always via a secure protocol to protect data integrity and privacy.

[1182] 3. Data analysis phase

[1183] The server applies a noise reduction filter to the received audio data to make the audio clearer, for example by removing background noise to make the conversation clearer.

[1184] For video data, a facial recognition algorithm is used to detect the customer's face in each frame and extract facial features, capturing subtle facial features such as eye movements and changes in the mouth.

[1185] 4. Emotion Identification Phase

[1186] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained using a large amount of past data sets, enabling highly accurate emotion identification.

[1187] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "joy" or "anger."

[1188] The video data is used to evaluate changes in facial expressions and classify emotions such as smiles, surprise, and anger.

[1189] The accuracy of emotion identification is improved by taking into account user emotion data obtained from the emotion engine.

[1190] 5. Sentiment score and NPS calculation phase

[1191] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[1192] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[1193] 6. Reporting Phase

[1194] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard also displays the sentiment score history and detailed analysis results, allowing users to quickly identify problems and take measures to improve.

[1195] Specific examples

[1196] Example 1: Calculating NPS in a call center

[1197] Data collection: The terminal collects the voices of the operator and the customer during the call, records the operator's facial expressions with a camera, and the emotion engine recognizes the operator's emotional data in real time.

[1198] Data transmission: These data are transmitted to the server in real time.

[1199] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[1200] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[1201] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[1202] Example 2: Customer service at the hotel front desk

[1203] Data collection: The terminal records audio and video of the interaction between the customer and staff during check-in. The emotion engine also recognizes the staff's emotions in real time and stores them as emotion data.

[1204] Data transmission: Send the recorded data to the server.

[1205] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[1206] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[1207] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[1208] This is a groundbreaking system that precisely understands customer emotions based on audio and video analysis and accurately reflects NPS to improve service quality. Furthermore, by grasping user emotions in real time through the emotion engine, even more advanced analysis and feedback become possible. In this way, it is possible to specifically and rationally evaluate customer satisfaction and accurately identify areas for improvement.

[1209] The processing flow will be explained below.

[1210] Step 1: Collect audio and video data

[1211] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with the customer.

[1212] Specifically, a microphone collects the conversational audio, while a camera records the customer's face frame by frame in high resolution.

[1213] Step 2: Real-time emotion recognition

[1214] The device's built-in emotion engine analyzes collected audio and video data in real time to recognize the user's emotions.

[1215] The system analyzes voice characteristics such as tone, pitch, and speed from audio data, and detects facial expressions and subtle movements from video data.

[1216] Step 3: Storing emotion data

[1217] The emotion engine recognizes emotion data in real time and stores it in internal storage, which is later used for detailed analysis.

[1218] The data stored includes the type of emotion recognized (e.g., joy, surprise, anger), intensity, and time stamp.

[1219] Step 4: Sending data to the server

[1220] The terminal transmits the collected audio data, video data, and stored emotion data all at once to the server.

[1221] The data is divided into data packets and transferred via a secure protocol.

[1222] Step 5: Receiving and Preprocessing Data

[1223] The server receives the data packets sent from the terminal.

[1224] It checks the integrity of the data packets and merges the audio and video data into their original format.

[1225] For audio data, a noise reduction filter is applied to remove background noise, and for video data, face detection and tracking is performed for each frame.

[1226] Step 6: Audio data analysis

[1227] The server inputs the voice data into a speech recognition model and converts it into text.

[1228] It extracts features such as voice tone, pitch, and speed, and uses a generative AI model to identify emotions.

[1229] Step 7: Video data analysis

[1230] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[1231] Emotions are classified by analyzing features such as eye movements, changes in the mouth, and facial expressions.

[1232] Step 8: Aggregating and calculating sentiment scores

[1233] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[1234] Each feature is weighted and an overall evaluation is performed.

[1235] Step 9: Calculate your NPS

[1236] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the sentiment score.

[1237] Based on the sentiment score, we calculate an accurate NPS and apply corrections to eliminate the tendency for responses to be centered, which is unique to Japan.

[1238] Step 10: Reporting results

[1239] The server sends the calculated NPS value and related analytical data to the device.

[1240] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[1241] Specific examples

[1242] Example 1: Calculating NPS in a call center

[1243] Step 1: The device collects the voice of the operator and the customer during the call, and records the operator's facial expressions with a camera.

[1244] Step 2: The emotion engine analyzes the operator's voice and facial expressions in real time to recognize emotions.

[1245] Step 3: The device stores the recognized emotion data in its internal storage.

[1246] Step 4: Send the collected data to the server.

[1247] Step 5: The server receives the data and performs audio and video preprocessing.

[1248] Step 6: Analyze the audio data and identify emotions.

[1249] Step 7: Analyze the video data and identify emotions from facial expressions.

[1250] Step 8: Combine the sentiment scores to generate a single overall sentiment score.

[1251] Step 9: Calculate the adjusted NPS based on the sentiment score.

[1252] Step 10: The device displays the NPS on the dashboard, which administrators can use to improve services.

[1253] Example 2: Customer service at the hotel front desk

[1254] Step 1: The device records audio and video of the customer and staff interaction during check-in.

[1255] Step 2: The emotion engine recognizes staff emotions in real time and stores the emotion data.

[1256] Step 3: Send the collected data to the server.

[1257] Step 4: The server receives the data and performs preprocessing.

[1258] Step 5: Noise reduction and analysis of the audio data.

[1259] Step 6: Analyze video data to identify customer sentiment.

[1260] Step 7: Combine the emotion scores to make an overall rating.

[1261] Step 8: Calculate the adjusted NPS from the sentiment score.

[1262] Step 9: The device displays the NPS, which the hotel manager uses as feedback.

[1263] Example 2

[1264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1265] Previous systems have made it difficult to accurately assess customer satisfaction (NPS). In particular, it was difficult to quickly and accurately analyze audio and video data, and the tendency for response centering, which is unique to Japan, impacted NPS. Furthermore, it was inadequate to recognize customer sentiment in real time and immediately incorporate that data into analysis. This made it difficult to accurately grasp true customer satisfaction and link it to service improvements.

[1266] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for encrypting collected audio data and video data and transmitting them to the server using a secure protocol, means for performing noise reduction and audio feature extraction processing on the audio data in the server, means for performing facial recognition and extracting facial expression features on the video data in the server, and means for recognizing user emotions in real time using an emotion engine and storing them as emotion data. This enables secure processing of audio data and video data in real time and highly accurate identification of customer emotions. Furthermore, it enables accurate calculation of NPS, eliminating the tendency toward response centering that is unique to Japan, enabling rapid and accurate service improvements.

[1267] "Customer Satisfaction (NPS)" is an index that measures a customer's willingness to recommend a product or service to others, and is also known as the Net Promoter Score.

[1268] "Voice Data" refers to sound information collected using a microphone during a customer interaction.

[1269] "Video Data" refers to image information collected using a camera during an interaction with a customer.

[1270] A "generative AI model" refers to an artificial intelligence algorithm that is trained on large datasets and uses audio and video data to analyze customer sentiment.

[1271] An "emotion engine" refers to a software component that recognizes emotions in real time from voice tone and facial expressions.

[1272] "Emotion score" refers to an indicator that quantifies and objectively evaluates customer emotions.

[1273] "Correction to eliminate the tendency to center responses" refers to the process of using statistical methods to correct responses that tend to be biased toward intermediate evaluations, which are unique to Japan.

[1274] "Noise reduction" refers to the process of removing unnecessary background sounds from audio data and processing it into clear audio.

[1275] "Audio features" refers to attribute information necessary for analysis, such as tone, pitch, and speed, extracted from audio data.

[1276] "Facial recognition" refers to the technology of detecting and identifying human faces from video data frame by frame.

[1277] "Facial expression features" refer to attribute information such as eye movements and changes in the mouth that are extracted from video data to capture changes in facial expressions.

[1278] A "secure protocol" refers to a communication protocol that implements security measures such as encryption during data communication to ensure data protection.

[1279] The present invention relates to a system for calculating customer satisfaction (NPS) more accurately and efficiently. Specific embodiments of this system will be described below.

[1280] Hardware and software used

[1281] The device is equipped with a built-in microphone and camera to collect audio and video data during interactions with customers. It also has an emotion engine for real-time emotion recognition. The device encrypts the collected data and transmits it to a server using a secure protocol.

[1282] The server runs a generative AI model to analyze the received audio and video data. The analysis process involves noise reduction of the audio data, extraction of audio features, and facial recognition and extraction of facial features of the video data. Based on the generated emotion data, calculations are performed to calculate the emotion score and NPS.

[1283] Specific processing of the system

[1284] 1. Data Collection

[1285] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with a customer.

[1286] The device's emotion engine analyzes voice tone and facial expressions to generate and store emotion data in real time.

[1287] 2. Data Transmission

[1288] The data collected by the device is AES encrypted and sent securely to the server using the HTTPS protocol.

[1289] 3. Data Analysis

[1290] The server applies a noise reduction filter to the audio data to process it into clearer audio, and extracts audio features (e.g., tone, pitch, and speed).

[1291] The server applies a facial recognition algorithm to the video data and extracts facial features (e.g., eye movements and changes in the mouth).

[1292] 4. Emotion identification

[1293] The server uses the generative AI model to input the analyzed audio and video data to accurately identify the customer's emotions, and also integrates real-time emotional data to determine the most accurate emotional state.

[1294] 5. NPS calculation

[1295] The server quantifies the identified emotions and generates an emotion score. Based on the emotion score, the NPS is calculated and a statistical method is used to correct for the tendency for responses to be centered, which is unique to Japan.

[1296] 6.Result report

[1297] The server sends the calculated NPS to the device, which displays the results in the form of a dashboard or report.

[1298] Specific examples

[1299] Example 1: Calculating NPS in a call center

[1300] Data collection: The device collects the voices of both the operator and the customer during the call using a high-quality microphone, and records the operator's facial expressions frame by frame using a built-in camera. The emotion engine stores the operator's emotional data in real time.

[1301] Data transmission: Encrypted audio and video data is sent to the server using the HTTPS protocol.

[1302] Emotion analysis: The server inputs noise-reduced audio data into a generative AI model to identify emotions. Facial features are extracted and classified from video data.

[1303] NPS calculation: Calculate the corrected NPS based on the sentiment score and send the result to the device.

[1304] Displaying results: The device displays NPS and detailed analysis results on a dashboard, which administrators can use to improve services.

[1305] Example prompt sentence:

[1306] Please explain in detail the processing flow of a system that collects conversation data between agents and customers in a call center, analyzes agent emotions in real time using an emotion engine, and calculates NPS, including the specific operation of each processing step.

[1307] Example 2: Customer service at the hotel front desk

[1308] Data collection: The device records the interaction between the customer and the staff at check-in. The microphone collects audio data, and the camera collects high-resolution video data. The emotion engine recognizes and stores the staff's emotions in real time.

[1309] Data transmission: Encrypted data is sent to the server using a secure protocol.

[1310] Sentiment analysis: The server analyzes the audio and video data using a generative AI model to determine an emotion score.

[1311] NPS calculation: Calculate the corrected NPS based on the sentiment score and send it to the device.

[1312] Results display: The device displays a detailed NPS report on a dashboard for hotel managers to use for feedback.

[1313] Example prompt sentence:

[1314] Please explain the system that collects real-time audio and video data of customer interactions at hotel front desks, analyzes staff emotions using an emotion engine, and calculates emotion scores and NPS, including the specific operation of each processing step.

[1315] In this way, the present invention improves service quality by precisely understanding customer emotions based on audio and video analysis and calculating an accurate NPS. Real-time emotion analysis is possible, which allows for rapid feedback and the implementation of appropriate improvement measures.

[1316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1317] Step 1: Data collection

[1318] The device uses a built-in microphone to collect voice data in real time while interacting with the customer.

[1319] The device uses a built-in high-resolution camera to record the customer's facial expressions frame by frame.

[1320] The device's emotion engine analyzes voice tone and facial expressions in real time and stores them as emotional data.

[1321] Input: Voice recording of conversation between customer and staff, and video of customer's facial expressions.

[1322] Output: Audio and video data, real-time emotion data.

[1323] Step 2: Send data

[1324] The audio and video data collected by the device, as well as real-time emotion data, are encrypted using AES.

[1325] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[1326] Input: Audio data, video data, and emotion data before encryption.

[1327] Output: The encrypted data sent to the server.

[1328] Step 3: Noise reduction of audio data

[1329] The server applies a noise reduction filter to the audio data it receives, removing background noise and producing clear audio.

[1330] Input: Encrypted audio data.

[1331] Output: Clear audio data after noise reduction.

[1332] Step 4: Extract audio features

[1333] The server extracts speech features such as tone, pitch, and speed from the noise-reduced speech data.

[1334] Input: Noise-reduced audio data.

[1335] Output: Audio feature data.

[1336] Step 5: Facial recognition of video data

[1337] The server applies a facial recognition algorithm to the video data it receives and detects the customer's face in each frame.

[1338] Input: Encrypted video data.

[1339] Output: Face recognition results and facial expression information per frame.

[1340] Step 6: Extraction of facial features

[1341] The server extracts facial expression changes (for example, eye movements or changes in the mouth) as features from the video data that has undergone facial recognition.

[1342] Input: Face-recognized video data.

[1343] Output: Facial expression feature data.

[1344] Step 7: Identify emotions

[1345] The server inputs the analyzed audio and video data using the generative AI model to identify customer emotions.

[1346] The server integrates this with real-time emotional data to determine the most accurate emotional state.

[1347] Input: Audio feature data, facial expression feature data, real-time emotion data.

[1348] Output: Identified emotion data.

[1349] Step 8: Sentiment score and NPS calculation

[1350] The server quantifies the identified emotion and calculates it as an emotion score.

[1351] The server calculates the NPS based on this emotional score and corrects for the tendency for responses to be centered, which is unique to Japan, using statistical methods.

[1352] Input: Identified emotion data.

[1353] Output: Sentiment score and adjusted NPS.

[1354] Step 9: Report your findings

[1355] The server sends the calculated NPS to the device.

[1356] The NPS sent by the device is displayed to the user in the form of a dashboard or report.

[1357] Users use this data to evaluate customer satisfaction and consider ways to improve services.

[1358] Input: Calculated NPS.

[1359] Output: NPS and detailed analysis results displayed in dashboards and reports.

[1360] (Application example 2)

[1361] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1362] Conventional systems for calculating customer satisfaction (NPS) have had difficulty eliminating the tendency for Japanese people to center their responses and accurately measure customer satisfaction. There was also a lack of means to recognize customer emotions in real time and immediately improve services. Furthermore, there were issues with the accuracy and practicality of systems that effectively collect and analyze customer voice and video data during interactions.

[1363] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data and video data during a conversation with a customer, means for analyzing the collected voice data and video data and using a generative AI model to identify the customer's emotions, means for calculating a customer's emotion score based on the analysis results and correcting it to eliminate response centralization to accurately calculate customer satisfaction, and means for performing the collection and analysis in real time via smart glasses. This makes it possible to recognize customer emotions with high accuracy in real time and accurately measure customer satisfaction. Furthermore, staff and managers can receive instant feedback and respond quickly to service improvements.

[1364] "Voice Data" refers to the audio information exchanged between a customer and a staff member during an interaction.

[1365] "Video data" refers to video information such as the faces and expressions of customers and staff during a conversation.

[1366] "Generative AI model" refers to an artificial intelligence model trained to analyze collected audio and video data and identify customer sentiment.

[1367] "Emotion score" refers to the customer's emotional state, expressed numerically as a result of analysis using a generative AI model.

[1368] "Correction to eliminate the tendency to center responses" refers to a statistical method for more accurately calculating customer satisfaction by eliminating the intermediate response tendency that is unique to Japanese people.

[1369] "Smart glasses" refers to a wearable eyeglass-type device that has the ability to collect audio and video data and analyze it in real time.

[1370] The present invention relates to a system for measuring customer satisfaction (NPS) in real time using smart glasses in a brick-and-mortar store. Specific embodiments thereof will be described below.

[1371] Hardware Configuration

[1372] The system includes smart glasses for collecting audio and video data, and a back-end server for analyzing the data. The smart glasses can be wearable devices such as Google Glass or Vuzix Blade. The back-end server has a high-performance processor and ample memory for processing and analyzing the collected data. It uses MySQL as the database and GPT-3 as the generative AI model.

[1373] Software Configuration

[1374] The following software modules are installed on the server:

[1375] Data collection module: Receives audio and video data transmitted from the smart glasses.

[1376] Noise reduction filter: Removes noise from audio data.

[1377] Audio feature extraction module: Extracts features from the audio data after noise removal.

[1378] Facial recognition module: Recognizes the customer's face for each frame of video data and extracts facial features.

[1379] Generative AI model module: A generative AI model to analyze collected data and identify customer sentiment.

[1380] Emotion score calculation module: Calculates the emotion score based on the analysis results.

[1381] NPS calculation module: Eliminates the tendency to center responses and calculates accurate customer satisfaction.

[1382] Data processing and calculation

[1383] 1. Data collection: Store staff wearing smart glasses collect audio and video data while interacting with customers. The data is sent to a server in real time.

[1384] 2. Noise reduction: Remove background noise from the audio data on the server to make the conversation clearer.

[1385] 3. Feature extraction: Extract audio features from the noise-removed audio data, and extract facial features for each frame from the video data.

[1386] 4. Emotion Identification: Using a generative AI model, the extracted feature data is analyzed to identify the customer's emotions.

[1387] 5. Emotion score calculation: The identified emotions are quantified and an emotion score is calculated.

[1388] 6. NPS calculation: NPS is calculated based on the emotional score, and the tendency for responses to be centered, which is unique to Japan, is corrected using statistical methods to calculate accurate customer satisfaction.

[1389] 7. Displaying results: The NPS is displayed on the smart glasses screen or on the store's management terminal, allowing staff and managers to receive feedback in real time.

[1390] Examples and prompts

[1391] As a concrete example, when a store staff member explains a product to a customer, audio and video data is collected and emotional data is generated in real time using an emotion engine. This data is sent from the smart glasses to a server, where it is analyzed by a generative AI model. An NPS is calculated based on the emotional score, and the results are displayed on the smart glasses. This allows staff to instantly grasp customer satisfaction levels and use the information to improve service.

[1392] Example prompt sentence:

[1393] "Using the audio and video data collected during the conversation, identify the customer's emotions and calculate the NPS. What adjustments are necessary to eliminate the tendency for responses to be centered, which is unique to Japan? Explain in detail how to calculate the emotion score and NPS."

[1394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1395] Step 1:

[1396] The terminal (smart glasses) collects audio and video data during a conversation with a customer. Specifically, audio and video information is captured in real time using the smart glasses' built-in microphone and camera. The input is the customer's audio and video during the conversation, and the output is digital audio and video files.

[1397] Step 2:

[1398] The audio and video data collected by the device is sent to the server via wireless communication (Bluetooth or Wi-Fi). The input is the collected audio and video files, and the output is the audio and video files stored on the server.

[1399] Step 3:

[1400] The server applies a noise reduction filter to the audio data it receives to process it into clear audio. Specifically, it applies an algorithm to remove background noise from the audio data. The input is the original audio data, and the output is clear audio data with noise removed.

[1401] Step 4:

[1402] The server extracts speech features from the noise-removed speech data. Specifically, it analyzes information such as tone, pitch, and speed of the speech. The input is the noise-removed speech data, and the output is the extracted speech features.

[1403] Step 5:

[1404] The server performs facial recognition and emotion classification for each frame of the video data it receives. Specifically, it uses a facial recognition algorithm to detect the customer's face and analyze their facial features. The input is the original video data, and the output is the extracted facial feature values.

[1405] Step 6:

[1406] The server inputs the extracted voice and facial features into a generative AI model to identify the customer's emotion. The input is the voice and facial features, and the output is the determined customer emotion.

[1407] Step 7:

[1408] The server quantifies the identified customer emotions and calculates an emotion score. Specifically, it assigns a certain score to each emotion based on the prediction results of the generative AI model. The input is the identified emotion, and the output is the emotion score.

[1409] Step 8:

[1410] The server calculates the NPS based on the sentiment score and corrects it using a statistical method to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the sentiment score, and the output is the corrected NPS.

[1411] Step 9:

[1412] The server transmits the calculated NPS to the device (smart glasses) via wireless communication. The input is the corrected NPS, and the output is the NPS displayed on the smart glasses' display.

[1413] Step 10:

[1414] The user (store staff) checks the NPS displayed on the smart glasses display, instantly understanding customer satisfaction and helping to improve services. The input is the NPS displayed on the display, and the output is customer satisfaction information recognized by the user.

[1415] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1417] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1418] [Fourth embodiment]

[1419] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1420] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1422] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1426] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1427] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1430] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1431] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[1433] System Overview

[1434] Terminal: A front-end device that handles customer service and collects audio and video data.

[1435] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[1436] User: Operates the system, understands customer satisfaction, and improves services.

[1437] Program processing explanation

[1438] 1. Data Collection Phase

[1439] During a conversation with a customer, the device uses a built-in microphone and camera to collect audio and video data in real time, for example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[1440] 2. Data transmission phase

[1441] The collected audio and video data is sent from the device to a server, and data transmission is always via a secure protocol to protect the integrity and privacy of the data.

[1442] 3. Data analysis phase

[1443] The server applies a noise reduction filter to the received audio data to make it clearer, for example, by removing background speech and other noise.

[1444] Next, a facial recognition algorithm is used to detect the customer's face from the video data and extract facial features, which can detect minute changes such as eye and mouth movements and wrinkles.

[1445] 4. Emotion Identification Phase

[1446] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained based on a large amount of past data sets, enabling highly accurate emotion identification.

[1447] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "excited" or "calm."

[1448] The system evaluates changes in facial expressions from video data and classifies emotions such as smiles, surprise, and anger with high accuracy.

[1449] 5. Sentiment score and NPS calculation phase

[1450] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[1451] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[1452] 6. Reporting Phase

[1453] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the history of sentiment scores and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[1454] Specific examples

[1455] Example 1: Calculating NPS in a call center

[1456] Data collection: The device collects the voices of both the operator and the customer during the call, and records the operator's facial expressions with a camera.

[1457] Data transmission: These data are transmitted to the server in real time.

[1458] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[1459] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[1460] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[1461] Example 2: Customer service at the hotel front desk

[1462] Data collection: The device records audio and video of the customer and staff interaction during check-in.

[1463] Data transmission: Send the recorded data to the server.

[1464] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[1465] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[1466] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[1467] This is a groundbreaking system that improves service quality by precisely understanding customer emotions based on audio and video analysis and more accurately reflecting NPS. In this way, it is possible to evaluate customer satisfaction more specifically and rationally and accurately derive areas for improvement.

[1468] The processing flow will be explained below.

[1469] Step 1: Data collection

[1470] While the device is interacting with the customer, it uses its built-in microphone and camera to collect audio and video data in real time.

[1471] Specifically, a microphone captures the conversational audio, and a camera records the customer's facial expressions frame by frame.

[1472] Step 2: Send data

[1473] The terminal converts the collected audio data into a digital signal and compresses and encodes the video data frame by frame.

[1474] The encoded data is divided into data packets and sent to the server using a secure protocol.

[1475] Step 3: Data reception and preprocessing

[1476] The server receives the data packets sent from the terminal.

[1477] The integrity of the data packets is verified, and the data is consolidated and restored to its original form.

[1478] The server applies a noise reduction filter to the audio data to remove unwanted background sounds.

[1479] Video data is analyzed frame by frame to perform face detection and tracking.

[1480] Step 4: Audio data analysis

[1481] The server inputs the voice data into a speech recognition model and converts it into text.

[1482] Extract speech features such as tone, pitch, and speed.

[1483] Based on the extracted features, a generative AI model identifies emotions.

[1484] Step 5: Video data analysis

[1485] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[1486] It analyzes features such as eye movements, mouth movements, and facial expressions to classify emotions.

[1487] Step 6: Integrating sentiment scores

[1488] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[1489] Each feature is weighted and an overall evaluation is calculated.

[1490] Step 7: Calculate your NPS

[1491] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the emotional scores.

[1492] Calculate an accurate NPS based on sentiment scores and apply a correction to eliminate response centralization bias.

[1493] Step 8: Report your findings

[1494] The server sends the calculated NPS value and related analytical data to the device.

[1495] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[1496] Example 1

[1497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1498] Accurately calculating customer satisfaction (NPS) is essential for improving a company's services. However, due to the tendency toward response-centering that is unique to Japan, it has been difficult to accurately calculate NPS using conventional methods. Furthermore, even when identifying customer emotions using audio and video data, noise and unclear data can reduce the accuracy of the analysis, which has been a problem. Therefore, there is a strong demand for the development of a system that can analyze customer audio and video data to accurately identify emotions and calculate an adjusted NPS based on the results.

[1499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1500] In this invention, the server includes means for transmitting collected audio and video data using an Internet Protocol, means for applying a noise reduction filter to the received audio data, means for detecting a customer's face from the video data using a facial recognition algorithm and extracting facial expression features, means for inputting the analyzed audio and video data into a generative AI model and identifying the customer's emotion using a prompt sentence, means for quantifying the identified emotion and expressing it as an emotion score, means for calculating customer satisfaction based on the emotion score and correcting it to eliminate answer centralization tendency, and means for generating a dashboard for transmitting the calculated customer satisfaction to a terminal and visualizing it. This improves the accuracy of analysis of audio and video data, making it possible to accurately identify customer emotion and calculate a corrected NPS.

[1501] "Voice data" refers to recordings of speech collected during customer interactions and is a digital audio signal captured using a microphone.

[1502] "Video Data" refers to video recordings collected during interactions with customers and is a digital video signal captured using a camera.

[1503] The "Internet Protocol" is a set of protocols for sending and receiving data over a network, and defines how data is packetized and transmitted.

[1504] A "noise reduction filter" is an algorithm or software process used to remove unwanted noise and interference from audio data.

[1505] A "face recognition algorithm" is a computational method for detecting a person's face from video data and extracting its features.

[1506] A "generative AI model" is an artificial intelligence algorithm that is trained on large datasets to generate specific information from audio or video data.

[1507] A "prompt" is an instruction or question that is input into a generative AI model to enable it to perform a specific task.

[1508] An "emotion score" is a numerical representation of the emotion identified from the analyzed audio and video data.

[1509] "Customer satisfaction" is an indicator of how customers rate their satisfaction with their experiences and services, and includes the Net Promoter Score (NPS).

[1510] "Response centralization" is the tendency of respondents in a particular cultural context to choose the middle or average answer.

[1511] A "dashboard" is an interface for centrally displaying data visualization and analysis results.

[1512] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the tendency for Japanese people to center their responses. This system collects audio and video data during customer interactions, analyzes the data using a generative AI model to identify customer sentiment, and calculates NPS based on the results.

[1513] System Overview

[1514] Terminal: A front-end device that handles customer service and collects audio and video data. Specific hardware features include high-sensitivity microphones and HD cameras. Examples include the Bose NC700 and Logitech C920.

[1515] Server: A backend device that analyzes data, calculates sentiment scores, and calculates NPS. The server has advanced computing power and large data storage capacity.

[1516] Users: Operate the system, understand customer satisfaction, and improve services. Users can use the dashboard to visually check the analysis results.

[1517] Program processing explanation

[1518] 1. Data Collection Phase

[1519] While the terminal is interacting with the customer, it uses a built-in microphone (e.g., a high-sensitivity microphone) and a camera (e.g., an HD camera) to collect audio and video data in real time. For example, the camera captures the customer's face frame by frame and records the video data in high resolution.

[1520] 2. Data transmission phase

[1521] The collected audio and video data is sent from the device to the server. Data transmission is via a secure protocol (e.g., SSL / TLS) to protect the integrity and privacy of the data.

[1522] 3. Data analysis phase

[1523] The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data to remove background noise and unwanted audio components. Specifically, the server analyzes the audio spectrum using an FFT (Fast Fourier Transform) and attenuates the noise components.

[1524] Next, the server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and creates a list of feature points.

[1525] 4. Emotion Identification Phase

[1526] The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) to identify the customer's emotions. For example, for audio data, a request is sent along with the prompt, "Please identify the customer's emotions from this audio data." The returned output includes emotions such as "elated," "calm," and "anxious."

[1527] For video data, the prompt is, "Analyze the customer's facial expressions in this video and identify their emotions." The model's output includes emotions such as "smile," "surprise," and "anger."

[1528] 5. Sentiment score and NPS calculation phase

[1529] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience. The NPS is calculated based on the emotion score, and by adjusting for the bias of intermediate responses using statistical methods, the tendency to center responses, which is unique to Japan, can be eliminated.

[1530] 6. Reporting Phase

[1531] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard visually displays the sentiment score history and detailed analysis results, making it easy for users to immediately identify problems and take measures to improve.

[1532] Specific examples

[1533] Example 1: Calculating NPS in a call center

[1534] Data collection: The device collects the voices of both the operator and the customer during the call using a built-in microphone, and records the operator's facial expressions using a camera.

[1535] Data transmission: Collected data is transmitted to the server in real time.

[1536] Emotion analysis: The server analyzes the operator's tone and pitch from the voice data and extracts changes in the operator's facial expressions from the video data.

[1537] Emotion Identification: The server identifies emotions using a generative AI model.

[1538] NPS calculation: Calculate the adjusted NPS based on the sentiment score and send the result to the device.

[1539] Displaying results: The device displays the NPS on the dashboard, which administrators can use to improve services.

[1540] Example 2: Customer service at the hotel front desk

[1541] Data collection: The device records the customer and staff interaction during check-in using its built-in microphone and camera.

[1542] Data transmission: The recorded data is securely transmitted to the server.

[1543] Sentiment Analysis: The server analyzes the audio and video data and uses a generative AI model to identify the customer's sentiment score.

[1544] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[1545] Results display: The device displays the NPS on a dashboard, which the hotel manager can use as feedback.

[1546] This invention makes it possible to precisely understand customer emotions based on audio and video analysis, more accurately reflecting NPS and improving service quality. Customer satisfaction can be specifically and rationally evaluated, and areas for improvement can be accurately identified.

[1547] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1548] Program processing steps

[1549] Step 1: Data collection phase

[1550] Input: Customer audio and video data

[1551] Output: Collected audio and video data

[1552] Description: As a front-end device for customer service, the terminal uses a built-in microphone (e.g., a high-sensitivity microphone) and camera (e.g., an HD camera) to collect audio and video data in real time during conversations with customers. When the customer starts speaking, the terminal automatically starts recording and saving the audio data as a digital audio signal and the video data as a digital video signal.

[1553] Step 2: Data transmission phase

[1554] Input: Collected audio and video data

[1555] Output: Audio and video data sent to the server

[1556] Description: The device packetizes collected audio and video data and sends it to the server via a secure protocol (e.g., SSL / TLS). Specifically, it buffers and timestams the data, consolidates it, encodes it, and transmits it using Internet Protocol.

[1557] Step 3: Data Analysis Phase (Audio)

[1558] Input: Audio data sent to the server

[1559] Output: Clear audio data after noise reduction

[1560] Description: The server applies a noise reduction filter (e.g., a noise suppression algorithm) to the received audio data. Specifically, it analyzes the audio spectrum using an FFT (Fast Fourier Transform), attenuates noise components, and generates audio data with improved voice clarity.

[1561] Step 4: Data analysis phase (video)

[1562] Input: Video data sent to the server

[1563] Output: Facial video data with extracted features

[1564] Description: The server uses a facial recognition algorithm (for example, OpenCV's face detection library) to detect the customer's face from the video data and extract facial expression features. Specifically, it uses Haarcascade or Dlib's facial recognition technology to capture eye and mouth movements and changes in facial expression, and lists the obtained feature points.

[1565] Step 5: Emotion Identification Phase

[1566] Input: Noise-reduced audio data and facial video data with extracted features

[1567] Output: Customer sentiment data

[1568] Description: The server inputs the analyzed audio and video data into a generative AI model (e.g., GPT-4) and identifies the customer's emotions using prompts. Specific prompts include, "Please identify the customer's emotions from this audio data" or "Please analyze the customer's facial expressions from this video and identify their emotions." The generative AI model identifies emotions based on the prompts and outputs emotional data such as "excited," "calm," "anxious," "smiling," "surprised," and "anger."

[1569] Step 6: Sentiment score and NPS calculation phase

[1570] Input: Customer sentiment data

[1571] Output: Sentiment score and adjusted NPS

[1572] Description: The server quantifies the identified emotions and expresses them as an emotion score. Based on this emotion score, the NPS is calculated and statistical methods (e.g., Z-score or standard deviation) are used to adjust for bias in intermediate responses and eliminate the tendency for responses to be centered, which is unique to Japan. As a result, a corrected NPS is calculated.

[1573] Step 7: Debriefing Phase

[1574] Inputs: Adjusted NPS, Sentiment Score

[1575] Output: A dashboard display of visualized NPS and sentiment scores

[1576] Description: The server sends the calculated NPS to the terminal and displays it to the user in the form of a dashboard or report. Specifically, the web server dynamically generates HTML and JavaScript to display it in the form of graphs and charts, allowing the user to intuitively understand the information. Based on this, the user can consider ways to improve the service and take specific actions.

[1577] (Application example 1)

[1578] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1579] Conventional systems for evaluating customer satisfaction (NPS) have difficulty accurately grasping customer satisfaction levels because they are unable to properly eliminate the tendency for Japanese customers to center their responses. Furthermore, the lack of a system for analyzing conversations in physical stores in real time and providing immediate feedback makes it difficult to quickly improve services.

[1580] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1581] In this invention, the server includes a means for collecting voice data and video data during a conversation with a customer, a means for using a generative AI model to analyze the collected voice data and video data to identify the customer's emotions, a means for calculating a customer's emotion score based on the analysis results, correcting the score to eliminate response centralization, and calculating an accurate customer satisfaction level, and a data collection means for evaluating the conversation between the customer and an employee in a physical store. This allows for an accurate NPS to be calculated immediately based on the customer's emotion evaluation, enabling prompt service improvements in the physical store.

[1582] "Customer" means a person to whom a service is provided, who purchases or uses goods or services.

[1583] "Auditory data" is information collected as sound, and is digital data of sound acquired by a sensor such as a microphone.

[1584] "Video data" refers to image data collected by visual sensors such as video cameras and cameras, and is composed of a series of still images divided into frames.

[1585] A "generative AI model" is an artificial intelligence model that has been trained using a large amount of learning data and has the ability to generate responses based on specific input data.

[1586] The "emotion score" is an index that numerically expresses the emotional state of the customer based on the analysis results of the audio data and video data.

[1587] "Response centralization" refers to the tendency for responses to be biased toward the middle of the evaluation scale in certain cultures and response patterns.

[1588] A "brick and mortar store" is a location that provides goods or services in person, either within a physical building or other location.

[1589] A system for implementing the present invention has a function for identifying customer emotions using audio and video data in order to accurately calculate customer satisfaction (NPS). The system includes the following components.

[1590] Hardware and Software Configuration

[1591] 1. Device:

[1592] To collect audio and video data, devices such as smartphones and smart glasses are used, which have built-in microphones and cameras and can collect data in real time.

[1593] 2. Server:

[1594] To analyze the collected audio and video data, a high-performance computer server is used. The server is equipped with the voice recognition library "Vosk" and the image recognition library "OpenCV," and these are used to analyze the data.

[1595] Emotion identification uses a generative AI model: a RoBERTa-based emotion classification model using the Hugging Face transforms library.

[1596] Data collection and transmission

[1597] The device collects audio and video data in real time during customer interactions, and the collected data is sent to the server via a secure protocol, such as Transport Layer Security (TLS), to protect the integrity and privacy of the data.

[1598] Data analysis

[1599] The server parses the data as follows:

[1600] Audio data analysis:

[1601] Apply noise reduction filters to get clear audio data. Filter to remove background noise.

[1602] It uses a speech recognition algorithm to convert the voice data into text, analyzes what the customer is saying, extracts features such as tone, pitch, and speed, and evaluates the sentiment.

[1603] Video data analysis:

[1604] A facial recognition algorithm is used to detect the customer's face from the video data, extracting facial features (e.g., eye and mouth movements, wrinkles, etc.) to identify emotions.

[1605] Sentiment score and NPS calculation

[1606] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The identified emotions are quantified and expressed as an emotion score. The NPS is calculated based on the emotion score and adjusted using statistical methods to eliminate the tendency for response centering, which is unique to Japan.

[1607] Reporting the results

[1608] The calculated NPS is returned to the device and displayed in the form of a dashboard and report. The dashboard visually displays the history of sentiment scores and detailed analysis results, allowing users to immediately identify problems and use the information to improve their services.

[1609] Specific examples

[1610] Example 1: Evaluation of cashier service at a physical store

[1611] Data collection: Smart glasses record the conversations and facial expressions between store staff and customers.

[1612] Data transmission: The recorded data is transmitted to the server via a secure protocol.

[1613] Sentiment analysis: The server analyzes voice tone and facial expressions to identify the customer's emotions.

[1614] NPS calculation: Calculates the adjusted NPS from the sentiment score and displays the results in real time.

[1615] View the results: See your NPS score displayed on the dashboard and take action to improve your customer experience.

[1616] Example prompts to input to the generative AI model

[1617] "Please identify customer emotions and calculate NPS based on audio and video data recorded from conversations between customers and store staff in-store. Specifically, I would like to analyze the tone of the voice and changes in facial expressions in the video to identify emotions such as 'excited' or 'calm'."

[1618] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1619] Step 1:

[1620] The device collects audio and video data during interactions with customers. The input is the interaction between the customer and the store clerk in the physical store, and the output is the collected audio file (e.g., customer_interaction.wav) and video file (e.g., customer_interaction.mp4). Specifically, the smart glasses' built-in microphone and camera are used to record the conversation between the customer and the store clerk, as well as their facial expressions.

[1621] Step 2:

[1622] The audio and video data collected by the device is sent to the server via a secure protocol such as TLS (Transport Layer Security). The input is the audio and video files collected in step 1, and the output is these data transferred to the server. Specifically, when sending data from the device to the server, an encrypted connection is maintained to protect the integrity and privacy of the data.

[1623] Step 3:

[1624] The server applies a noise reduction filter to the received audio data to generate clear audio data. The input is the transferred audio file, and the output is the audio data with reduced noise. Specifically, it uses the "Vosk" speech recognition library and applies a specific filtering algorithm to remove background noise.

[1625] Step 4:

[1626] The server analyzes the noise-reduced speech data and extracts features such as speech tone, pitch, and speed. The input is the noise-reduced speech data, and the output is the extracted speech features. Specifically, a speech signal processing library is used to calculate speech parameters that indicate customer emotions.

[1627] Step 5:

[1628] The server uses a facial recognition algorithm to detect customer faces for each frame of the video data it receives and extracts facial expression features. The input is the video file, and the output is the detected face image and its facial expression features. Specifically, it uses OpenCV to extract face regions from the video frames and classify each facial expression (e.g., smile, surprise, anger).

[1629] Step 6:

[1630] The server inputs the analyzed audio and video data into a generative AI model to identify the customer's emotions. The inputs are audio features and video features, and the output is an identified emotion score. Specifically, a RoBERTa-based emotion classification model using the Hugging Face transformers library is used to predict the customer's emotional state from each feature.

[1631] Step 7:

[1632] The server quantifies the identified emotion scores and performs statistical corrections to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the identified emotion score, and the output is the corrected NPS score. Specifically, statistical methods are used to adjust responses that are biased toward the median value, resulting in a more accurate NPS.

[1633] Step 8:

[1634] The server sends the calculated NPS score to the terminal and displays the results in the form of a dashboard or report. The input is the corrected NPS score, and the output is the NPS score displayed on the terminal and its detailed analysis results. Specifically, the server provides a GUI (graphical user interface) to visually display the NPS score and its analysis results.

[1635] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1636] This invention relates to a system for more accurately calculating customer satisfaction (NPS), with a particular focus on eliminating the response-centered tendency that is unique to Japanese people. Furthermore, by incorporating an emotion engine that recognizes user emotions in real time, this system evaluates customer satisfaction with even greater precision. This system collects audio and video data during customer interactions, analyzes the collected data using a generative AI model to identify customer emotions, and calculates NPS based on the results. Furthermore, by recognizing user emotions in real time through the emotion engine and storing them as emotion data, further analysis and service improvement are possible.

[1637] System Overview

[1638] Terminal: A front-end device that handles customer service, it collects audio and video data and has a built-in emotion engine that recognizes user emotion data in real time.

[1639] Server: A back-end device that performs data analysis, calculates sentiment scores, and calculates NPS.

[1640] User: Operates the system, understands customer satisfaction, and improves services.

[1641] Program processing explanation

[1642] 1. Data Collection Phase

[1643] During a conversation with a customer, the device uses a built-in microphone and camera to collect real-time audio and video data, and the camera records the customer's facial expressions frame by frame in high resolution.

[1644] The emotion engine recognizes the user's emotions in real time and stores them as emotion data, which is based on information obtained from voice tone and facial expressions.

[1645] 2. Data transmission phase

[1646] The collected audio, video, and emotion data is sent from the device to a server, and data transmission is always via a secure protocol to protect data integrity and privacy.

[1647] 3. Data analysis phase

[1648] The server applies a noise reduction filter to the received audio data to make the audio clearer, for example by removing background noise to make the conversation clearer.

[1649] For video data, a facial recognition algorithm is used to detect the customer's face in each frame and extract facial features, capturing subtle facial features such as eye movements and changes in the mouth.

[1650] 4. Emotion Identification Phase

[1651] The server inputs the analyzed audio and video data into a generative AI model to identify customer emotions. The generative AI model is trained using a large amount of past data sets, enabling highly accurate emotion identification.

[1652] Parameters such as tone, pitch, and speed are evaluated from the audio data to identify emotions such as "joy" or "anger."

[1653] The video data is used to evaluate changes in facial expressions and classify emotions such as smiles, surprise, and anger.

[1654] The accuracy of emotion identification is improved by taking into account user emotion data obtained from the emotion engine.

[1655] 5. Sentiment score and NPS calculation phase

[1656] The identified emotions are quantified and expressed as an emotion score, which allows for a more objective evaluation of the customer experience.

[1657] The server calculates the NPS based on the sentiment score and applies corrections to eliminate the tendency for Japanese respondents to be centered on their answers, for example, by using statistical methods to adjust for bias in intermediate answers.

[1658] 6. Reporting Phase

[1659] The calculated NPS is sent to the device and displayed to the user in the form of a dashboard or report. The dashboard also displays the sentiment score history and detailed analysis results, allowing users to quickly identify problems and take measures to improve.

[1660] Specific examples

[1661] Example 1: Calculating NPS in a call center

[1662] Data collection: The terminal collects the voices of the operator and the customer during the call, records the operator's facial expressions with a camera, and the emotion engine recognizes the operator's emotional data in real time.

[1663] Data transmission: These data are transmitted to the server in real time.

[1664] Emotion analysis: The server analyzes the collected data and identifies the emotion from the operator's tone and facial expression.

[1665] NPS calculation: Calculates the adjusted NPS based on the sentiment score and sends the result to the device.

[1666] Displaying results: The device displays the NPS on a dashboard, which administrators can use to improve services.

[1667] Example 2: Customer service at the hotel front desk

[1668] Data collection: The terminal records audio and video of the interaction between the customer and staff during check-in. The emotion engine also recognizes the staff's emotions in real time and stores them as emotion data.

[1669] Data transmission: Send the recorded data to the server.

[1670] Sentiment analysis: The server analyzes the generated AI model and identifies the customer's satisfaction as an emotion score.

[1671] NPS calculation: Calculate the corrected NPS from the sentiment score and send the result to the device.

[1672] Result display: The terminal displays the NPS, which the hotel manager can use as feedback.

[1673] This is a groundbreaking system that precisely understands customer emotions based on audio and video analysis and accurately reflects NPS to improve service quality. Furthermore, by grasping user emotions in real time through the emotion engine, even more advanced analysis and feedback become possible. In this way, it is possible to specifically and rationally evaluate customer satisfaction and accurately identify areas for improvement.

[1674] The processing flow will be explained below.

[1675] Step 1: Collect audio and video data

[1676] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with the customer.

[1677] Specifically, a microphone collects the conversational audio, while a camera records the customer's face frame by frame in high resolution.

[1678] Step 2: Real-time emotion recognition

[1679] The device's built-in emotion engine analyzes collected audio and video data in real time to recognize the user's emotions.

[1680] The system analyzes voice characteristics such as tone, pitch, and speed from audio data, and detects facial expressions and subtle movements from video data.

[1681] Step 3: Storing emotion data

[1682] The emotion engine recognizes emotion data in real time and stores it in internal storage, which is later used for detailed analysis.

[1683] The data stored includes the type of emotion recognized (e.g., joy, surprise, anger), intensity, and time stamp.

[1684] Step 4: Sending data to the server

[1685] The terminal transmits the collected audio data, video data, and stored emotion data all at once to the server.

[1686] The data is divided into data packets and transferred via a secure protocol.

[1687] Step 5: Receiving and Preprocessing Data

[1688] The server receives the data packets sent from the terminal.

[1689] It checks the integrity of the data packets and merges the audio and video data into their original format.

[1690] For audio data, a noise reduction filter is applied to remove background noise, and for video data, face detection and tracking is performed for each frame.

[1691] Step 6: Audio data analysis

[1692] The server inputs the voice data into a speech recognition model and converts it into text.

[1693] It extracts features such as voice tone, pitch, and speed, and uses a generative AI model to identify emotions.

[1694] Step 7: Video data analysis

[1695] The server inputs the video data frame by frame into a generative AI model to extract the customer's facial features.

[1696] Emotions are classified by analyzing features such as eye movements, changes in the mouth, and facial expressions.

[1697] Step 8: Aggregating and calculating sentiment scores

[1698] The server combines the results of the analysis of the audio and video data to generate a single overall emotion score.

[1699] Each feature is weighted and an overall evaluation is performed.

[1700] Step 9: Calculate your NPS

[1701] The server classifies customer experiences as "promoters," "passives," or "detractors" based on the sentiment score.

[1702] Based on the sentiment score, we calculate an accurate NPS and apply corrections to eliminate the tendency for responses to be centered, which is unique to Japan.

[1703] Step 10: Reporting results

[1704] The server sends the calculated NPS value and related analytical data to the device.

[1705] The NPS results received by the device are displayed to the user in the form of a dashboard or report.

[1706] Specific examples

[1707] Example 1: Calculating NPS in a call center

[1708] Step 1: The device collects the voice of the operator and the customer during the call, and records the operator's facial expressions with a camera.

[1709] Step 2: The emotion engine analyzes the operator's voice and facial expressions in real time to recognize emotions.

[1710] Step 3: The device stores the recognized emotion data in its internal storage.

[1711] Step 4: Send the collected data to the server.

[1712] Step 5: The server receives the data and performs audio and video preprocessing.

[1713] Step 6: Analyze the audio data and identify emotions.

[1714] Step 7: Analyze the video data and identify emotions from facial expressions.

[1715] Step 8: Combine the sentiment scores to generate a single overall sentiment score.

[1716] Step 9: Calculate the adjusted NPS based on the sentiment score.

[1717] Step 10: The device displays the NPS on the dashboard, which administrators can use to improve services.

[1718] Example 2: Customer service at the hotel front desk

[1719] Step 1: The device records audio and video of the customer and staff interaction during check-in.

[1720] Step 2: The emotion engine recognizes staff emotions in real time and stores the emotion data.

[1721] Step 3: Send the collected data to the server.

[1722] Step 4: The server receives the data and performs preprocessing.

[1723] Step 5: Noise reduction and analysis of the audio data.

[1724] Step 6: Analyze video data to identify customer sentiment.

[1725] Step 7: Combine the emotion scores to make an overall rating.

[1726] Step 8: Calculate the adjusted NPS from the sentiment score.

[1727] Step 9: The device displays the NPS, which the hotel manager uses as feedback.

[1728] Example 2

[1729] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] Previous systems have made it difficult to accurately assess customer satisfaction (NPS). In particular, it was difficult to quickly and accurately analyze audio and video data, and the tendency for response centering, which is unique to Japan, impacted NPS. Furthermore, it was inadequate to recognize customer sentiment in real time and immediately incorporate that data into analysis. This made it difficult to accurately grasp true customer satisfaction and link it to service improvements.

[1731] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for encrypting collected audio data and video data and transmitting them to the server using a secure protocol, means for performing noise reduction and audio feature extraction processing on the audio data in the server, means for performing facial recognition and extracting facial expression features on the video data in the server, and means for recognizing user emotions in real time using an emotion engine and storing them as emotion data. This enables secure processing of audio data and video data in real time and highly accurate identification of customer emotions. Furthermore, it enables accurate calculation of NPS, eliminating the tendency toward response centering that is unique to Japan, enabling rapid and accurate service improvements.

[1732] "Customer Satisfaction (NPS)" is an index that measures a customer's willingness to recommend a product or service to others, and is also known as the Net Promoter Score.

[1733] "Voice Data" refers to sound information collected using a microphone during a customer interaction.

[1734] "Video Data" refers to image information collected using a camera during an interaction with a customer.

[1735] A "generative AI model" refers to an artificial intelligence algorithm that is trained on large datasets and uses audio and video data to analyze customer sentiment.

[1736] An "emotion engine" refers to a software component that recognizes emotions in real time from voice tone and facial expressions.

[1737] "Emotion score" refers to an indicator that quantifies and objectively evaluates customer emotions.

[1738] "Correction to eliminate the tendency to center responses" refers to the process of using statistical methods to correct responses that tend to be biased toward intermediate evaluations, which are unique to Japan.

[1739] "Noise reduction" refers to the process of removing unnecessary background sounds from audio data and processing it into clear audio.

[1740] "Audio features" refers to attribute information necessary for analysis, such as tone, pitch, and speed, extracted from audio data.

[1741] "Facial recognition" refers to the technology of detecting and identifying human faces from video data frame by frame.

[1742] "Facial expression features" refer to attribute information such as eye movements and changes in the mouth that are extracted from video data to capture changes in facial expressions.

[1743] A "secure protocol" refers to a communication protocol that implements security measures such as encryption during data communication to ensure data protection.

[1744] The present invention relates to a system for calculating customer satisfaction (NPS) more accurately and efficiently. Specific embodiments of this system will be described below.

[1745] Hardware and software used

[1746] The device is equipped with a built-in microphone and camera to collect audio and video data during interactions with customers. It also has an emotion engine for real-time emotion recognition. The device encrypts the collected data and transmits it to a server using a secure protocol.

[1747] The server runs a generative AI model to analyze the received audio and video data. The analysis process involves noise reduction of the audio data, extraction of audio features, and facial recognition and extraction of facial features of the video data. Based on the generated emotion data, calculations are performed to calculate the emotion score and NPS.

[1748] Specific processing of the system

[1749] 1. Data Collection

[1750] The device uses a built-in microphone and camera to collect audio and video data in real time while interacting with a customer.

[1751] The device's emotion engine analyzes voice tone and facial expressions to generate and store emotion data in real time.

[1752] 2. Data Transmission

[1753] The data collected by the device is AES encrypted and sent securely to the server using the HTTPS protocol.

[1754] 3. Data Analysis

[1755] The server applies a noise reduction filter to the audio data to process it into clearer audio, and extracts audio features (e.g., tone, pitch, and speed).

[1756] The server applies a facial recognition algorithm to the video data and extracts facial features (e.g., eye movements and changes in the mouth).

[1757] 4. Emotion identification

[1758] The server uses the generative AI model to input the analyzed audio and video data to accurately identify the customer's emotions, and also integrates real-time emotional data to determine the most accurate emotional state.

[1759] 5. NPS calculation

[1760] The server quantifies the identified emotions and generates an emotion score. Based on the emotion score, the NPS is calculated and a statistical method is used to correct for the tendency for responses to be centered, which is unique to Japan.

[1761] 6.Result report

[1762] The server sends the calculated NPS to the device, which displays the results in the form of a dashboard or report.

[1763] Specific examples

[1764] Example 1: Calculating NPS in a call center

[1765] Data collection: The device collects the voices of both the operator and the customer during the call using a high-quality microphone, and records the operator's facial expressions frame by frame using a built-in camera. The emotion engine stores the operator's emotional data in real time.

[1766] Data transmission: Encrypted audio and video data is sent to the server using the HTTPS protocol.

[1767] Emotion analysis: The server inputs noise-reduced audio data into a generative AI model to identify emotions. Facial features are extracted and classified from video data.

[1768] NPS calculation: Calculate the corrected NPS based on the sentiment score and send the result to the device.

[1769] Displaying results: The device displays NPS and detailed analysis results on a dashboard, which administrators can use to improve services.

[1770] Example prompt sentence:

[1771] Please explain in detail the processing flow of a system that collects conversation data between agents and customers in a call center, analyzes agent emotions in real time using an emotion engine, and calculates NPS, including the specific operation of each processing step.

[1772] Example 2: Customer service at the hotel front desk

[1773] Data collection: The device records the interaction between the customer and the staff at check-in. The microphone collects audio data, and the camera collects high-resolution video data. The emotion engine recognizes and stores the staff's emotions in real time.

[1774] Data transmission: Encrypted data is sent to the server using a secure protocol.

[1775] Sentiment analysis: The server analyzes the audio and video data using a generative AI model to determine an emotion score.

[1776] NPS calculation: Calculate the corrected NPS based on the sentiment score and send it to the device.

[1777] Results display: The device displays a detailed NPS report on a dashboard for hotel managers to use for feedback.

[1778] Example prompt sentence:

[1779] Please explain the system that collects real-time audio and video data of customer interactions at hotel front desks, analyzes staff emotions using an emotion engine, and calculates emotion scores and NPS, including the specific operation of each processing step.

[1780] In this way, the present invention improves service quality by precisely understanding customer emotions based on audio and video analysis and calculating an accurate NPS. Real-time emotion analysis is possible, which allows for rapid feedback and the implementation of appropriate improvement measures.

[1781] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1782] Step 1: Data collection

[1783] The device uses a built-in microphone to collect voice data in real time while interacting with the customer.

[1784] The device uses a built-in high-resolution camera to record the customer's facial expressions frame by frame.

[1785] The device's emotion engine analyzes voice tone and facial expressions in real time and stores them as emotional data.

[1786] Input: Voice recording of conversation between customer and staff, and video of customer's facial expressions.

[1787] Output: Audio and video data, real-time emotion data.

[1788] Step 2: Send data

[1789] The audio and video data collected by the device, as well as real-time emotion data, are encrypted using AES.

[1790] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[1791] Input: Audio data, video data, and emotion data before encryption.

[1792] Output: The encrypted data sent to the server.

[1793] Step 3: Noise reduction of audio data

[1794] The server applies a noise reduction filter to the audio data it receives, removing background noise and producing clear audio.

[1795] Input: Encrypted audio data.

[1796] Output: Clear audio data after noise reduction.

[1797] Step 4: Extract audio features

[1798] The server extracts speech features such as tone, pitch, and speed from the noise-reduced speech data.

[1799] Input: Noise-reduced audio data.

[1800] Output: Audio feature data.

[1801] Step 5: Facial recognition of video data

[1802] The server applies a facial recognition algorithm to the video data it receives and detects the customer's face in each frame.

[1803] Input: Encrypted video data.

[1804] Output: Face recognition results and facial expression information per frame.

[1805] Step 6: Extraction of facial features

[1806] The server extracts facial expression changes (for example, eye movements or changes in the mouth) as features from the video data that has undergone facial recognition.

[1807] Input: Face-recognized video data.

[1808] Output: Facial expression feature data.

[1809] Step 7: Identify emotions

[1810] The server inputs the analyzed audio and video data using the generative AI model to identify customer emotions.

[1811] The server integrates this with real-time emotional data to determine the most accurate emotional state.

[1812] Input: Audio feature data, facial expression feature data, real-time emotion data.

[1813] Output: Identified emotion data.

[1814] Step 8: Sentiment score and NPS calculation

[1815] The server quantifies the identified emotion and calculates it as an emotion score.

[1816] The server calculates the NPS based on this emotional score and corrects for the tendency for responses to be centered, which is unique to Japan, using statistical methods.

[1817] Input: Identified emotion data.

[1818] Output: Sentiment score and adjusted NPS.

[1819] Step 9: Report your findings

[1820] The server sends the calculated NPS to the device.

[1821] The NPS sent by the device is displayed to the user in the form of a dashboard or report.

[1822] Users use this data to evaluate customer satisfaction and consider ways to improve services.

[1823] Input: Calculated NPS.

[1824] Output: NPS and detailed analysis results displayed in dashboards and reports.

[1825] (Application example 2)

[1826] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1827] Conventional systems for calculating customer satisfaction (NPS) have had difficulty eliminating the tendency for Japanese people to center their responses and accurately measure customer satisfaction. There was also a lack of means to recognize customer emotions in real time and immediately improve services. Furthermore, there were issues with the accuracy and practicality of systems that effectively collect and analyze customer voice and video data during interactions.

[1828] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data and video data during a conversation with a customer, means for analyzing the collected voice data and video data and using a generative AI model to identify the customer's emotions, means for calculating a customer's emotion score based on the analysis results and correcting it to eliminate response centralization to accurately calculate customer satisfaction, and means for performing the collection and analysis in real time via smart glasses. This makes it possible to recognize customer emotions with high accuracy in real time and accurately measure customer satisfaction. Furthermore, staff and managers can receive instant feedback and respond quickly to service improvements.

[1829] "Voice Data" refers to the audio information exchanged between a customer and a staff member during an interaction.

[1830] "Video data" refers to video information such as the faces and expressions of customers and staff during a conversation.

[1831] "Generative AI model" refers to an artificial intelligence model trained to analyze collected audio and video data and identify customer sentiment.

[1832] "Emotion score" refers to the customer's emotional state, expressed numerically as a result of analysis using a generative AI model.

[1833] "Correction to eliminate the tendency to center responses" refers to a statistical method for more accurately calculating customer satisfaction by eliminating the intermediate response tendency that is unique to Japanese people.

[1834] "Smart glasses" refers to a wearable eyeglass-type device that has the ability to collect audio and video data and analyze it in real time.

[1835] The present invention relates to a system for measuring customer satisfaction (NPS) in real time using smart glasses in a brick-and-mortar store. Specific embodiments thereof will be described below.

[1836] Hardware Configuration

[1837] The system includes smart glasses for collecting audio and video data, and a back-end server for analyzing the data. The smart glasses can be wearable devices such as Google Glass or Vuzix Blade. The back-end server has a high-performance processor and ample memory for processing and analyzing the collected data. It uses MySQL as the database and GPT-3 as the generative AI model.

[1838] Software Configuration

[1839] The following software modules are installed on the server:

[1840] Data collection module: Receives audio and video data transmitted from the smart glasses.

[1841] Noise reduction filter: Removes noise from audio data.

[1842] Audio feature extraction module: Extracts features from the audio data after noise removal.

[1843] Facial recognition module: Recognizes the customer's face for each frame of video data and extracts facial features.

[1844] Generative AI model module: A generative AI model to analyze collected data and identify customer sentiment.

[1845] Emotion score calculation module: Calculates the emotion score based on the analysis results.

[1846] NPS calculation module: Eliminates the tendency to center responses and calculates accurate customer satisfaction.

[1847] Data processing and calculation

[1848] 1. Data collection: Store staff wearing smart glasses collect audio and video data while interacting with customers. The data is sent to a server in real time.

[1849] 2. Noise reduction: Remove background noise from the audio data on the server to make the conversation clearer.

[1850] 3. Feature extraction: Extract audio features from the noise-removed audio data, and extract facial features for each frame from the video data.

[1851] 4. Emotion Identification: Using a generative AI model, the extracted feature data is analyzed to identify the customer's emotions.

[1852] 5. Emotion score calculation: The identified emotions are quantified and an emotion score is calculated.

[1853] 6. NPS calculation: NPS is calculated based on the emotional score, and the tendency for responses to be centered, which is unique to Japan, is corrected using statistical methods to calculate accurate customer satisfaction.

[1854] 7. Displaying results: The NPS is displayed on the smart glasses screen or on the store's management terminal, allowing staff and managers to receive feedback in real time.

[1855] Examples and prompts

[1856] As a concrete example, when a store staff member explains a product to a customer, audio and video data is collected and emotional data is generated in real time using an emotion engine. This data is sent from the smart glasses to a server, where it is analyzed by a generative AI model. An NPS is calculated based on the emotional score, and the results are displayed on the smart glasses. This allows staff to instantly grasp customer satisfaction levels and use the information to improve service.

[1857] Example prompt sentence:

[1858] "Using the audio and video data collected during the conversation, identify the customer's emotions and calculate the NPS. What adjustments are necessary to eliminate the tendency for responses to be centered, which is unique to Japan? Explain in detail how to calculate the emotion score and NPS."

[1859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1860] Step 1:

[1861] The terminal (smart glasses) collects audio and video data during a conversation with a customer. Specifically, audio and video information is captured in real time using the smart glasses' built-in microphone and camera. The input is the customer's audio and video during the conversation, and the output is digital audio and video files.

[1862] Step 2:

[1863] The audio and video data collected by the device is sent to the server via wireless communication (Bluetooth or Wi-Fi). The input is the collected audio and video files, and the output is the audio and video files stored on the server.

[1864] Step 3:

[1865] The server applies a noise reduction filter to the audio data it receives to process it into clear audio. Specifically, it applies an algorithm to remove background noise from the audio data. The input is the original audio data, and the output is clear audio data with noise removed.

[1866] Step 4:

[1867] The server extracts speech features from the noise-removed speech data. Specifically, it analyzes information such as tone, pitch, and speed of the speech. The input is the noise-removed speech data, and the output is the extracted speech features.

[1868] Step 5:

[1869] The server performs facial recognition and emotion classification for each frame of the video data it receives. Specifically, it uses a facial recognition algorithm to detect the customer's face and analyze their facial features. The input is the original video data, and the output is the extracted facial feature values.

[1870] Step 6:

[1871] The server inputs the extracted voice and facial features into a generative AI model to identify the customer's emotion. The input is the voice and facial features, and the output is the determined customer emotion.

[1872] Step 7:

[1873] The server quantifies the identified customer emotions and calculates an emotion score. Specifically, it assigns a certain score to each emotion based on the prediction results of the generative AI model. The input is the identified emotion, and the output is the emotion score.

[1874] Step 8:

[1875] The server calculates the NPS based on the sentiment score and corrects it using a statistical method to eliminate the tendency for responses to be centered, which is unique to Japan. The input is the sentiment score, and the output is the corrected NPS.

[1876] Step 9:

[1877] The server transmits the calculated NPS to the device (smart glasses) via wireless communication. The input is the corrected NPS, and the output is the NPS displayed on the smart glasses' display.

[1878] Step 10:

[1879] The user (store staff) checks the NPS displayed on the smart glasses display, instantly understanding customer satisfaction and helping to improve services. The input is the NPS displayed on the display, and the output is customer satisfaction information recognized by the user.

[1880] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1881] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1882] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1883] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1884] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1885] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1886] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1887] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1888] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1889] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1890] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1891] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1892] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1893] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1894] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1895] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1896] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1897] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1898] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1899] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion...

Claims

1. means for collecting audio and video data during a customer interaction; using a generative AI model to analyze collected audio and video data to identify customer sentiment; A means for calculating a customer sentiment score based on the analysis results, correcting the score to eliminate response centralization tendency, and calculating an accurate customer satisfaction level; A system including:

2. The system according to claim 1 , further comprising means for performing noise reduction and speech feature extraction processing on speech data collected during a conversation with a customer.

3. The system of claim 1 further comprising means for processing video data collected during a customer interaction for frame-by-frame facial recognition and emotion classification.

Images