System

Abstract

A system is provided.SOLUTION: A system comprising: data collection means for collecting facial data of a customer; data collection means for collecting voice data of the customer; facial expression recognition means for recognizing facial expressions of the customer from the collected facial data; voice analysis means for analyzing a tone of the customer from the collected voice data; emotion analysis means for determining an emotional state of the customer based on data obtained by the facial expression recognition means and the voice analysis means; and feedback generation means for suggesting a service improvement based on the determined emotional state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the restaurant industry, it is difficult to accurately grasp customer emotions and satisfaction, resulting in problems such as ineffective service improvement and feedback collection. Currently, there is a lack of effective means to accurately understand customer emotions and improve the dining out experience. For this reason, there is a need for a system that can grasp customer emotions in real time and help improve services and maximize the effectiveness of campaigns. [Means for solving the problem]

[0005] The present invention provides a system including a data collection means for collecting customer facial data, a data collection means for collecting customer voice data, a facial expression recognition means for recognizing the customer's facial expression from the collected facial data, a voice analysis means for analyzing the customer's tone of voice from the collected voice data, an emotion analysis means for determining the customer's emotional state based on the data obtained by the facial expression recognition means and the voice analysis means, and a feedback generation means for proposing service improvement measures based on the determined emotional state, thereby making it possible to grasp customer emotions in real time and provide accurate feedback and improvement measures to improve the customer experience.

[0006] "Data Collection Instruments" refers to the equipment and processes used to capture Customer facial and voice data.

[0007] "Facial Expression Recognition Means" refers to technologies and processes for analyzing a customer's facial expressions from collected facial data and recognizing specific emotional states (e.g., joy, anger, sadness, etc.).

[0008] "Voice Analysis Methods" refers to technologies and processes for analyzing the tone, pitch, speed, etc. of a customer's voice from collected voice data and recognizing a particular emotional state (e.g., excitement, calm, etc.).

[0009] "Emotion Analysis Means" refers to techniques and processes for integrating data obtained by facial expression recognition means and voice analysis means to determine the overall emotional state of a customer.

[0010] "Feedback generator" refers to techniques and processes for automatically suggesting and providing service improvements to management based on the determined emotional state.

[0011] "Transmission means" refers to the technology and process for transmitting collected facial and audio data to a server using a secure communication protocol.

[0012] "Data Storage Means" refers to the technology and processes for storing analyzed emotion data in a database for later searching and comparison. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly consists of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means.

[0035] Collection of customer data

[0036] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with noise-canceling functions to eliminate background noise and record customer voices clearly.

[0037] Sending data

[0038] Device: Collected face and voice data is sent to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0039] Data preprocessing and analysis

[0040] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0041] Emotion determination

[0042] Server: The preprocessed data is then passed to the emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0043] Data storage and analysis

[0044] Server: The determined emotion data is stored in a database and analyzed by an emotion analysis tool, comparing it with past data. This allows trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0045] Generate feedback

[0046] Server: Based on the stored emotion data, the system uses feedback generation methods to suggest optimal service improvement measures, such as "We should increase the number of staff during lunch hours" or "We should provide more detailed explanations about certain menu items."

[0047] Alerts and Reports

[0048] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator when certain conditions are met, enabling prompt action in real time. Aggregated reports of emotion data are also periodically created and distributed to administrators. These reports include weekly and monthly emotion trends, customer satisfaction trends, and suggestions for improvement.

[0049] Specific examples

[0050] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0051] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0052] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0053] In this way, the system can understand customer sentiment in real time and significantly improve the dining experience.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] Terminal: In-store cameras capture customer facial data in real time. Specifically, the camera takes a certain number of frames per second and performs image processing to detect the customer's face. For example, the OpenCV library is used to detect faces and extract the face area from each frame.

[0057] Step 2:

[0058] Device: A microphone installed in the store collects customer voice data. Specifically, a noise-canceling microphone records customer conversations and saves the voice data in a format (e.g., WAV file) using an audio processing library (e.g., PyAudio).

[0059] Step 3:

[0060] Device: Collected face and voice data is sent to the server in batches (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it is encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server.

[0061] Step 4:

[0062] Server: Preprocesses the received face data using image processing libraries (e.g., OpenCV and Dlib) to identify facial regions and extract key feature points (e.g., the positions of the eyes, nose, and mouth). The feature points are normalized and converted into a format suitable for input to the deep learning model.

[0063] Step 5:

[0064] Server: Preprocesses the received audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and obtain the frequency spectrum. It then extracts features such as pitch, tone, and speed and converts them into input data for sentiment analysis.

[0065] Step 6:

[0066] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0067] Step 7:

[0068] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0069] Step 8:

[0070] Server: Integrates emotional data obtained from facial expression recognition and voice analysis to determine the overall emotional state of the customer, which is then stored in a database in a format that can be searched and compared later.

[0071] Step 9:

[0072] Server: Analyzes the integrated sentiment data to extract trends and patterns, for example, analyzing customer reactions to specific times of day or menu items, and generating statistics.

[0073] Step 10:

[0074] Server: Generates feedback based on emotional data. Specifically, it automatically creates improvement suggestions such as "increase staff during this time" or "add explanations to specific menu items."

[0075] Step 11:

[0076] Server: Sends real-time alerts to administrators when certain conditions are met (e.g., a large number of customers are unhappy). Alerts can be sent, for example, via email or SMS.

[0077] Step 12:

[0078] Server: Periodically creates aggregate reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement, providing reference for administrators to take appropriate measures.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] Conventionally, restaurants have lacked a means to quickly and accurately grasp customer sentiment in order to improve customer service. This can result in lower customer satisfaction and delays in service improvements, potentially affecting the restaurant's reputation and revenue. Furthermore, the collection and analysis of customer feedback is often done manually, limiting the accuracy and speed of the analysis results. This invention solves these problems and provides a system for grasping customer sentiment in real time and utilizing it to improve service.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes means for collecting customer facial data, means for collecting customer voice data, means for recognizing customer facial expressions from the collected facial data, means for analyzing customer tones from the collected voice data, means for determining the customer's emotional state based on data obtained by the facial expression recognition means and the voice analysis means, means for proposing service improvement measures based on the determined emotional state, means for sending an alert to a manager when the customer's emotional data meets certain conditions, and means for periodically creating and distributing a summary report of the emotional data to the manager. This makes it possible to accurately grasp customer emotions in real time and improve services quickly and effectively.

[0084] "Customer" refers to a user who visits a restaurant and uses the service.

[0085] "Facial data" refers to image information of a customer's face captured by an image capture device such as a camera.

[0086] "Voice data" refers to audio information of customer voices and conversations recorded using a voice capture device such as a microphone.

[0087] "Means" refers to any device or method, or combination thereof, used to accomplish a particular purpose.

[0088] "Facial expression" refers to facial expressions that change depending on the movement of a person's facial muscles.

[0089] "Tone" refers to vocal characteristics such as pitch, volume, and rhythm.

[0090] "Emotional state" refers to a specific emotion a customer feels, such as joy, anger, or sadness.

[0091] "Sentiment analysis" refers to the technology of determining customer emotions from collected data.

[0092] "Service Improvement Measures" refers to specific measures or action plans proposed to increase customer satisfaction.

[0093] An "alert" is a notification or warning to an administrator when a pre-defined condition is met.

[0094] An "aggregate report" refers to a report that analyzes and organizes collected data and summarizes the results.

[0095] "Means and" refers to a combination in which the aforementioned means and methods work together to perform a single function.

[0096] The present invention relates to a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly comprises a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means. Specific examples are shown below.

[0097] Collection of customer data

[0098] Device: High-resolution cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture video at 30 frames per second in 1080p resolution. The microphones are noise-canceling, eliminating background noise and allowing customers' voices to be recorded clearly.

[0099] Sending data

[0100] Device: The collected face and voice data is sent to the server using a secure communication protocol (e.g., HTTPS) with AES-256 encryption, which prevents unauthorized access and tampering of the data and protects privacy.

[0101] Data Preprocessing

[0102] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, the audio processing library SciPy is used to perform a Fourier transform and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0103] Emotion determination

[0104] Server: The preprocessed data is passed to the emotion analysis means. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0105] Data storage and analysis

[0106] Server: The determined emotion data is stored in a database and analyzed against past data, allowing trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0107] Generate feedback

[0108] Server: Based on the stored emotion data, the server uses a feedback generator to suggest optimal service improvement measures. For example, specific suggestions such as "We should increase the number of staff during lunchtime" or "We should provide more detailed explanations about specific menu items" are automatically generated.

[0109] Alerts and Reports

[0110] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator if certain conditions are met. For example, if a customer's "anger" emotion exceeds a certain threshold, an alert is sent immediately. In addition, aggregated reports of emotion data are periodically created and delivered to the administrator. The reports include weekly and monthly emotion trends, changes in customer satisfaction, and suggestions for improvement.

[0111] Specific examples

[0112] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0113] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0114] Server: Feedback will be generated suggesting "serve this menu item at dinner time as well." Based on this trend, the manager can receive a report saying "customer satisfaction is high at lunchtime."

[0115] Prompt Sentence Examples

[0116] "Cameras and microphones in the store captured customer smiles and positive conversations. Could you please outline a program that analyzes this data to determine customer sentiment and generate feedback?"

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1: Collect customer data

[0119] Terminal: A high-resolution camera installed in the store activates and collects facial data of customers in the store in real time. As input, it captures video at 30 frames per second at 1080p resolution. As output, it generates video frames containing facial features. At the same time, a noise-canceling microphone records the customer's conversation and collects its audio data. As input, the customer's speech is collected in real time, and as output, clear audio data with noise removed is provided.

[0120] Step 2: Sending data

[0121] Terminal: The collected face and voice data is sent to the server using the secure communication protocol HTTPS. As input, face and voice data are provided, and as output, encrypted data packets are sent to the server. During this process, the data is encrypted using the AES-256 encryption method to ensure security during transmission.

[0122] Step 3: Data Preprocessing

[0123] Server: Data arriving at the server is first preprocessed. The transmitted face data and audio data are provided as input. For the face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For example, the distance from one eye corner to the other and the shape of the mouth are output as numerical data. For the audio data, a Fourier transform is performed using the SciPy library to calculate the frequency spectrum and feature quantities (e.g., pitch, volume, and tone). As the output of this processing, a dataset for each feature point is generated.

[0124] Step 4: Determine the emotion

[0125] Server: The preprocessed data is fed into the emotion analysis model. Facial feature points and audio features are provided as input. For facial data, a convolutional neural network (CNN) model is used to identify emotions such as "happiness," "anger," and "sadness" from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) model is used to accurately determine emotions from vocal tones. The output is the detected emotion label and its confidence level.

[0126] Step 5: Save your data

[0127] Server: The determined emotion data is stored in a database and used for analysis. As input, emotion labels and their confidence levels are provided. This data is recorded in the database as time-series data. As output, emotion data including time information is stored in the database.

[0128] Step 6: Generate feedback

[0129] Server: Based on the stored emotion data, the feedback generation means creates service improvement proposals. The most recent emotion data and past emotion data are provided as input. Based on comparative analysis with past data, specific trends and patterns, such as "certain menu items are highly rated at lunchtime," are derived. Specific service improvement measures (e.g., improving the menu, increasing the number of staff, etc.) are generated as output.

[0130] Step 7: Alerts and Reports

[0131] Server: Based on the results of sentiment analysis, an alert is sent to the administrator when certain conditions are met. Real-time sentiment data is provided as input. For example, if a customer's "anger" sentiment exceeds a certain threshold, an alert is sent immediately. As output, a notification is generated for the administrator. In addition, aggregated reports of sentiment data are periodically created and delivered to the administrator. As output, reports are created that include weekly and monthly sentiment trends, customer satisfaction levels, and improvement suggestions.

[0132] The above are the specific processing steps of this system.

[0133] (Application example 1)

[0134] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] In traditional restaurants, there were limited ways to understand customer satisfaction and dissatisfaction in real time, which often delayed the timing of service improvements. It was also difficult to quantitatively analyze customer sentiment and derive specific improvement measures. Furthermore, there was a lack of alert functions to respond quickly when specific problems occurred.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0137] In this invention, the server includes a feedback generation means for proposing service improvement measures based on the determined emotional state, an alert means for notifying staff or managers of an alert when a specific emotion is recognized frequently, and a data analysis means for analyzing past customer data and generating periodic reports. This makes it possible to grasp customer emotions in real time and quickly propose appropriate service improvement measures. Furthermore, when a specific problem occurs, an alert is sent immediately, encouraging a prompt response. Furthermore, the periodic reports can be compared with past data to derive more accurate service improvement measures.

[0138] "Data collection means" refers to a device or group of devices for collecting face data and voice data of customers.

[0139] "Facial expression recognition means" refers to an algorithm or device for recognizing customer facial expressions from collected facial data.

[0140] The "voice analysis means" refers to an algorithm or device for analyzing the customer's tone of voice from collected voice data.

[0141] The "emotion analysis means" refers to an algorithm or device for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means.

[0142] A "feedback generator" is an algorithm or device that suggests service improvements based on the determined emotional state.

[0143] An "alert means" is a device or group of devices that alerts staff or managers when a specific emotion is recognized frequently.

[0144] "Data analysis tools" are algorithms or devices used to analyze historical customer data and generate periodic reports.

[0145] The "transmission means" refers to a device or group of devices for transmitting the collected face data and voice data using a secure communication protocol.

[0146] The "data storage means" refers to a device or group of devices for storing the emotional states obtained from the facial expression recognition means and the voice analysis means in a database.

[0147] The "analysis means" refers to an algorithm or device for comparing and analyzing new data with past data stored in the data storage means.

[0148] To implement this invention, a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, an alert means, and a data analysis means are mainly required, as well as a transmission means for securely transmitting the collected data, and a data storage means and analysis means for storing and analyzing the data.

[0149] Data collection

[0150] The device uses high-resolution cameras and noise-canceling microphones installed in the store to collect customer facial and voice data in real time, allowing for accurate capture of customer facial expressions and tone of voice.

[0151] Data transmission

[0152] Facial and audio data collected on the device is transmitted to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0153] Data Preprocessing and Analysis

[0154] The data that arrives at the server is first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points. For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features.

[0155] Emotion analysis

[0156] The preprocessed data is then passed to an emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This analysis allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0157] Feedback and Alerts

[0158] The server then uses a feedback generator to generate optimal service improvement measures based on the determined emotional state. For example, specific suggestions such as "We need to hire more staff" or "We should provide more detailed explanations for certain menu items" are generated. Additionally, an alerting mechanism immediately notifies staff and managers when a specific emotion is detected frequently.

[0159] Data Storage and Analysis

[0160] The emotional data is stored in a database and compared with past data using data analysis tools. This allows trends and patterns, such as customer reactions to specific time periods or menu items, to be visualized. Furthermore, aggregated reports of the emotional data are periodically created and distributed to managers.

[0161] Specific examples

[0162] For example, suppose a customer enters a restaurant during lunchtime, and the camera captures their smile and the microphone records their positive conversation. This data is sent to a server, where a deep learning model determines their happiness. Based on this information, feedback is generated that says "This customer is satisfied," and the information is shared with staff. This data is also stored in a database and used to improve service in the future.

[0163] Here are some example prompts to input to a generative AI model:

[0164] "The camera captures a smiling customer entering your restaurant during lunchtime, and the microphone records his positive conversation. Based on this situation, generate feedback and improvement suggestions if the analysis determines that he is happy."

[0165] This invention makes it possible to grasp customer sentiment in real time and quickly propose appropriate service improvement measures. It also sends an immediate alert when a specific problem occurs, encouraging a prompt response. Furthermore, periodic reports can be compared with past data to derive more accurate service improvement measures.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The terminal uses a high-resolution camera and a noise-canceling microphone installed in the store to collect customer facial and voice data in real time. The inputs for collection are camera images and microphone audio. To obtain this data clearly, the camera recognizes the customer's face and the microphone removes background noise. The output is the customer's facial image data and voice data.

[0169] Step 2:

[0170] The device sends the collected face data and voice data to the server using a secure communication protocol (e.g., HTTPS). The input is the face image data and voice data collected in step 1. The data is encrypted and sent to the server over the network. The output is the face image data and voice data that arrives at the server.

[0171] Step 3:

[0172] The server preprocesses the received facial image data using an image processing library (e.g., OpenCV). The input is the facial image data received by the server. Facial feature points (e.g., the positions of the eyes, nose, and mouth) are extracted through image processing. The output is the extracted facial feature points.

[0173] Step 4:

[0174] The server preprocesses the incoming audio data using an audio processing library (e.g., SciPy). The input is the audio data arriving at the server. The audio data is Fourier transformed and the frequency spectrum and features (e.g., pitch, volume, and tone) are analyzed. The output is the analyzed audio features.

[0175] Step 5:

[0176] The server analyzes emotions from the preprocessed facial data using a convolutional neural network (CNN) model. The input is the facial feature points extracted in step 3. The CNN model determines emotions such as "happiness," "anger," and "sadness" with high accuracy from these feature points. The output is the analyzed emotion data.

[0177] Step 6:

[0178] The server analyzes the preprocessed speech data for emotion using a recurrent neural network (RNN) model. The input is the speech features analyzed in step 4. The RNN model determines the emotion from the speech data. The output is the analyzed emotion data.

[0179] Step 7:

[0180] The server uses the feedback generation means to propose optimal service improvement measures based on the emotional state of the face data and voice data obtained from the emotion analysis means. The input is the emotional data obtained in steps 5 and 6. The algorithm generates specific service improvement proposals. The output is the proposed service improvement measures.

[0181] Step 8:

[0182] The server uses an alerting means to notify staff or administrators when a specific emotion is recognized frequently. The input is emotion data from the emotion analysis means. An alert is generated based on specific conditions and a notification is sent. The output is an alert notification to staff or administrators.

[0183] Step 9:

[0184] The server stores the emotion data in a database and performs comparative analysis with past data using data analysis means. The input is the emotion data obtained in steps 7 and 8. This allows analysis of customer reactions to specific time periods and menu items, and visualizes trends. The output is the analysis results and data stored in the database.

[0185] Step 10:

[0186] The server periodically generates an aggregate report of the emotion data and delivers it to the administrator. The input is the emotion data saved in step 9. The aggregate report includes weekly and monthly emotion trends, customer satisfaction trends, improvement suggestions, etc. The output is the emotion data report delivered to the administrator.

[0187] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0188] The present invention is a system for understanding customer emotions in real time at restaurants, helping to improve service and feedback accuracy. This system is composed of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, a transmission means, a data storage means, and an emotion engine.

[0189] Collection of customer data

[0190] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with background noise reduction.

[0191] Sending data

[0192] On the device: The collected face and voice data is sent to the server in batches (e.g., every 10 seconds). The data is encrypted using a secure communication protocol (e.g., HTTPS).

[0193] Data preprocessing and analysis

[0194] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, a Fourier transform is performed using an audio processing library (e.g., SciPy) to analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0195] Emotion assessment and integration

[0196] Server: The pre-processed data is first processed using facial expression recognition means to determine customer emotions using a deep learning model (e.g., CNN). Similarly, audio analysis means is used to determine emotions from audio data using a machine learning model (e.g., RNN). These data are then integrated by the emotion engine to determine the overall customer emotional state.

[0197] Data storage and analysis

[0198] Server: The determined emotion data is stored in a database and compared with past data for analysis. Here, customer reactions to specific times of day or menu items are analyzed, and trends and patterns are visualized.

[0199] Generate feedback

[0200] Server: Based on the sentiment data, the feedback generator automatically suggests optimal service improvement measures, such as "increase staffing during certain times" or "add explanations for certain menu items."

[0201] Alerts and Reports

[0202] Server: Based on the results of the emotion data analysis, real-time alerts are sent to administrators when certain conditions are met. Aggregated reports of emotion data are also periodically created and distributed to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement.

[0203] Specific examples

[0204] Device: The in-store camera captures the customer's smile, and the microphone simultaneously records the customer's positive conversation. These data are sent to the server, and feature points of the video and audio are extracted.

[0205] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0206] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0207] In this way, the system can grasp customer sentiment in real time and provide accurate feedback and solutions to improve the dining experience.

[0208] The processing flow will be explained below.

[0209] Step 1:

[0210] Terminal: A camera installed in the store captures customer facial data in real time. Specifically, the camera takes a certain number of frames per second (e.g., 30 frames per second) and detects facial regions using a face detection algorithm (e.g., Haar cascade). The detected facial regions are segmented and passed to subsequent processing steps.

[0211] Step 2:

[0212] Device: A microphone installed in the store collects customer voice data. Specifically, the microphone records the voice at a certain sample rate (e.g., 44.1 kHz), and the voice data is processed using noise cancellation and then saved in an audio file format (e.g., WAV file).

[0213] Step 3:

[0214] Device: Sends face and voice data to the server in batches of a certain size (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it encrypts the data using a secure communication protocol (e.g., HTTPS) and sends it to the server.

[0215] Step 4:

[0216] Server: Preprocesses the face data. Specifically, it uses an image processing library (e.g., OpenCV) to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). This process converts the data into a format that can be used by the deep learning model. The extracted feature points are also normalized and passed on to the next step.

[0217] Step 5:

[0218] Server: Preprocesses the audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and analyze the frequency spectrum. It also extracts features such as pitch, tone, and speed, and converts them into input data for emotion analysis.

[0219] Step 6:

[0220] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0221] Step 7:

[0222] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0223] Step 8:

[0224] Server: The emotion data obtained from facial expression recognition and voice analysis is integrated into the emotion engine to determine the overall emotional state of the customer. Specifically, the emotion engine integrates the facial expression data and voice data to calculate a single integrated emotion score. This information is stored in a database.

[0225] Step 9:

[0226] Server: Analyzes the integrated sentiment data and extracts trends and patterns. For example, analyzes customer reactions to specific times or menu items and generates statistical information. Data analysis tools (e.g., Pandas, NumPy) are used for the analysis.

[0227] Step 10:

[0228] Server: Generates feedback based on emotion data. Specifically, the feedback generator automatically creates specific improvement suggestions, such as "increase staff during this time period" or "add explanations for specific menu items."

[0229] Step 11:

[0230] Server: Sends real-time alerts to administrators when certain conditions are met. For example, if customer satisfaction is low or if there is a high level of dissatisfaction with a particular menu item, an alert will be sent to the administrator via email or SMS.

[0231] Step 12:

[0232] Server: Periodically creates aggregated reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement. The reports are generated in PDF format and sent via email.

[0233] Example 2

[0234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] Restaurants are required to understand the emotional state of customers in real time and use that information to improve and optimize the quality of their services. However, conventional systems collect, analyze, store, and generate feedback from facial and voice data separately, making efficient and integrated operation difficult.

[0236] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image capture means for collecting customer facial data, an audio collection means for collecting customer voice data, a transmission means for transmitting the collected facial data, a transmission means for transmitting the collected voice data, an image processing means for preprocessing the collected facial data, an audio processing means for preprocessing the collected voice data, an expression analysis means for recognizing the customer's facial expression from the preprocessed facial data, an audio analysis means for analyzing the customer's tone of voice from the preprocessed audio data, an emotion analysis means for integrating data obtained from the expression analysis means and the audio analysis means to determine the customer's emotional state, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to efficiently integrate the customer's emotional state in real time and quickly suggest optimal service improvement measures.

[0237] "Image Capture Device" means a device used to collect customer facial data in real time.

[0238] An "audio collection means" is a device used to collect customer audio data.

[0239] "Transmission means" refers to a communication device or protocol for transmitting collected data to a server.

[0240] "Image processing means" refers to software or hardware for preprocessing collected face data and extracting feature points.

[0241] The "acoustic processing means" refers to software or hardware for preprocessing collected audio data and analyzing its features.

[0242] "Facial expression analysis means" refers to algorithms or models used to recognize customer emotions from pre-processed facial data.

[0243] "Voice analysis means" refers to algorithms or models for analyzing the customer's tone of voice from pre-processed voice data.

[0244] The "emotion analysis means" is a system for integrating data obtained from the facial expression analysis means and the voice analysis means to determine the emotional state of the customer.

[0245] The "feedback generation means" is a mechanism for suggesting service improvements based on the determined emotional state.

[0246] The present invention is a system for understanding customer emotions in real time at restaurants and optimizing and improving service quality based on that information. This system is composed of an image capture means, an audio collection means, a transmission means, an image processing means, an audio processing means, a facial expression analysis means, a voice analysis means, an emotion analysis means, and a feedback generation means.

[0247] First, cameras and microphones installed in the store function as image capture and audio collection devices. The high-resolution cameras collect customer facial data in real time, while the microphones record customer conversations. The cameras capture details from the moment customers enter the store to the moment they smile, while the microphones record the conversations while filtering out surrounding noise.

[0248] This data is then sent from the device to the server in batches (e.g., every 10 seconds) using a secure communication protocol (e.g., HTTPS) to encrypt and transmit the data safely. This process reduces the risk of data eavesdropping or tampering.

[0249] The data that arrives at the server is first preprocessed using image processing and audio processing means. Specifically, an image processing library (e.g., OpenCV) is used to extract facial feature points (the positions of the eyes, nose, and mouth, etc.), and an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data, analyzing the frequency spectrum and features (pitch, volume, tone, etc.).

[0250] The preprocessed data is then subjected to emotion recognition using facial expression analysis means within the server. This analysis uses a deep learning model (e.g., CNN) to determine the customer's emotion from the image data. Similarly, emotion is determined from the voice data using a machine learning model (e.g., RNN) through the voice analysis means. The results of both analyses are integrated by the emotion analysis means to comprehensively determine the customer's final emotional state.

[0251] The emotional state determined is stored in a database and compared with past data using analytical tools. This analysis visualizes customer reactions to specific time periods and menu items, highlighting trends and patterns. For example, a trend such as "a particular pasta menu item is popular during lunchtime" can be analyzed.

[0252] Finally, the feedback generation means automatically proposes service improvement measures based on the emotional data. For example, a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well" may be automatically generated. Furthermore, when certain conditions are met, an alert is sent to the manager in real time, and a summary report of the emotional data is periodically created and distributed to the manager.

[0253] As a concrete example, suppose an in-store camera captures a customer's smile, while a microphone simultaneously records a customer's positive conversation. This data is sent to a server, where feature points from the video and audio are extracted. Using CNN and RNN models, the smile and positive tone of voice are used to determine the emotion of "happiness." The recognized emotion data is then stored in a database, and analysis derives a trend, such as "a particular lunchtime menu item is popular." As feedback, a suggestion is generated to "serve this menu item at dinnertime as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0254] The system allows for real-time understanding of customer sentiment and provides targeted feedback and improvements to improve the dining experience.

[0255] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0256] Step 1:

[0257] Terminal: A camera installed in the store captures customers' faces in real time. Specifically, the camera captures high-resolution video and outputs each frame as image data. The input is a video of customers in the store, and the output is image data containing the customer's face.

[0258] Step 2:

[0259] Terminal: A microphone records customer conversations in real time. Noise-canceling technology is used to remove background noise and obtain clean audio data. The input is the in-store audio environment, and the output is audio data with the customer's conversations emphasized.

[0260] Step 3:

[0261] Terminal: The collected face data and audio data are batch processed (e.g., every 10 seconds) and sent to the server. When sent, the data is encrypted using a secure communication protocol (e.g., HTTPS). The input is image data and audio data, and the output is the encrypted transmission data.

[0262] Step 4:

[0263] Server: Receives the transmitted face data and performs preprocessing using an image processing library (e.g., OpenCV). Specifically, the processing involves extracting facial feature points (e.g., the positions of the eyes, nose, and mouth). The input is encrypted image data, and the output is preprocessed image data with feature points extracted.

[0264] Step 5:

[0265] Server: Receives the transmitted audio data and performs preprocessing using an audio processing library (e.g., SciPy). Performs a Fourier transform on the audio data and analyzes the frequency spectrum and features (e.g., pitch, volume, tone). The input is encrypted audio data, and the output is preprocessed audio data with the frequency spectrum and features analyzed.

[0266] Step 6:

[0267] Server: Using the preprocessed image data, a deep learning model (e.g., CNN) is run to analyze facial expressions and determine the customer's emotions. Specifically, the model outputs emotion labels such as "joy," "anger," and "sadness" from the input image data. The input is the preprocessed image data, and the output is the emotion label.

[0268] Step 7:

[0269] Server: Using the preprocessed voice data, a machine learning model (e.g., RNN) is run to analyze the voice and determine the emotion from the customer's tone of voice. Specifically, the model outputs an emotion label such as "satisfied" or "dissatisfied" from the input voice data. The input is the preprocessed voice data, and the output is the emotion label.

[0270] Step 8:

[0271] Server: The emotion analysis means integrates the emotion labels obtained from the facial expression analysis means and the voice analysis means to determine the overall emotional state of the customer. For example, if the facial expression of "happiness" matches the tone of voice of "satisfied," the overall judgment is "very satisfied." The input is the emotion labels of facial expressions and voice, and the output is the integrated emotional state.

[0272] Step 9:

[0273] Server: Stores the determined emotional state in a database. It then performs comparative analysis with past data, analyzes customer reactions to specific time periods and menu items, and visualizes trends and patterns. For example, it analyzes the trend that "a specific lunchtime menu item is popular." The input is the integrated emotional state data, and the output is the visualized results of trends and patterns.

[0274] Step 10:

[0275] Server: The feedback generation means automatically proposes optimal service improvement measures based on the emotion data. For example, it generates a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well." The input is the analysis results and emotion data, and the output is a service improvement proposal.

[0276] Step 11:

[0277] Server: When a specific condition is met (for example, when customer satisfaction declines), it sends an alert to the administrator in real time. It also periodically creates aggregate reports of emotion data and delivers them to the administrator. The input is emotion data that meets the condition, and the output is the alert notification and aggregate reports.

[0278] (Application example 2)

[0279] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] Traditional customer service in restaurants relies heavily on staff experience and intuition, making it difficult to accurately grasp customer emotions and satisfaction. Furthermore, there are limited means to grasp customers' real-time emotional states and provide appropriate feedback based on them, resulting in a lack of speed in service improvement. Therefore, in order to improve customer satisfaction and service quality, real-time emotion analysis and immediate feedback based on the results are required.

[0281] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting customer facial data, a data collection means for collecting customer voice data, a facial expression recognition means for recognizing the customer's facial expression from the collected facial data, a voice analysis means for analyzing the customer's tone of voice from the collected voice data, an emotion analysis means for determining the customer's emotional state based on the data obtained by the facial expression recognition means and the voice analysis means, a display means for displaying the determined emotional state in real time, a feedback presentation means for providing feedback to the staff based on the emotional state displayed in real time, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to accurately grasp the customer's emotional state in real time and provide immediate feedback based on that.

[0282] The "data collection means" is a mechanism for collecting face data and voice data of customers.

[0283] The "facial expression recognition means" is a mechanism that analyzes the facial expression of the customer from the collected facial data and recognizes that expression.

[0284] The "voice analysis means" is a mechanism that analyzes collected voice data to analyze the customer's voice tone and other characteristics.

[0285] The "emotion analysis means" is a mechanism that determines the emotional state of the customer using data obtained from the facial expression recognition means and the voice analysis means.

[0286] The "display means" is a mechanism for displaying the determined emotional state in real time.

[0287] A "feedback presentation means" is a mechanism that provides feedback to staff based on their emotional state displayed in real time.

[0288] A "feedback generator" is a mechanism that automatically suggests service improvements based on the determined emotional state.

[0289] The "transmission means" is a mechanism for transmitting the collected face data and voice data to a server using a secure communication protocol.

[0290] The "data storage means" is a mechanism used to store the emotional states obtained from the facial expression recognition means and the voice analysis means.

[0291] The "analysis means" is a mechanism for comparing and analyzing stored data with past data.

[0292] System configuration

[0293] The present invention is a system that analyzes customer sentiment in real time and provides feedback to staff. This system is broadly composed of data collection means, data analysis means, feedback means, transmission means, data storage means, and analysis means.

[0294] Hardware and Software Use

[0295] The present invention is implemented using the following hardware and software:

[0296] Hardware: smart glasses, camera, microphone

[0297] Software: OpenCV (face recognition), TensorFlow (sentiment analysis), SciPy (voice analysis), HTTP request library (data transmission)

[0298] Data collection methods

[0299] The device collects customer facial and voice data in real time using the smart glasses' built-in camera and microphone. The camera captures high-resolution footage, and the microphone has the ability to filter out background noise.

[0300] Data Analysis Methods

[0301] To analyze the collected data, the server first extracts facial feature points using an image processing library (e.g., OpenCV) for face data, performs a Fourier transform on the voice data using a voice processing library (e.g., SciPy) to analyze the frequency spectrum and features, and then uses a deep learning model (e.g., TensorFlow) to determine the customer's emotional state.

[0302] Feedback Methods

[0303] The server then displays the determined emotional state on the staff member's smart glasses in real time, allowing the staff member to receive real-time feedback to provide optimal service based on the customer's emotions. Feedback is provided using audio guidance and visual alerts.

[0304] Means of transmission and data storage

[0305] The server sends the collected facial and voice data to a central server using a secure communication protocol (e.g., HTTPS), where it is stored in a database and used for analysis.

[0306] Data analysis and feedback generation

[0307] The emotion data stored in the data storage means is periodically analyzed by the analysis means. Here, customer responses to specific time periods and services are compared and analyzed, and trends and patterns are visualized. The feedback generation means then proposes optimal service improvement measures based on the analysis results.

[0308] Specific examples

[0309] For example, suppose a staff member wearing smart glasses captures a customer's smile while serving them. The system analyzes the video and recognizes the emotion of "happiness." It then displays feedback to the staff member's smart glasses, saying, "This customer is enjoying themselves. Please consider providing additional services." This feedback allows the staff member to immediately provide the optimal service to the customer.

[0310] Prompt Sentence Examples

[0311] "The following Python program implements a system that analyzes customer emotions in real time. The system uses OpenCV for facial recognition from camera footage and TensorFlow for emotion recognition. Audio data is analyzed using SciPy, and the data is sent to a server using HTTPS. Please complete the following code to enhance the emotion recognition functionality."

[0312] In this way, the present invention is a system that increases customer satisfaction by grasping customer emotions in real time and providing immediate feedback based on the emotions.

[0313] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0314] Step 1:

[0315] The device uses a camera and microphone to collect customer facial and audio data. The camera captures high-resolution video, and the microphone filters out background noise. The input is real-time facial video and audio, and the output is raw video and audio data.

[0316] Step 2:

[0317] The server receives the video data sent from the device and extracts facial feature points using an image processing library (e.g., OpenCV). The input is raw video data, and the output is processed image data containing facial feature points (e.g., the positions of the eyes, nose, and mouth).

[0318] Step 3:

[0319] The server simultaneously receives the audio data sent from the device and performs a Fourier transform using an audio processing library (e.g., SciPy). This analyzes the frequency spectrum and features (e.g., pitch, volume, and tone) of the audio data. The input is raw audio data, and the output is audio data with extracted features.

[0320] Step 4:

[0321] The server inputs the processed image data into a deep learning model (e.g., TensorFlow) and analyzes the customer's facial expression using facial expression recognition. The input is image data containing facial feature points, and the output is the analyzed facial expression (e.g., "happiness," "sadness," etc.).

[0322] Step 5:

[0323] The server inputs the feature-extracted voice data into a machine learning model (e.g., RNN) and analyzes the customer's voice tone using a voice analysis method. The input is the feature-extracted voice data, and the output is the analyzed emotional state (e.g., "joy," "anger," etc.).

[0324] Step 6:

[0325] The server integrates the data obtained from the facial expression recognition means and the voice analysis means and determines the overall emotional state of the customer using the emotion analysis means, where the input is the emotion result from the facial expression data and the voice data, and the output is the integrated emotional state.

[0326] Step 7:

[0327] The server displays the determined emotional state in real time on the staff member's smart glasses using a display means, where the input is the integrated emotional state and the output is feedback information that the staff member can see through the glasses.

[0328] Step 8:

[0329] The server provides feedback to staff using a feedback presentation tool based on the emotional state displayed in real time. The input is the displayed emotional state, and the output is a specific action or suggestion (e.g., "The customer is enjoying themselves. Please consider additional services.").

[0330] Step 9:

[0331] The server transmits the collected face and audio data to a central server using a secure communication protocol (e.g., HTTPS). The input is raw video and audio data, and the output is encrypted data communication.

[0332] Step 10:

[0333] The server stores the transmitted data in a database and compares it with past data. The input is the stored emotion data, and the output is the analysis results and visualization of trends and patterns.

[0334] Step 11:

[0335] The server uses a feedback generation means to propose optimal service improvement measures based on the analysis results. The input is the analysis results, and the output is automatically generated service improvement proposals.

[0336] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0337] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0338] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0339] [Second embodiment]

[0340] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0341] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0342] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0343] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0344] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0345] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0346] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0347] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0348] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0349] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0350] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0351] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0352] The present invention is a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly consists of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means.

[0353] Collection of customer data

[0354] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with noise-canceling functions to eliminate background noise and record customer voices clearly.

[0355] Sending data

[0356] Device: Collected face and voice data is sent to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0357] Data preprocessing and analysis

[0358] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0359] Emotion determination

[0360] Server: The preprocessed data is then passed to the emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0361] Data storage and analysis

[0362] Server: The determined emotion data is stored in a database and analyzed by an emotion analysis tool, comparing it with past data. This allows trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0363] Generate feedback

[0364] Server: Based on the stored emotion data, the system uses feedback generation methods to suggest optimal service improvement measures, such as "We should increase the number of staff during lunch hours" or "We should provide more detailed explanations about certain menu items."

[0365] Alerts and Reports

[0366] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator when certain conditions are met, enabling prompt action in real time. Aggregated reports of emotion data are also periodically created and distributed to administrators. These reports include weekly and monthly emotion trends, customer satisfaction trends, and suggestions for improvement.

[0367] Specific examples

[0368] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0369] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0370] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0371] In this way, the system can understand customer sentiment in real time and significantly improve the dining experience.

[0372] The processing flow will be explained below.

[0373] Step 1:

[0374] Terminal: In-store cameras capture customer facial data in real time. Specifically, the camera takes a certain number of frames per second and performs image processing to detect the customer's face. For example, the OpenCV library is used to detect faces and extract the face area from each frame.

[0375] Step 2:

[0376] Device: A microphone installed in the store collects customer voice data. Specifically, a noise-canceling microphone records customer conversations and saves the voice data in a format (e.g., WAV file) using an audio processing library (e.g., PyAudio).

[0377] Step 3:

[0378] Device: Collected face and voice data is sent to the server in batches (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it is encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server.

[0379] Step 4:

[0380] Server: Preprocesses the received face data using image processing libraries (e.g., OpenCV and Dlib) to identify facial regions and extract key feature points (e.g., the positions of the eyes, nose, and mouth). The feature points are normalized and converted into a format suitable for input to the deep learning model.

[0381] Step 5:

[0382] Server: Preprocesses the received audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and obtain the frequency spectrum. It then extracts features such as pitch, tone, and speed and converts them into input data for sentiment analysis.

[0383] Step 6:

[0384] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0385] Step 7:

[0386] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0387] Step 8:

[0388] Server: Integrates emotional data obtained from facial expression recognition and voice analysis to determine the overall emotional state of the customer, which is then stored in a database in a format that can be searched and compared later.

[0389] Step 9:

[0390] Server: Analyzes the integrated sentiment data to extract trends and patterns, for example, analyzing customer reactions to specific times of day or menu items, and generating statistics.

[0391] Step 10:

[0392] Server: Generates feedback based on emotional data. Specifically, it automatically creates improvement suggestions such as "increase staff during this time" or "add explanations to specific menu items."

[0393] Step 11:

[0394] Server: Sends real-time alerts to administrators when certain conditions are met (e.g., a large number of customers are unhappy). Alerts can be sent, for example, via email or SMS.

[0395] Step 12:

[0396] Server: Periodically creates aggregate reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement, providing reference for administrators to take appropriate measures.

[0397] Example 1

[0398] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0399] Conventionally, restaurants have lacked a means to quickly and accurately grasp customer sentiment in order to improve customer service. This can result in lower customer satisfaction and delays in service improvements, potentially affecting the restaurant's reputation and revenue. Furthermore, the collection and analysis of customer feedback is often done manually, limiting the accuracy and speed of the analysis results. This invention solves these problems and provides a system for grasping customer sentiment in real time and utilizing it to improve service.

[0400] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0401] In this invention, the server includes means for collecting customer facial data, means for collecting customer voice data, means for recognizing customer facial expressions from the collected facial data, means for analyzing customer tones from the collected voice data, means for determining the customer's emotional state based on data obtained by the facial expression recognition means and the voice analysis means, means for proposing service improvement measures based on the determined emotional state, means for sending an alert to a manager when the customer's emotional data meets certain conditions, and means for periodically creating and distributing a summary report of the emotional data to the manager. This makes it possible to accurately grasp customer emotions in real time and improve services quickly and effectively.

[0402] "Customer" refers to a user who visits a restaurant and uses the service.

[0403] "Facial data" refers to image information of a customer's face captured by an image capture device such as a camera.

[0404] "Voice data" refers to audio information of customer voices and conversations recorded using a voice capture device such as a microphone.

[0405] "Means" refers to any device or method, or combination thereof, used to accomplish a particular purpose.

[0406] "Facial expression" refers to facial expressions that change depending on the movement of a person's facial muscles.

[0407] "Tone" refers to vocal characteristics such as pitch, volume, and rhythm.

[0408] "Emotional state" refers to a specific emotion a customer feels, such as joy, anger, or sadness.

[0409] "Sentiment analysis" refers to the technology of determining customer emotions from collected data.

[0410] "Service Improvement Measures" refers to specific measures or action plans proposed to increase customer satisfaction.

[0411] An "alert" is a notification or warning to an administrator when a pre-defined condition is met.

[0412] An "aggregate report" refers to a report that analyzes and organizes collected data and summarizes the results.

[0413] "Means and" refers to a combination in which the aforementioned means and methods work together to perform a single function.

[0414] The present invention relates to a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly comprises a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means. Specific examples are shown below.

[0415] Collection of customer data

[0416] Device: High-resolution cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture video at 30 frames per second in 1080p resolution. The microphones are noise-canceling, eliminating background noise and allowing customers' voices to be recorded clearly.

[0417] Sending data

[0418] Device: The collected face and voice data is sent to the server using a secure communication protocol (e.g., HTTPS) with AES-256 encryption, which prevents unauthorized access and tampering of the data and protects privacy.

[0419] Data Preprocessing

[0420] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, the audio processing library SciPy is used to perform a Fourier transform and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0421] Emotion determination

[0422] Server: The preprocessed data is passed to the emotion analysis means. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0423] Data storage and analysis

[0424] Server: The determined emotion data is stored in a database and analyzed against past data, allowing trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0425] Generate feedback

[0426] Server: Based on the stored emotion data, the server uses a feedback generator to suggest optimal service improvement measures. For example, specific suggestions such as "We should increase the number of staff during lunchtime" or "We should provide more detailed explanations about specific menu items" are automatically generated.

[0427] Alerts and Reports

[0428] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator if certain conditions are met. For example, if a customer's "anger" emotion exceeds a certain threshold, an alert is sent immediately. In addition, aggregated reports of emotion data are periodically created and delivered to the administrator. The reports include weekly and monthly emotion trends, changes in customer satisfaction, and suggestions for improvement.

[0429] Specific examples

[0430] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0431] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0432] Server: Feedback will be generated suggesting "serve this menu item at dinner time as well." Based on this trend, the manager can receive a report saying "customer satisfaction is high at lunchtime."

[0433] Prompt Sentence Examples

[0434] "Cameras and microphones in the store captured customer smiles and positive conversations. Could you please outline a program that analyzes this data to determine customer sentiment and generate feedback?"

[0435] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0436] Step 1: Collect customer data

[0437] Terminal: A high-resolution camera installed in the store activates and collects facial data of customers in the store in real time. As input, it captures video at 30 frames per second at 1080p resolution. As output, it generates video frames containing facial features. At the same time, a noise-canceling microphone records the customer's conversation and collects its audio data. As input, the customer's speech is collected in real time, and as output, clear audio data with noise removed is provided.

[0438] Step 2: Sending data

[0439] Terminal: The collected face and voice data is sent to the server using the secure communication protocol HTTPS. As input, face and voice data are provided, and as output, encrypted data packets are sent to the server. During this process, the data is encrypted using the AES-256 encryption method to ensure security during transmission.

[0440] Step 3: Data Preprocessing

[0441] Server: Data arriving at the server is first preprocessed. The transmitted face data and audio data are provided as input. For the face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For example, the distance from one eye corner to the other and the shape of the mouth are output as numerical data. For the audio data, a Fourier transform is performed using the SciPy library to calculate the frequency spectrum and feature quantities (e.g., pitch, volume, and tone). As the output of this processing, a dataset for each feature point is generated.

[0442] Step 4: Determine the emotion

[0443] Server: The preprocessed data is fed into the emotion analysis model. Facial feature points and audio features are provided as input. For facial data, a convolutional neural network (CNN) model is used to identify emotions such as "happiness," "anger," and "sadness" from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) model is used to accurately determine emotions from vocal tones. The output is the detected emotion label and its confidence level.

[0444] Step 5: Save your data

[0445] Server: The determined emotion data is stored in a database and used for analysis. As input, emotion labels and their confidence levels are provided. This data is recorded in the database as time-series data. As output, emotion data including time information is stored in the database.

[0446] Step 6: Generate feedback

[0447] Server: Based on the stored emotion data, the feedback generation means creates service improvement proposals. The most recent emotion data and past emotion data are provided as input. Based on comparative analysis with past data, specific trends and patterns, such as "certain menu items are highly rated at lunchtime," are derived. Specific service improvement measures (e.g., improving the menu, increasing the number of staff, etc.) are generated as output.

[0448] Step 7: Alerts and Reports

[0449] Server: Based on the results of sentiment analysis, an alert is sent to the administrator when certain conditions are met. Real-time sentiment data is provided as input. For example, if a customer's "anger" sentiment exceeds a certain threshold, an alert is sent immediately. As output, a notification is generated for the administrator. In addition, aggregated reports of sentiment data are periodically created and delivered to the administrator. As output, reports are created that include weekly and monthly sentiment trends, customer satisfaction levels, and improvement suggestions.

[0450] The above are the specific processing steps of this system.

[0451] (Application example 1)

[0452] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0453] In traditional restaurants, there were limited ways to understand customer satisfaction and dissatisfaction in real time, which often delayed the timing of service improvements. It was also difficult to quantitatively analyze customer sentiment and derive specific improvement measures. Furthermore, there was a lack of alert functions to respond quickly when specific problems occurred.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0455] In this invention, the server includes a feedback generation means for proposing service improvement measures based on the determined emotional state, an alert means for notifying staff or managers of an alert when a specific emotion is recognized frequently, and a data analysis means for analyzing past customer data and generating periodic reports. This makes it possible to grasp customer emotions in real time and quickly propose appropriate service improvement measures. Furthermore, when a specific problem occurs, an alert is sent immediately, encouraging a prompt response. Furthermore, the periodic reports can be compared with past data to derive more accurate service improvement measures.

[0456] "Data collection means" refers to a device or group of devices for collecting face data and voice data of customers.

[0457] "Facial expression recognition means" refers to an algorithm or device for recognizing customer facial expressions from collected facial data.

[0458] The "voice analysis means" refers to an algorithm or device for analyzing the customer's tone of voice from collected voice data.

[0459] The "emotion analysis means" refers to an algorithm or device for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means.

[0460] A "feedback generator" is an algorithm or device that suggests service improvements based on the determined emotional state.

[0461] An "alert means" is a device or group of devices that alerts staff or managers when a specific emotion is recognized frequently.

[0462] "Data analysis tools" are algorithms or devices used to analyze historical customer data and generate periodic reports.

[0463] The "transmission means" refers to a device or group of devices for transmitting the collected face data and voice data using a secure communication protocol.

[0464] The "data storage means" refers to a device or group of devices for storing the emotional states obtained from the facial expression recognition means and the voice analysis means in a database.

[0465] The "analysis means" refers to an algorithm or device for comparing and analyzing new data with past data stored in the data storage means.

[0466] To implement this invention, a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, an alert means, and a data analysis means are mainly required, as well as a transmission means for securely transmitting the collected data, and a data storage means and analysis means for storing and analyzing the data.

[0467] Data collection

[0468] The device uses high-resolution cameras and noise-canceling microphones installed in the store to collect customer facial and voice data in real time, allowing for accurate capture of customer facial expressions and tone of voice.

[0469] Data transmission

[0470] Facial and audio data collected on the device is transmitted to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0471] Data Preprocessing and Analysis

[0472] The data that arrives at the server is first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points. For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features.

[0473] Emotion analysis

[0474] The preprocessed data is then passed to an emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This analysis allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0475] Feedback and Alerts

[0476] The server then uses a feedback generator to generate optimal service improvement measures based on the determined emotional state. For example, specific suggestions such as "We need to hire more staff" or "We should provide more detailed explanations for certain menu items" are generated. Additionally, an alerting mechanism immediately notifies staff and managers when a specific emotion is detected frequently.

[0477] Data Storage and Analysis

[0478] The emotional data is stored in a database and compared with past data using data analysis tools. This allows trends and patterns, such as customer reactions to specific time periods or menu items, to be visualized. Furthermore, aggregated reports of the emotional data are periodically created and distributed to managers.

[0479] Specific examples

[0480] For example, suppose a customer enters a restaurant during lunchtime, and the camera captures their smile and the microphone records their positive conversation. This data is sent to a server, where a deep learning model determines their happiness. Based on this information, feedback is generated that says "This customer is satisfied," and the information is shared with staff. This data is also stored in a database and used to improve service in the future.

[0481] Here are some example prompts to input to a generative AI model:

[0482] "The camera captures a smiling customer entering your restaurant during lunchtime, and the microphone records his positive conversation. Based on this situation, generate feedback and improvement suggestions if the analysis determines that he is happy."

[0483] This invention makes it possible to grasp customer sentiment in real time and quickly propose appropriate service improvement measures. It also sends an immediate alert when a specific problem occurs, encouraging a prompt response. Furthermore, periodic reports can be compared with past data to derive more accurate service improvement measures.

[0484] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0485] Step 1:

[0486] The terminal uses a high-resolution camera and a noise-canceling microphone installed in the store to collect customer facial and voice data in real time. The inputs for collection are camera images and microphone audio. To obtain this data clearly, the camera recognizes the customer's face and the microphone removes background noise. The output is the customer's facial image data and voice data.

[0487] Step 2:

[0488] The device sends the collected face data and voice data to the server using a secure communication protocol (e.g., HTTPS). The input is the face image data and voice data collected in step 1. The data is encrypted and sent to the server over the network. The output is the face image data and voice data that arrives at the server.

[0489] Step 3:

[0490] The server preprocesses the received facial image data using an image processing library (e.g., OpenCV). The input is the facial image data received by the server. Facial feature points (e.g., the positions of the eyes, nose, and mouth) are extracted through image processing. The output is the extracted facial feature points.

[0491] Step 4:

[0492] The server preprocesses the incoming audio data using an audio processing library (e.g., SciPy). The input is the audio data arriving at the server. The audio data is Fourier transformed and the frequency spectrum and features (e.g., pitch, volume, and tone) are analyzed. The output is the analyzed audio features.

[0493] Step 5:

[0494] The server analyzes emotions from the preprocessed facial data using a convolutional neural network (CNN) model. The input is the facial feature points extracted in step 3. The CNN model determines emotions such as "happiness," "anger," and "sadness" with high accuracy from these feature points. The output is the analyzed emotion data.

[0495] Step 6:

[0496] The server analyzes the preprocessed speech data for emotion using a recurrent neural network (RNN) model. The input is the speech features analyzed in step 4. The RNN model determines the emotion from the speech data. The output is the analyzed emotion data.

[0497] Step 7:

[0498] The server uses the feedback generation means to propose optimal service improvement measures based on the emotional state of the face data and voice data obtained from the emotion analysis means. The input is the emotional data obtained in steps 5 and 6. The algorithm generates specific service improvement proposals. The output is the proposed service improvement measures.

[0499] Step 8:

[0500] The server uses an alerting means to notify staff or administrators when a specific emotion is recognized frequently. The input is emotion data from the emotion analysis means. An alert is generated based on specific conditions and a notification is sent. The output is an alert notification to staff or administrators.

[0501] Step 9:

[0502] The server stores the emotion data in a database and performs comparative analysis with past data using data analysis means. The input is the emotion data obtained in steps 7 and 8. This allows analysis of customer reactions to specific time periods and menu items, and visualizes trends. The output is the analysis results and data stored in the database.

[0503] Step 10:

[0504] The server periodically generates an aggregate report of the emotion data and delivers it to the administrator. The input is the emotion data saved in step 9. The aggregate report includes weekly and monthly emotion trends, customer satisfaction trends, improvement suggestions, etc. The output is the emotion data report delivered to the administrator.

[0505] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0506] The present invention is a system for understanding customer emotions in real time at restaurants, helping to improve service and feedback accuracy. This system is composed of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, a transmission means, a data storage means, and an emotion engine.

[0507] Collection of customer data

[0508] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with background noise reduction.

[0509] Sending data

[0510] On the device: The collected face and voice data is sent to the server in batches (e.g., every 10 seconds). The data is encrypted using a secure communication protocol (e.g., HTTPS).

[0511] Data preprocessing and analysis

[0512] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, a Fourier transform is performed using an audio processing library (e.g., SciPy) to analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0513] Emotion assessment and integration

[0514] Server: The pre-processed data is first processed using facial expression recognition means to determine customer emotions using a deep learning model (e.g., CNN). Similarly, audio analysis means is used to determine emotions from audio data using a machine learning model (e.g., RNN). These data are then integrated by the emotion engine to determine the overall customer emotional state.

[0515] Data storage and analysis

[0516] Server: The determined emotion data is stored in a database and compared with past data for analysis. Here, customer reactions to specific times of day or menu items are analyzed, and trends and patterns are visualized.

[0517] Generate feedback

[0518] Server: Based on the sentiment data, the feedback generator automatically suggests optimal service improvement measures, such as "increase staffing during certain times" or "add explanations for certain menu items."

[0519] Alerts and Reports

[0520] Server: Based on the results of the emotion data analysis, real-time alerts are sent to administrators when certain conditions are met. Aggregated reports of emotion data are also periodically created and distributed to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement.

[0521] Specific examples

[0522] Device: The in-store camera captures the customer's smile, and the microphone simultaneously records the customer's positive conversation. These data are sent to the server, and feature points of the video and audio are extracted.

[0523] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0524] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0525] In this way, the system can grasp customer sentiment in real time and provide accurate feedback and solutions to improve the dining experience.

[0526] The processing flow will be explained below.

[0527] Step 1:

[0528] Terminal: A camera installed in the store captures customer facial data in real time. Specifically, the camera takes a certain number of frames per second (e.g., 30 frames per second) and detects facial regions using a face detection algorithm (e.g., Haar cascade). The detected facial regions are segmented and passed to subsequent processing steps.

[0529] Step 2:

[0530] Device: A microphone installed in the store collects customer voice data. Specifically, the microphone records the voice at a certain sample rate (e.g., 44.1 kHz), and the voice data is processed using noise cancellation and then saved in an audio file format (e.g., WAV file).

[0531] Step 3:

[0532] Device: Sends face and voice data to the server in batches of a certain size (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it encrypts the data using a secure communication protocol (e.g., HTTPS) and sends it to the server.

[0533] Step 4:

[0534] Server: Preprocesses the face data. Specifically, it uses an image processing library (e.g., OpenCV) to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). This process converts the data into a format that can be used by the deep learning model. The extracted feature points are also normalized and passed on to the next step.

[0535] Step 5:

[0536] Server: Preprocesses the audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and analyze the frequency spectrum. It also extracts features such as pitch, tone, and speed, and converts them into input data for emotion analysis.

[0537] Step 6:

[0538] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0539] Step 7:

[0540] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0541] Step 8:

[0542] Server: The emotion data obtained from facial expression recognition and voice analysis is integrated into the emotion engine to determine the overall emotional state of the customer. Specifically, the emotion engine integrates the facial expression data and voice data to calculate a single integrated emotion score. This information is stored in a database.

[0543] Step 9:

[0544] Server: Analyzes the integrated sentiment data and extracts trends and patterns. For example, analyzes customer reactions to specific times or menu items and generates statistical information. Data analysis tools (e.g., Pandas, NumPy) are used for the analysis.

[0545] Step 10:

[0546] Server: Generates feedback based on emotion data. Specifically, the feedback generator automatically creates specific improvement suggestions, such as "increase staff during this time period" or "add explanations for specific menu items."

[0547] Step 11:

[0548] Server: Sends real-time alerts to administrators when certain conditions are met. For example, if customer satisfaction is low or if there is a high level of dissatisfaction with a particular menu item, an alert will be sent to the administrator via email or SMS.

[0549] Step 12:

[0550] Server: Periodically creates aggregated reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement. The reports are generated in PDF format and sent via email.

[0551] Example 2

[0552] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Restaurants are required to understand the emotional state of customers in real time and use that information to improve and optimize the quality of their services. However, conventional systems collect, analyze, store, and generate feedback from facial and voice data separately, making efficient and integrated operation difficult.

[0554] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image capture means for collecting customer facial data, an audio collection means for collecting customer voice data, a transmission means for transmitting the collected facial data, a transmission means for transmitting the collected voice data, an image processing means for preprocessing the collected facial data, an audio processing means for preprocessing the collected voice data, an expression analysis means for recognizing the customer's facial expression from the preprocessed facial data, an audio analysis means for analyzing the customer's tone of voice from the preprocessed audio data, an emotion analysis means for integrating data obtained from the expression analysis means and the audio analysis means to determine the customer's emotional state, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to efficiently integrate the customer's emotional state in real time and quickly suggest optimal service improvement measures.

[0555] "Image Capture Device" means a device used to collect customer facial data in real time.

[0556] An "audio collection means" is a device used to collect customer audio data.

[0557] "Transmission means" refers to a communication device or protocol for transmitting collected data to a server.

[0558] "Image processing means" refers to software or hardware for preprocessing collected face data and extracting feature points.

[0559] The "acoustic processing means" refers to software or hardware for preprocessing collected audio data and analyzing its features.

[0560] "Facial expression analysis means" refers to algorithms or models used to recognize customer emotions from pre-processed facial data.

[0561] "Voice analysis means" refers to algorithms or models for analyzing the customer's tone of voice from pre-processed voice data.

[0562] The "emotion analysis means" is a system for integrating data obtained from the facial expression analysis means and the voice analysis means to determine the emotional state of the customer.

[0563] The "feedback generation means" is a mechanism for suggesting service improvements based on the determined emotional state.

[0564] The present invention is a system for understanding customer emotions in real time at restaurants and optimizing and improving service quality based on that information. This system is composed of an image capture means, an audio collection means, a transmission means, an image processing means, an audio processing means, a facial expression analysis means, a voice analysis means, an emotion analysis means, and a feedback generation means.

[0565] First, cameras and microphones installed in the store function as image capture and audio collection devices. The high-resolution cameras collect customer facial data in real time, while the microphones record customer conversations. The cameras capture details from the moment customers enter the store to the moment they smile, while the microphones record the conversations while filtering out surrounding noise.

[0566] This data is then sent from the device to the server in batches (e.g., every 10 seconds) using a secure communication protocol (e.g., HTTPS) to encrypt and transmit the data safely. This process reduces the risk of data eavesdropping or tampering.

[0567] The data that arrives at the server is first preprocessed using image processing and audio processing means. Specifically, an image processing library (e.g., OpenCV) is used to extract facial feature points (the positions of the eyes, nose, and mouth, etc.), and an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data, analyzing the frequency spectrum and features (pitch, volume, tone, etc.).

[0568] The preprocessed data is then subjected to emotion recognition using facial expression analysis means within the server. This analysis uses a deep learning model (e.g., CNN) to determine the customer's emotion from the image data. Similarly, emotion is determined from the voice data using a machine learning model (e.g., RNN) through the voice analysis means. The results of both analyses are integrated by the emotion analysis means to comprehensively determine the customer's final emotional state.

[0569] The emotional state determined is stored in a database and compared with past data using analytical tools. This analysis visualizes customer reactions to specific time periods and menu items, highlighting trends and patterns. For example, a trend such as "a particular pasta menu item is popular during lunchtime" can be analyzed.

[0570] Finally, the feedback generation means automatically proposes service improvement measures based on the emotional data. For example, a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well" may be automatically generated. Furthermore, when certain conditions are met, an alert is sent to the manager in real time, and a summary report of the emotional data is periodically created and distributed to the manager.

[0571] As a concrete example, suppose an in-store camera captures a customer's smile, while a microphone simultaneously records a customer's positive conversation. This data is sent to a server, where feature points from the video and audio are extracted. Using CNN and RNN models, the smile and positive tone of voice are used to determine the emotion of "happiness." The recognized emotion data is then stored in a database, and analysis derives a trend, such as "a particular lunchtime menu item is popular." As feedback, a suggestion is generated to "serve this menu item at dinnertime as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0572] The system allows for real-time understanding of customer sentiment and provides targeted feedback and improvements to improve the dining experience.

[0573] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] Terminal: A camera installed in the store captures customers' faces in real time. Specifically, the camera captures high-resolution video and outputs each frame as image data. The input is a video of customers in the store, and the output is image data containing the customer's face.

[0576] Step 2:

[0577] Terminal: A microphone records customer conversations in real time. Noise-canceling technology is used to remove background noise and obtain clean audio data. The input is the in-store audio environment, and the output is audio data with the customer's conversations emphasized.

[0578] Step 3:

[0579] Terminal: The collected face data and audio data are batch processed (e.g., every 10 seconds) and sent to the server. When sent, the data is encrypted using a secure communication protocol (e.g., HTTPS). The input is image data and audio data, and the output is the encrypted transmission data.

[0580] Step 4:

[0581] Server: Receives the transmitted face data and performs preprocessing using an image processing library (e.g., OpenCV). Specifically, the processing involves extracting facial feature points (e.g., the positions of the eyes, nose, and mouth). The input is encrypted image data, and the output is preprocessed image data with feature points extracted.

[0582] Step 5:

[0583] Server: Receives the transmitted audio data and performs preprocessing using an audio processing library (e.g., SciPy). Performs a Fourier transform on the audio data and analyzes the frequency spectrum and features (e.g., pitch, volume, tone). The input is encrypted audio data, and the output is preprocessed audio data with the frequency spectrum and features analyzed.

[0584] Step 6:

[0585] Server: Using the preprocessed image data, a deep learning model (e.g., CNN) is run to analyze facial expressions and determine the customer's emotions. Specifically, the model outputs emotion labels such as "joy," "anger," and "sadness" from the input image data. The input is the preprocessed image data, and the output is the emotion label.

[0586] Step 7:

[0587] Server: Using the preprocessed voice data, a machine learning model (e.g., RNN) is run to analyze the voice and determine the emotion from the customer's tone of voice. Specifically, the model outputs an emotion label such as "satisfied" or "dissatisfied" from the input voice data. The input is the preprocessed voice data, and the output is the emotion label.

[0588] Step 8:

[0589] Server: The emotion analysis means integrates the emotion labels obtained from the facial expression analysis means and the voice analysis means to determine the overall emotional state of the customer. For example, if the facial expression of "happiness" matches the tone of voice of "satisfied," the overall judgment is "very satisfied." The input is the emotion labels of facial expressions and voice, and the output is the integrated emotional state.

[0590] Step 9:

[0591] Server: Stores the determined emotional state in a database. It then performs comparative analysis with past data, analyzes customer reactions to specific time periods and menu items, and visualizes trends and patterns. For example, it analyzes the trend that "a specific lunchtime menu item is popular." The input is the integrated emotional state data, and the output is the visualized results of trends and patterns.

[0592] Step 10:

[0593] Server: The feedback generation means automatically proposes optimal service improvement measures based on the emotion data. For example, it generates a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well." The input is the analysis results and emotion data, and the output is a service improvement proposal.

[0594] Step 11:

[0595] Server: When a specific condition is met (for example, when customer satisfaction declines), it sends an alert to the administrator in real time. It also periodically creates aggregate reports of emotion data and delivers them to the administrator. The input is emotion data that meets the condition, and the output is the alert notification and aggregate reports.

[0596] (Application example 2)

[0597] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0598] Traditional customer service in restaurants relies heavily on staff experience and intuition, making it difficult to accurately grasp customer emotions and satisfaction. Furthermore, there are limited means to grasp customers' real-time emotional states and provide appropriate feedback based on them, resulting in a lack of speed in service improvement. Therefore, in order to improve customer satisfaction and service quality, real-time emotion analysis and immediate feedback based on the results are required.

[0599] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting customer facial data, a data collection means for collecting customer voice data, a facial expression recognition means for recognizing the customer's facial expression from the collected facial data, a voice analysis means for analyzing the customer's tone of voice from the collected voice data, an emotion analysis means for determining the customer's emotional state based on the data obtained by the facial expression recognition means and the voice analysis means, a display means for displaying the determined emotional state in real time, a feedback presentation means for providing feedback to the staff based on the emotional state displayed in real time, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to accurately grasp the customer's emotional state in real time and provide immediate feedback based on that.

[0600] The "data collection means" is a mechanism for collecting face data and voice data of customers.

[0601] The "facial expression recognition means" is a mechanism that analyzes the facial expression of the customer from the collected facial data and recognizes that expression.

[0602] The "voice analysis means" is a mechanism that analyzes collected voice data to analyze the customer's voice tone and other characteristics.

[0603] The "emotion analysis means" is a mechanism that determines the emotional state of the customer using data obtained from the facial expression recognition means and the voice analysis means.

[0604] The "display means" is a mechanism for displaying the determined emotional state in real time.

[0605] A "feedback presentation means" is a mechanism that provides feedback to staff based on their emotional state displayed in real time.

[0606] A "feedback generator" is a mechanism that automatically suggests service improvements based on the determined emotional state.

[0607] The "transmission means" is a mechanism for transmitting the collected face data and voice data to a server using a secure communication protocol.

[0608] The "data storage means" is a mechanism used to store the emotional states obtained from the facial expression recognition means and the voice analysis means.

[0609] The "analysis means" is a mechanism for comparing and analyzing stored data with past data.

[0610] System configuration

[0611] The present invention is a system that analyzes customer sentiment in real time and provides feedback to staff. This system is broadly composed of data collection means, data analysis means, feedback means, transmission means, data storage means, and analysis means.

[0612] Hardware and Software Use

[0613] The present invention is implemented using the following hardware and software:

[0614] Hardware: smart glasses, camera, microphone

[0615] Software: OpenCV (face recognition), TensorFlow (sentiment analysis), SciPy (voice analysis), HTTP request library (data transmission)

[0616] Data collection methods

[0617] The device collects customer facial and voice data in real time using the smart glasses' built-in camera and microphone. The camera captures high-resolution footage, and the microphone has the ability to filter out background noise.

[0618] Data Analysis Methods

[0619] To analyze the collected data, the server first extracts facial feature points using an image processing library (e.g., OpenCV) for face data, performs a Fourier transform on the voice data using a voice processing library (e.g., SciPy) to analyze the frequency spectrum and features, and then uses a deep learning model (e.g., TensorFlow) to determine the customer's emotional state.

[0620] Feedback Methods

[0621] The server then displays the determined emotional state on the staff member's smart glasses in real time, allowing the staff member to receive real-time feedback to provide optimal service based on the customer's emotions. Feedback is provided using audio guidance and visual alerts.

[0622] Means of transmission and data storage

[0623] The server sends the collected facial and voice data to a central server using a secure communication protocol (e.g., HTTPS), where it is stored in a database and used for analysis.

[0624] Data analysis and feedback generation

[0625] The emotion data stored in the data storage means is periodically analyzed by the analysis means. Here, customer responses to specific time periods and services are compared and analyzed, and trends and patterns are visualized. The feedback generation means then proposes optimal service improvement measures based on the analysis results.

[0626] Specific examples

[0627] For example, suppose a staff member wearing smart glasses captures a customer's smile while serving them. The system analyzes the video and recognizes the emotion of "happiness." It then displays feedback to the staff member's smart glasses, saying, "This customer is enjoying themselves. Please consider providing additional services." This feedback allows the staff member to immediately provide the optimal service to the customer.

[0628] Prompt Sentence Examples

[0629] "The following Python program implements a system that analyzes customer emotions in real time. The system uses OpenCV for facial recognition from camera footage and TensorFlow for emotion recognition. Audio data is analyzed using SciPy, and the data is sent to a server using HTTPS. Please complete the following code to enhance the emotion recognition functionality."

[0630] In this way, the present invention is a system that increases customer satisfaction by grasping customer emotions in real time and providing immediate feedback based on the emotions.

[0631] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0632] Step 1:

[0633] The device uses a camera and microphone to collect customer facial and audio data. The camera captures high-resolution video, and the microphone filters out background noise. The input is real-time facial video and audio, and the output is raw video and audio data.

[0634] Step 2:

[0635] The server receives the video data sent from the device and extracts facial feature points using an image processing library (e.g., OpenCV). The input is raw video data, and the output is processed image data containing facial feature points (e.g., the positions of the eyes, nose, and mouth).

[0636] Step 3:

[0637] The server simultaneously receives the audio data sent from the device and performs a Fourier transform using an audio processing library (e.g., SciPy). This analyzes the frequency spectrum and features (e.g., pitch, volume, and tone) of the audio data. The input is raw audio data, and the output is audio data with extracted features.

[0638] Step 4:

[0639] The server inputs the processed image data into a deep learning model (e.g., TensorFlow) and analyzes the customer's facial expression using facial expression recognition. The input is image data containing facial feature points, and the output is the analyzed facial expression (e.g., "happiness," "sadness," etc.).

[0640] Step 5:

[0641] The server inputs the feature-extracted voice data into a machine learning model (e.g., RNN) and analyzes the customer's voice tone using a voice analysis method. The input is the feature-extracted voice data, and the output is the analyzed emotional state (e.g., "joy," "anger," etc.).

[0642] Step 6:

[0643] The server integrates the data obtained from the facial expression recognition means and the voice analysis means and determines the overall emotional state of the customer using the emotion analysis means, where the input is the emotion result from the facial expression data and the voice data, and the output is the integrated emotional state.

[0644] Step 7:

[0645] The server displays the determined emotional state in real time on the staff member's smart glasses using a display means, where the input is the integrated emotional state and the output is feedback information that the staff member can see through the glasses.

[0646] Step 8:

[0647] The server provides feedback to staff using a feedback presentation tool based on the emotional state displayed in real time. The input is the displayed emotional state, and the output is a specific action or suggestion (e.g., "The customer is enjoying themselves. Please consider additional services.").

[0648] Step 9:

[0649] The server transmits the collected face and audio data to a central server using a secure communication protocol (e.g., HTTPS). The input is raw video and audio data, and the output is encrypted data communication.

[0650] Step 10:

[0651] The server stores the transmitted data in a database and compares it with past data. The input is the stored emotion data, and the output is the analysis results and visualization of trends and patterns.

[0652] Step 11:

[0653] The server uses a feedback generation means to propose optimal service improvement measures based on the analysis results. The input is the analysis results, and the output is automatically generated service improvement proposals.

[0654] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0655] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0656] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0657] [Third embodiment]

[0658] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0659] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0660] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0661] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0662] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0663] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0664] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0665] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0666] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0669] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0670] The present invention is a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly consists of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means.

[0671] Collection of customer data

[0672] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with noise-canceling functions to eliminate background noise and record customer voices clearly.

[0673] Sending data

[0674] Device: Collected face and voice data is sent to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0675] Data preprocessing and analysis

[0676] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0677] Emotion determination

[0678] Server: The preprocessed data is then passed to the emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0679] Data storage and analysis

[0680] Server: The determined emotion data is stored in a database and analyzed by an emotion analysis tool, comparing it with past data. This allows trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0681] Generate feedback

[0682] Server: Based on the stored emotion data, the system uses feedback generation methods to suggest optimal service improvement measures, such as "We should increase the number of staff during lunch hours" or "We should provide more detailed explanations about certain menu items."

[0683] Alerts and Reports

[0684] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator when certain conditions are met, enabling prompt action in real time. Aggregated reports of emotion data are also periodically created and distributed to administrators. These reports include weekly and monthly emotion trends, customer satisfaction trends, and suggestions for improvement.

[0685] Specific examples

[0686] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0687] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0688] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0689] In this way, the system can understand customer sentiment in real time and significantly improve the dining experience.

[0690] The processing flow will be explained below.

[0691] Step 1:

[0692] Terminal: In-store cameras capture customer facial data in real time. Specifically, the camera takes a certain number of frames per second and performs image processing to detect the customer's face. For example, the OpenCV library is used to detect faces and extract the face area from each frame.

[0693] Step 2:

[0694] Device: A microphone installed in the store collects customer voice data. Specifically, a noise-canceling microphone records customer conversations and saves the voice data in a format (e.g., WAV file) using an audio processing library (e.g., PyAudio).

[0695] Step 3:

[0696] Device: Collected face and voice data is sent to the server in batches (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it is encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server.

[0697] Step 4:

[0698] Server: Preprocesses the received face data using image processing libraries (e.g., OpenCV and Dlib) to identify facial regions and extract key feature points (e.g., the positions of the eyes, nose, and mouth). The feature points are normalized and converted into a format suitable for input to the deep learning model.

[0699] Step 5:

[0700] Server: Preprocesses the received audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and obtain the frequency spectrum. It then extracts features such as pitch, tone, and speed and converts them into input data for sentiment analysis.

[0701] Step 6:

[0702] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0703] Step 7:

[0704] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0705] Step 8:

[0706] Server: Integrates emotional data obtained from facial expression recognition and voice analysis to determine the overall emotional state of the customer, which is then stored in a database in a format that can be searched and compared later.

[0707] Step 9:

[0708] Server: Analyzes the integrated sentiment data to extract trends and patterns, for example, analyzing customer reactions to specific times of day or menu items, and generating statistics.

[0709] Step 10:

[0710] Server: Generates feedback based on emotional data. Specifically, it automatically creates improvement suggestions such as "increase staff during this time" or "add explanations to specific menu items."

[0711] Step 11:

[0712] Server: Sends real-time alerts to administrators when certain conditions are met (e.g., a large number of customers are unhappy). Alerts can be sent, for example, via email or SMS.

[0713] Step 12:

[0714] Server: Periodically creates aggregate reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement, providing reference for administrators to take appropriate measures.

[0715] Example 1

[0716] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0717] Conventionally, restaurants have lacked a means to quickly and accurately grasp customer sentiment in order to improve customer service. This can result in lower customer satisfaction and delays in service improvements, potentially affecting the restaurant's reputation and revenue. Furthermore, the collection and analysis of customer feedback is often done manually, limiting the accuracy and speed of the analysis results. This invention solves these problems and provides a system for grasping customer sentiment in real time and utilizing it to improve service.

[0718] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0719] In this invention, the server includes means for collecting customer facial data, means for collecting customer voice data, means for recognizing customer facial expressions from the collected facial data, means for analyzing customer tones from the collected voice data, means for determining the customer's emotional state based on data obtained by the facial expression recognition means and the voice analysis means, means for proposing service improvement measures based on the determined emotional state, means for sending an alert to a manager when the customer's emotional data meets certain conditions, and means for periodically creating and distributing a summary report of the emotional data to the manager. This makes it possible to accurately grasp customer emotions in real time and improve services quickly and effectively.

[0720] "Customer" refers to a user who visits a restaurant and uses the service.

[0721] "Facial data" refers to image information of a customer's face captured by an image capture device such as a camera.

[0722] "Voice data" refers to audio information of customer voices and conversations recorded using a voice capture device such as a microphone.

[0723] "Means" refers to any device or method, or combination thereof, used to accomplish a particular purpose.

[0724] "Facial expression" refers to facial expressions that change depending on the movement of a person's facial muscles.

[0725] "Tone" refers to vocal characteristics such as pitch, volume, and rhythm.

[0726] "Emotional state" refers to a specific emotion a customer feels, such as joy, anger, or sadness.

[0727] "Sentiment analysis" refers to the technology of determining customer emotions from collected data.

[0728] "Service Improvement Measures" refers to specific measures or action plans proposed to increase customer satisfaction.

[0729] An "alert" is a notification or warning to an administrator when a pre-defined condition is met.

[0730] An "aggregate report" refers to a report that analyzes and organizes collected data and summarizes the results.

[0731] "Means and" refers to a combination in which the aforementioned means and methods work together to perform a single function.

[0732] The present invention relates to a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly comprises a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means. Specific examples are shown below.

[0733] Collection of customer data

[0734] Device: High-resolution cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture video at 30 frames per second in 1080p resolution. The microphones are noise-canceling, eliminating background noise and allowing customers' voices to be recorded clearly.

[0735] Sending data

[0736] Device: The collected face and voice data is sent to the server using a secure communication protocol (e.g., HTTPS) with AES-256 encryption, which prevents unauthorized access and tampering of the data and protects privacy.

[0737] Data Preprocessing

[0738] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, the audio processing library SciPy is used to perform a Fourier transform and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0739] Emotion determination

[0740] Server: The preprocessed data is passed to the emotion analysis means. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0741] Data storage and analysis

[0742] Server: The determined emotion data is stored in a database and analyzed against past data, allowing trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[0743] Generate feedback

[0744] Server: Based on the stored emotion data, the server uses a feedback generator to suggest optimal service improvement measures. For example, specific suggestions such as "We should increase the number of staff during lunchtime" or "We should provide more detailed explanations about specific menu items" are automatically generated.

[0745] Alerts and Reports

[0746] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator if certain conditions are met. For example, if a customer's "anger" emotion exceeds a certain threshold, an alert is sent immediately. In addition, aggregated reports of emotion data are periodically created and delivered to the administrator. The reports include weekly and monthly emotion trends, changes in customer satisfaction, and suggestions for improvement.

[0747] Specific examples

[0748] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[0749] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0750] Server: Feedback will be generated suggesting "serve this menu item at dinner time as well." Based on this trend, the manager can receive a report saying "customer satisfaction is high at lunchtime."

[0751] Prompt Sentence Examples

[0752] "Cameras and microphones in the store captured customer smiles and positive conversations. Could you please outline a program that analyzes this data to determine customer sentiment and generate feedback?"

[0753] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0754] Step 1: Collect customer data

[0755] Terminal: A high-resolution camera installed in the store activates and collects facial data of customers in the store in real time. As input, it captures video at 30 frames per second at 1080p resolution. As output, it generates video frames containing facial features. At the same time, a noise-canceling microphone records the customer's conversation and collects its audio data. As input, the customer's speech is collected in real time, and as output, clear audio data with noise removed is provided.

[0756] Step 2: Sending data

[0757] Terminal: The collected face and voice data is sent to the server using the secure communication protocol HTTPS. As input, face and voice data are provided, and as output, encrypted data packets are sent to the server. During this process, the data is encrypted using the AES-256 encryption method to ensure security during transmission.

[0758] Step 3: Data Preprocessing

[0759] Server: Data arriving at the server is first preprocessed. The transmitted face data and audio data are provided as input. For the face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For example, the distance from one eye corner to the other and the shape of the mouth are output as numerical data. For the audio data, a Fourier transform is performed using the SciPy library to calculate the frequency spectrum and feature quantities (e.g., pitch, volume, and tone). As the output of this processing, a dataset for each feature point is generated.

[0760] Step 4: Determine the emotion

[0761] Server: The preprocessed data is fed into the emotion analysis model. Facial feature points and audio features are provided as input. For facial data, a convolutional neural network (CNN) model is used to identify emotions such as "happiness," "anger," and "sadness" from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) model is used to accurately determine emotions from vocal tones. The output is the detected emotion label and its confidence level.

[0762] Step 5: Save your data

[0763] Server: The determined emotion data is stored in a database and used for analysis. As input, emotion labels and their confidence levels are provided. This data is recorded in the database as time-series data. As output, emotion data including time information is stored in the database.

[0764] Step 6: Generate feedback

[0765] Server: Based on the stored emotion data, the feedback generation means creates service improvement proposals. The most recent emotion data and past emotion data are provided as input. Based on comparative analysis with past data, specific trends and patterns, such as "certain menu items are highly rated at lunchtime," are derived. Specific service improvement measures (e.g., improving the menu, increasing the number of staff, etc.) are generated as output.

[0766] Step 7: Alerts and Reports

[0767] Server: Based on the results of sentiment analysis, an alert is sent to the administrator when certain conditions are met. Real-time sentiment data is provided as input. For example, if a customer's "anger" sentiment exceeds a certain threshold, an alert is sent immediately. As output, a notification is generated for the administrator. In addition, aggregated reports of sentiment data are periodically created and delivered to the administrator. As output, reports are created that include weekly and monthly sentiment trends, customer satisfaction levels, and improvement suggestions.

[0768] The above are the specific processing steps of this system.

[0769] (Application example 1)

[0770] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0771] In traditional restaurants, there were limited ways to understand customer satisfaction and dissatisfaction in real time, which often delayed the timing of service improvements. It was also difficult to quantitatively analyze customer sentiment and derive specific improvement measures. Furthermore, there was a lack of alert functions to respond quickly when specific problems occurred.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0773] In this invention, the server includes a feedback generation means for proposing service improvement measures based on the determined emotional state, an alert means for notifying staff or managers of an alert when a specific emotion is recognized frequently, and a data analysis means for analyzing past customer data and generating periodic reports. This makes it possible to grasp customer emotions in real time and quickly propose appropriate service improvement measures. Furthermore, when a specific problem occurs, an alert is sent immediately, encouraging a prompt response. Furthermore, the periodic reports can be compared with past data to derive more accurate service improvement measures.

[0774] "Data collection means" refers to a device or group of devices for collecting face data and voice data of customers.

[0775] "Facial expression recognition means" refers to an algorithm or device for recognizing customer facial expressions from collected facial data.

[0776] The "voice analysis means" refers to an algorithm or device for analyzing the customer's tone of voice from collected voice data.

[0777] The "emotion analysis means" refers to an algorithm or device for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means.

[0778] A "feedback generator" is an algorithm or device that suggests service improvements based on the determined emotional state.

[0779] An "alert means" is a device or group of devices that alerts staff or managers when a specific emotion is recognized frequently.

[0780] "Data analysis tools" are algorithms or devices used to analyze historical customer data and generate periodic reports.

[0781] The "transmission means" refers to a device or group of devices for transmitting the collected face data and voice data using a secure communication protocol.

[0782] The "data storage means" refers to a device or group of devices for storing the emotional states obtained from the facial expression recognition means and the voice analysis means in a database.

[0783] The "analysis means" refers to an algorithm or device for comparing and analyzing new data with past data stored in the data storage means.

[0784] To implement this invention, a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, an alert means, and a data analysis means are mainly required, as well as a transmission means for securely transmitting the collected data, and a data storage means and analysis means for storing and analyzing the data.

[0785] Data collection

[0786] The device uses high-resolution cameras and noise-canceling microphones installed in the store to collect customer facial and voice data in real time, allowing for accurate capture of customer facial expressions and tone of voice.

[0787] Data transmission

[0788] Facial and audio data collected on the device is transmitted to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0789] Data Preprocessing and Analysis

[0790] The data that arrives at the server is first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points. For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features.

[0791] Emotion analysis

[0792] The preprocessed data is then passed to an emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This analysis allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0793] Feedback and Alerts

[0794] The server then uses a feedback generator to generate optimal service improvement measures based on the determined emotional state. For example, specific suggestions such as "We need to hire more staff" or "We should provide more detailed explanations for certain menu items" are generated. Additionally, an alerting mechanism immediately notifies staff and managers when a specific emotion is detected frequently.

[0795] Data Storage and Analysis

[0796] The emotional data is stored in a database and compared with past data using data analysis tools. This allows trends and patterns, such as customer reactions to specific time periods or menu items, to be visualized. Furthermore, aggregated reports of the emotional data are periodically created and distributed to managers.

[0797] Specific examples

[0798] For example, suppose a customer enters a restaurant during lunchtime, and the camera captures their smile and the microphone records their positive conversation. This data is sent to a server, where a deep learning model determines their happiness. Based on this information, feedback is generated that says "This customer is satisfied," and the information is shared with staff. This data is also stored in a database and used to improve service in the future.

[0799] Here are some example prompts to input to a generative AI model:

[0800] "The camera captures a smiling customer entering your restaurant during lunchtime, and the microphone records his positive conversation. Based on this situation, generate feedback and improvement suggestions if the analysis determines that he is happy."

[0801] This invention makes it possible to grasp customer sentiment in real time and quickly propose appropriate service improvement measures. It also sends an immediate alert when a specific problem occurs, encouraging a prompt response. Furthermore, periodic reports can be compared with past data to derive more accurate service improvement measures.

[0802] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0803] Step 1:

[0804] The terminal uses a high-resolution camera and a noise-canceling microphone installed in the store to collect customer facial and voice data in real time. The inputs for collection are camera images and microphone audio. To obtain this data clearly, the camera recognizes the customer's face and the microphone removes background noise. The output is the customer's facial image data and voice data.

[0805] Step 2:

[0806] The device sends the collected face data and voice data to the server using a secure communication protocol (e.g., HTTPS). The input is the face image data and voice data collected in step 1. The data is encrypted and sent to the server over the network. The output is the face image data and voice data that arrives at the server.

[0807] Step 3:

[0808] The server preprocesses the received facial image data using an image processing library (e.g., OpenCV). The input is the facial image data received by the server. Facial feature points (e.g., the positions of the eyes, nose, and mouth) are extracted through image processing. The output is the extracted facial feature points.

[0809] Step 4:

[0810] The server preprocesses the incoming audio data using an audio processing library (e.g., SciPy). The input is the audio data arriving at the server. The audio data is Fourier transformed and the frequency spectrum and features (e.g., pitch, volume, and tone) are analyzed. The output is the analyzed audio features.

[0811] Step 5:

[0812] The server analyzes emotions from the preprocessed facial data using a convolutional neural network (CNN) model. The input is the facial feature points extracted in step 3. The CNN model determines emotions such as "happiness," "anger," and "sadness" with high accuracy from these feature points. The output is the analyzed emotion data.

[0813] Step 6:

[0814] The server analyzes the preprocessed speech data for emotion using a recurrent neural network (RNN) model. The input is the speech features analyzed in step 4. The RNN model determines the emotion from the speech data. The output is the analyzed emotion data.

[0815] Step 7:

[0816] The server uses the feedback generation means to propose optimal service improvement measures based on the emotional state of the face data and voice data obtained from the emotion analysis means. The input is the emotional data obtained in steps 5 and 6. The algorithm generates specific service improvement proposals. The output is the proposed service improvement measures.

[0817] Step 8:

[0818] The server uses an alerting means to notify staff or administrators when a specific emotion is recognized frequently. The input is emotion data from the emotion analysis means. An alert is generated based on specific conditions and a notification is sent. The output is an alert notification to staff or administrators.

[0819] Step 9:

[0820] The server stores the emotion data in a database and performs comparative analysis with past data using data analysis means. The input is the emotion data obtained in steps 7 and 8. This allows analysis of customer reactions to specific time periods and menu items, and visualizes trends. The output is the analysis results and data stored in the database.

[0821] Step 10:

[0822] The server periodically generates an aggregate report of the emotion data and delivers it to the administrator. The input is the emotion data saved in step 9. The aggregate report includes weekly and monthly emotion trends, customer satisfaction trends, improvement suggestions, etc. The output is the emotion data report delivered to the administrator.

[0823] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0824] The present invention is a system for understanding customer emotions in real time at restaurants, helping to improve service and feedback accuracy. This system is composed of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, a transmission means, a data storage means, and an emotion engine.

[0825] Collection of customer data

[0826] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with background noise reduction.

[0827] Sending data

[0828] On the device: The collected face and voice data is sent to the server in batches (e.g., every 10 seconds). The data is encrypted using a secure communication protocol (e.g., HTTPS).

[0829] Data preprocessing and analysis

[0830] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, a Fourier transform is performed using an audio processing library (e.g., SciPy) to analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0831] Emotion assessment and integration

[0832] Server: The pre-processed data is first processed using facial expression recognition means to determine customer emotions using a deep learning model (e.g., CNN). Similarly, audio analysis means is used to determine emotions from audio data using a machine learning model (e.g., RNN). These data are then integrated by the emotion engine to determine the overall customer emotional state.

[0833] Data storage and analysis

[0834] Server: The determined emotion data is stored in a database and compared with past data for analysis. Here, customer reactions to specific times of day or menu items are analyzed, and trends and patterns are visualized.

[0835] Generate feedback

[0836] Server: Based on the sentiment data, the feedback generator automatically suggests optimal service improvement measures, such as "increase staffing during certain times" or "add explanations for certain menu items."

[0837] Alerts and Reports

[0838] Server: Based on the results of the emotion data analysis, real-time alerts are sent to administrators when certain conditions are met. Aggregated reports of emotion data are also periodically created and distributed to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement.

[0839] Specific examples

[0840] Device: The in-store camera captures the customer's smile, and the microphone simultaneously records the customer's positive conversation. These data are sent to the server, and feature points of the video and audio are extracted.

[0841] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[0842] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0843] In this way, the system can grasp customer sentiment in real time and provide accurate feedback and solutions to improve the dining experience.

[0844] The processing flow will be explained below.

[0845] Step 1:

[0846] Terminal: A camera installed in the store captures customer facial data in real time. Specifically, the camera takes a certain number of frames per second (e.g., 30 frames per second) and detects facial regions using a face detection algorithm (e.g., Haar cascade). The detected facial regions are segmented and passed to subsequent processing steps.

[0847] Step 2:

[0848] Device: A microphone installed in the store collects customer voice data. Specifically, the microphone records the voice at a certain sample rate (e.g., 44.1 kHz), and the voice data is processed using noise cancellation and then saved in an audio file format (e.g., WAV file).

[0849] Step 3:

[0850] Device: Sends face and voice data to the server in batches of a certain size (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it encrypts the data using a secure communication protocol (e.g., HTTPS) and sends it to the server.

[0851] Step 4:

[0852] Server: Preprocesses the face data. Specifically, it uses an image processing library (e.g., OpenCV) to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). This process converts the data into a format that can be used by the deep learning model. The extracted feature points are also normalized and passed on to the next step.

[0853] Step 5:

[0854] Server: Preprocesses the audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and analyze the frequency spectrum. It also extracts features such as pitch, tone, and speed, and converts them into input data for emotion analysis.

[0855] Step 6:

[0856] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[0857] Step 7:

[0858] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[0859] Step 8:

[0860] Server: The emotion data obtained from facial expression recognition and voice analysis is integrated into the emotion engine to determine the overall emotional state of the customer. Specifically, the emotion engine integrates the facial expression data and voice data to calculate a single integrated emotion score. This information is stored in a database.

[0861] Step 9:

[0862] Server: Analyzes the integrated sentiment data and extracts trends and patterns. For example, analyzes customer reactions to specific times or menu items and generates statistical information. Data analysis tools (e.g., Pandas, NumPy) are used for the analysis.

[0863] Step 10:

[0864] Server: Generates feedback based on emotion data. Specifically, the feedback generator automatically creates specific improvement suggestions, such as "increase staff during this time period" or "add explanations for specific menu items."

[0865] Step 11:

[0866] Server: Sends real-time alerts to administrators when certain conditions are met. For example, if customer satisfaction is low or if there is a high level of dissatisfaction with a particular menu item, an alert will be sent to the administrator via email or SMS.

[0867] Step 12:

[0868] Server: Periodically creates aggregated reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement. The reports are generated in PDF format and sent via email.

[0869] Example 2

[0870] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0871] Restaurants are required to understand the emotional state of customers in real time and use that information to improve and optimize the quality of their services. However, conventional systems collect, analyze, store, and generate feedback from facial and voice data separately, making efficient and integrated operation difficult.

[0872] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image capture means for collecting customer facial data, an audio collection means for collecting customer voice data, a transmission means for transmitting the collected facial data, a transmission means for transmitting the collected voice data, an image processing means for preprocessing the collected facial data, an audio processing means for preprocessing the collected voice data, an expression analysis means for recognizing the customer's facial expression from the preprocessed facial data, an audio analysis means for analyzing the customer's tone of voice from the preprocessed audio data, an emotion analysis means for integrating data obtained from the expression analysis means and the audio analysis means to determine the customer's emotional state, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to efficiently integrate the customer's emotional state in real time and quickly suggest optimal service improvement measures.

[0873] "Image Capture Device" means a device used to collect customer facial data in real time.

[0874] An "audio collection means" is a device used to collect customer audio data.

[0875] "Transmission means" refers to a communication device or protocol for transmitting collected data to a server.

[0876] "Image processing means" refers to software or hardware for preprocessing collected face data and extracting feature points.

[0877] The "acoustic processing means" refers to software or hardware for preprocessing collected audio data and analyzing its features.

[0878] "Facial expression analysis means" refers to algorithms or models used to recognize customer emotions from pre-processed facial data.

[0879] "Voice analysis means" refers to algorithms or models for analyzing the customer's tone of voice from pre-processed voice data.

[0880] The "emotion analysis means" is a system for integrating data obtained from the facial expression analysis means and the voice analysis means to determine the emotional state of the customer.

[0881] The "feedback generation means" is a mechanism for suggesting service improvements based on the determined emotional state.

[0882] The present invention is a system for understanding customer emotions in real time at restaurants and optimizing and improving service quality based on that information. This system is composed of an image capture means, an audio collection means, a transmission means, an image processing means, an audio processing means, a facial expression analysis means, a voice analysis means, an emotion analysis means, and a feedback generation means.

[0883] First, cameras and microphones installed in the store function as image capture and audio collection devices. The high-resolution cameras collect customer facial data in real time, while the microphones record customer conversations. The cameras capture details from the moment customers enter the store to the moment they smile, while the microphones record the conversations while filtering out surrounding noise.

[0884] This data is then sent from the device to the server in batches (e.g., every 10 seconds) using a secure communication protocol (e.g., HTTPS) to encrypt and transmit the data safely. This process reduces the risk of data eavesdropping or tampering.

[0885] The data that arrives at the server is first preprocessed using image processing and audio processing means. Specifically, an image processing library (e.g., OpenCV) is used to extract facial feature points (the positions of the eyes, nose, and mouth, etc.), and an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data, analyzing the frequency spectrum and features (pitch, volume, tone, etc.).

[0886] The preprocessed data is then subjected to emotion recognition using facial expression analysis means within the server. This analysis uses a deep learning model (e.g., CNN) to determine the customer's emotion from the image data. Similarly, emotion is determined from the voice data using a machine learning model (e.g., RNN) through the voice analysis means. The results of both analyses are integrated by the emotion analysis means to comprehensively determine the customer's final emotional state.

[0887] The emotional state determined is stored in a database and compared with past data using analytical tools. This analysis visualizes customer reactions to specific time periods and menu items, highlighting trends and patterns. For example, a trend such as "a particular pasta menu item is popular during lunchtime" can be analyzed.

[0888] Finally, the feedback generation means automatically proposes service improvement measures based on the emotional data. For example, a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well" may be automatically generated. Furthermore, when certain conditions are met, an alert is sent to the manager in real time, and a summary report of the emotional data is periodically created and distributed to the manager.

[0889] As a concrete example, suppose an in-store camera captures a customer's smile, while a microphone simultaneously records a customer's positive conversation. This data is sent to a server, where feature points from the video and audio are extracted. Using CNN and RNN models, the smile and positive tone of voice are used to determine the emotion of "happiness." The recognized emotion data is then stored in a database, and analysis derives a trend, such as "a particular lunchtime menu item is popular." As feedback, a suggestion is generated to "serve this menu item at dinnertime as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[0890] The system allows for real-time understanding of customer sentiment and provides targeted feedback and improvements to improve the dining experience.

[0891] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0892] Step 1:

[0893] Terminal: A camera installed in the store captures customers' faces in real time. Specifically, the camera captures high-resolution video and outputs each frame as image data. The input is a video of customers in the store, and the output is image data containing the customer's face.

[0894] Step 2:

[0895] Terminal: A microphone records customer conversations in real time. Noise-canceling technology is used to remove background noise and obtain clean audio data. The input is the in-store audio environment, and the output is audio data with the customer's conversations emphasized.

[0896] Step 3:

[0897] Terminal: The collected face data and audio data are batch processed (e.g., every 10 seconds) and sent to the server. When sent, the data is encrypted using a secure communication protocol (e.g., HTTPS). The input is image data and audio data, and the output is the encrypted transmission data.

[0898] Step 4:

[0899] Server: Receives the transmitted face data and performs preprocessing using an image processing library (e.g., OpenCV). Specifically, the processing involves extracting facial feature points (e.g., the positions of the eyes, nose, and mouth). The input is encrypted image data, and the output is preprocessed image data with feature points extracted.

[0900] Step 5:

[0901] Server: Receives the transmitted audio data and performs preprocessing using an audio processing library (e.g., SciPy). Performs a Fourier transform on the audio data and analyzes the frequency spectrum and features (e.g., pitch, volume, tone). The input is encrypted audio data, and the output is preprocessed audio data with the frequency spectrum and features analyzed.

[0902] Step 6:

[0903] Server: Using the preprocessed image data, a deep learning model (e.g., CNN) is run to analyze facial expressions and determine the customer's emotions. Specifically, the model outputs emotion labels such as "joy," "anger," and "sadness" from the input image data. The input is the preprocessed image data, and the output is the emotion label.

[0904] Step 7:

[0905] Server: Using the preprocessed voice data, a machine learning model (e.g., RNN) is run to analyze the voice and determine the emotion from the customer's tone of voice. Specifically, the model outputs an emotion label such as "satisfied" or "dissatisfied" from the input voice data. The input is the preprocessed voice data, and the output is the emotion label.

[0906] Step 8:

[0907] Server: The emotion analysis means integrates the emotion labels obtained from the facial expression analysis means and the voice analysis means to determine the overall emotional state of the customer. For example, if the facial expression of "happiness" matches the tone of voice of "satisfied," the overall judgment is "very satisfied." The input is the emotion labels of facial expressions and voice, and the output is the integrated emotional state.

[0908] Step 9:

[0909] Server: Stores the determined emotional state in a database. It then performs comparative analysis with past data, analyzes customer reactions to specific time periods and menu items, and visualizes trends and patterns. For example, it analyzes the trend that "a specific lunchtime menu item is popular." The input is the integrated emotional state data, and the output is the visualized results of trends and patterns.

[0910] Step 10:

[0911] Server: The feedback generation means automatically proposes optimal service improvement measures based on the emotion data. For example, it generates a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well." The input is the analysis results and emotion data, and the output is a service improvement proposal.

[0912] Step 11:

[0913] Server: When a specific condition is met (for example, when customer satisfaction declines), it sends an alert to the administrator in real time. It also periodically creates aggregate reports of emotion data and delivers them to the administrator. The input is emotion data that meets the condition, and the output is the alert notification and aggregate reports.

[0914] (Application example 2)

[0915] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0916] Traditional customer service in restaurants relies heavily on staff experience and intuition, making it difficult to accurately grasp customer emotions and satisfaction. Furthermore, there are limited means to grasp customers' real-time emotional states and provide appropriate feedback based on them, resulting in a lack of speed in service improvement. Therefore, in order to improve customer satisfaction and service quality, real-time emotion analysis and immediate feedback based on the results are required.

[0917] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting customer facial data, a data collection means for collecting customer voice data, a facial expression recognition means for recognizing the customer's facial expression from the collected facial data, a voice analysis means for analyzing the customer's tone of voice from the collected voice data, an emotion analysis means for determining the customer's emotional state based on the data obtained by the facial expression recognition means and the voice analysis means, a display means for displaying the determined emotional state in real time, a feedback presentation means for providing feedback to the staff based on the emotional state displayed in real time, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to accurately grasp the customer's emotional state in real time and provide immediate feedback based on that.

[0918] The "data collection means" is a mechanism for collecting face data and voice data of customers.

[0919] The "facial expression recognition means" is a mechanism that analyzes the facial expression of the customer from the collected facial data and recognizes that expression.

[0920] The "voice analysis means" is a mechanism that analyzes collected voice data to analyze the customer's voice tone and other characteristics.

[0921] The "emotion analysis means" is a mechanism that determines the emotional state of the customer using data obtained from the facial expression recognition means and the voice analysis means.

[0922] The "display means" is a mechanism for displaying the determined emotional state in real time.

[0923] A "feedback presentation means" is a mechanism that provides feedback to staff based on their emotional state displayed in real time.

[0924] A "feedback generator" is a mechanism that automatically suggests service improvements based on the determined emotional state.

[0925] The "transmission means" is a mechanism for transmitting the collected face data and voice data to a server using a secure communication protocol.

[0926] The "data storage means" is a mechanism used to store the emotional states obtained from the facial expression recognition means and the voice analysis means.

[0927] The "analysis means" is a mechanism for comparing and analyzing stored data with past data.

[0928] System configuration

[0929] The present invention is a system that analyzes customer sentiment in real time and provides feedback to staff. This system is broadly composed of data collection means, data analysis means, feedback means, transmission means, data storage means, and analysis means.

[0930] Hardware and Software Use

[0931] The present invention is implemented using the following hardware and software:

[0932] Hardware: smart glasses, camera, microphone

[0933] Software: OpenCV (face recognition), TensorFlow (sentiment analysis), SciPy (voice analysis), HTTP request library (data transmission)

[0934] Data collection methods

[0935] The device collects customer facial and voice data in real time using the smart glasses' built-in camera and microphone. The camera captures high-resolution footage, and the microphone has the ability to filter out background noise.

[0936] Data Analysis Methods

[0937] To analyze the collected data, the server first extracts facial feature points using an image processing library (e.g., OpenCV) for face data, performs a Fourier transform on the voice data using a voice processing library (e.g., SciPy) to analyze the frequency spectrum and features, and then uses a deep learning model (e.g., TensorFlow) to determine the customer's emotional state.

[0938] Feedback Methods

[0939] The server then displays the determined emotional state on the staff member's smart glasses in real time, allowing the staff member to receive real-time feedback to provide optimal service based on the customer's emotions. Feedback is provided using audio guidance and visual alerts.

[0940] Means of transmission and data storage

[0941] The server sends the collected facial and voice data to a central server using a secure communication protocol (e.g., HTTPS), where it is stored in a database and used for analysis.

[0942] Data analysis and feedback generation

[0943] The emotion data stored in the data storage means is periodically analyzed by the analysis means. Here, customer responses to specific time periods and services are compared and analyzed, and trends and patterns are visualized. The feedback generation means then proposes optimal service improvement measures based on the analysis results.

[0944] Specific examples

[0945] For example, suppose a staff member wearing smart glasses captures a customer's smile while serving them. The system analyzes the video and recognizes the emotion of "happiness." It then displays feedback to the staff member's smart glasses, saying, "This customer is enjoying themselves. Please consider providing additional services." This feedback allows the staff member to immediately provide the optimal service to the customer.

[0946] Prompt Sentence Examples

[0947] "The following Python program implements a system that analyzes customer emotions in real time. The system uses OpenCV for facial recognition from camera footage and TensorFlow for emotion recognition. Audio data is analyzed using SciPy, and the data is sent to a server using HTTPS. Please complete the following code to enhance the emotion recognition functionality."

[0948] In this way, the present invention is a system that increases customer satisfaction by grasping customer emotions in real time and providing immediate feedback based on the emotions.

[0949] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0950] Step 1:

[0951] The device uses a camera and microphone to collect customer facial and audio data. The camera captures high-resolution video, and the microphone filters out background noise. The input is real-time facial video and audio, and the output is raw video and audio data.

[0952] Step 2:

[0953] The server receives the video data sent from the device and extracts facial feature points using an image processing library (e.g., OpenCV). The input is raw video data, and the output is processed image data containing facial feature points (e.g., the positions of the eyes, nose, and mouth).

[0954] Step 3:

[0955] The server simultaneously receives the audio data sent from the device and performs a Fourier transform using an audio processing library (e.g., SciPy). This analyzes the frequency spectrum and features (e.g., pitch, volume, and tone) of the audio data. The input is raw audio data, and the output is audio data with extracted features.

[0956] Step 4:

[0957] The server inputs the processed image data into a deep learning model (e.g., TensorFlow) and analyzes the customer's facial expression using facial expression recognition. The input is image data containing facial feature points, and the output is the analyzed facial expression (e.g., "happiness," "sadness," etc.).

[0958] Step 5:

[0959] The server inputs the feature-extracted voice data into a machine learning model (e.g., RNN) and analyzes the customer's voice tone using a voice analysis method. The input is the feature-extracted voice data, and the output is the analyzed emotional state (e.g., "joy," "anger," etc.).

[0960] Step 6:

[0961] The server integrates the data obtained from the facial expression recognition means and the voice analysis means and determines the overall emotional state of the customer using the emotion analysis means, where the input is the emotion result from the facial expression data and the voice data, and the output is the integrated emotional state.

[0962] Step 7:

[0963] The server displays the determined emotional state in real time on the staff member's smart glasses using a display means, where the input is the integrated emotional state and the output is feedback information that the staff member can see through the glasses.

[0964] Step 8:

[0965] The server provides feedback to staff using a feedback presentation tool based on the emotional state displayed in real time. The input is the displayed emotional state, and the output is a specific action or suggestion (e.g., "The customer is enjoying themselves. Please consider additional services.").

[0966] Step 9:

[0967] The server transmits the collected face and audio data to a central server using a secure communication protocol (e.g., HTTPS). The input is raw video and audio data, and the output is encrypted data communication.

[0968] Step 10:

[0969] The server stores the transmitted data in a database and compares it with past data. The input is the stored emotion data, and the output is the analysis results and visualization of trends and patterns.

[0970] Step 11:

[0971] The server uses a feedback generation means to propose optimal service improvement measures based on the analysis results. The input is the analysis results, and the output is automatically generated service improvement proposals.

[0972] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0973] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0974] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0975] [Fourth embodiment]

[0976] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0977] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0978] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0979] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0980] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0981] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0982] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0983] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0984] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0985] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0986] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0987] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0988] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0989] The present invention is a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly consists of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means.

[0990] Collection of customer data

[0991] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with noise-canceling functions to eliminate background noise and record customer voices clearly.

[0992] Sending data

[0993] Device: Collected face and voice data is sent to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[0994] Data preprocessing and analysis

[0995] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[0996] Emotion determination

[0997] Server: The preprocessed data is then passed to the emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[0998] Data storage and analysis

[0999] Server: The determined emotion data is stored in a database and analyzed by an emotion analysis tool, comparing it with past data. This allows trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[1000] Generate feedback

[1001] Server: Based on the stored emotion data, the system uses feedback generation methods to suggest optimal service improvement measures, such as "We should increase the number of staff during lunch hours" or "We should provide more detailed explanations about certain menu items."

[1002] Alerts and Reports

[1003] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator when certain conditions are met, enabling prompt action in real time. Aggregated reports of emotion data are also periodically created and distributed to administrators. These reports include weekly and monthly emotion trends, customer satisfaction trends, and suggestions for improvement.

[1004] Specific examples

[1005] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[1006] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[1007] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[1008] In this way, the system can understand customer sentiment in real time and significantly improve the dining experience.

[1009] The processing flow will be explained below.

[1010] Step 1:

[1011] Terminal: In-store cameras capture customer facial data in real time. Specifically, the camera takes a certain number of frames per second and performs image processing to detect the customer's face. For example, the OpenCV library is used to detect faces and extract the face area from each frame.

[1012] Step 2:

[1013] Device: A microphone installed in the store collects customer voice data. Specifically, a noise-canceling microphone records customer conversations and saves the voice data in a format (e.g., WAV file) using an audio processing library (e.g., PyAudio).

[1014] Step 3:

[1015] Device: Collected face and voice data is sent to the server in batches (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it is encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server.

[1016] Step 4:

[1017] Server: Preprocesses the received face data using image processing libraries (e.g., OpenCV and Dlib) to identify facial regions and extract key feature points (e.g., the positions of the eyes, nose, and mouth). The feature points are normalized and converted into a format suitable for input to the deep learning model.

[1018] Step 5:

[1019] Server: Preprocesses the received audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and obtain the frequency spectrum. It then extracts features such as pitch, tone, and speed and converts them into input data for sentiment analysis.

[1020] Step 6:

[1021] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[1022] Step 7:

[1023] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[1024] Step 8:

[1025] Server: Integrates emotional data obtained from facial expression recognition and voice analysis to determine the overall emotional state of the customer, which is then stored in a database in a format that can be searched and compared later.

[1026] Step 9:

[1027] Server: Analyzes the integrated sentiment data to extract trends and patterns, for example, analyzing customer reactions to specific times of day or menu items, and generating statistics.

[1028] Step 10:

[1029] Server: Generates feedback based on emotional data. Specifically, it automatically creates improvement suggestions such as "increase staff during this time" or "add explanations to specific menu items."

[1030] Step 11:

[1031] Server: Sends real-time alerts to administrators when certain conditions are met (e.g., a large number of customers are unhappy). Alerts can be sent, for example, via email or SMS.

[1032] Step 12:

[1033] Server: Periodically creates aggregate reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement, providing reference for administrators to take appropriate measures.

[1034] Example 1

[1035] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1036] Conventionally, restaurants have lacked a means to quickly and accurately grasp customer sentiment in order to improve customer service. This can result in lower customer satisfaction and delays in service improvements, potentially affecting the restaurant's reputation and revenue. Furthermore, the collection and analysis of customer feedback is often done manually, limiting the accuracy and speed of the analysis results. This invention solves these problems and provides a system for grasping customer sentiment in real time and utilizing it to improve service.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1038] In this invention, the server includes means for collecting customer facial data, means for collecting customer voice data, means for recognizing customer facial expressions from the collected facial data, means for analyzing customer tones from the collected voice data, means for determining the customer's emotional state based on data obtained by the facial expression recognition means and the voice analysis means, means for proposing service improvement measures based on the determined emotional state, means for sending an alert to a manager when the customer's emotional data meets certain conditions, and means for periodically creating and distributing a summary report of the emotional data to the manager. This makes it possible to accurately grasp customer emotions in real time and improve services quickly and effectively.

[1039] "Customer" refers to a user who visits a restaurant and uses the service.

[1040] "Facial data" refers to image information of a customer's face captured by an image capture device such as a camera.

[1041] "Voice data" refers to audio information of customer voices and conversations recorded using a voice capture device such as a microphone.

[1042] "Means" refers to any device or method, or combination thereof, used to accomplish a particular purpose.

[1043] "Facial expression" refers to facial expressions that change depending on the movement of a person's facial muscles.

[1044] "Tone" refers to vocal characteristics such as pitch, volume, and rhythm.

[1045] "Emotional state" refers to a specific emotion a customer feels, such as joy, anger, or sadness.

[1046] "Sentiment analysis" refers to the technology of determining customer emotions from collected data.

[1047] "Service Improvement Measures" refers to specific measures or action plans proposed to increase customer satisfaction.

[1048] An "alert" is a notification or warning to an administrator when a pre-defined condition is met.

[1049] An "aggregate report" refers to a report that analyzes and organizes collected data and summarizes the results.

[1050] "Means and" refers to a combination in which the aforementioned means and methods work together to perform a single function.

[1051] The present invention relates to a system for understanding customer emotions in real time at restaurants, which is useful for improving service and increasing the accuracy of feedback. This system mainly comprises a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, and a transmission means. Specific examples are shown below.

[1052] Collection of customer data

[1053] Device: High-resolution cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture video at 30 frames per second in 1080p resolution. The microphones are noise-canceling, eliminating background noise and allowing customers' voices to be recorded clearly.

[1054] Sending data

[1055] Device: The collected face and voice data is sent to the server using a secure communication protocol (e.g., HTTPS) with AES-256 encryption, which prevents unauthorized access and tampering of the data and protects privacy.

[1056] Data Preprocessing

[1057] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, the audio processing library SciPy is used to perform a Fourier transform and analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[1058] Emotion determination

[1059] Server: The preprocessed data is passed to the emotion analysis means. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[1060] Data storage and analysis

[1061] Server: The determined emotion data is stored in a database and analyzed against past data, allowing trends and patterns to be visualized, such as customer reactions to specific times of day or menu items.

[1062] Generate feedback

[1063] Server: Based on the stored emotion data, the server uses a feedback generator to suggest optimal service improvement measures. For example, specific suggestions such as "We should increase the number of staff during lunchtime" or "We should provide more detailed explanations about specific menu items" are automatically generated.

[1064] Alerts and Reports

[1065] Server: Based on the results of the emotion data analysis, an alert is sent to the administrator if certain conditions are met. For example, if a customer's "anger" emotion exceeds a certain threshold, an alert is sent immediately. In addition, aggregated reports of emotion data are periodically created and delivered to the administrator. The reports include weekly and monthly emotion trends, changes in customer satisfaction, and suggestions for improvement.

[1066] Specific examples

[1067] Terminal: For example, a camera in a store captures a customer's smile, and a microphone simultaneously records the customer's positive conversation. These data are sent to a server, where feature points of the video and audio are extracted.

[1068] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[1069] Server: Feedback will be generated suggesting "serve this menu item at dinner time as well." Based on this trend, the manager can receive a report saying "customer satisfaction is high at lunchtime."

[1070] Prompt Sentence Examples

[1071] "Cameras and microphones in the store captured customer smiles and positive conversations. Could you please outline a program that analyzes this data to determine customer sentiment and generate feedback?"

[1072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1073] Step 1: Collect customer data

[1074] Terminal: A high-resolution camera installed in the store activates and collects facial data of customers in the store in real time. As input, it captures video at 30 frames per second at 1080p resolution. As output, it generates video frames containing facial features. At the same time, a noise-canceling microphone records the customer's conversation and collects its audio data. As input, the customer's speech is collected in real time, and as output, clear audio data with noise removed is provided.

[1075] Step 2: Sending data

[1076] Terminal: The collected face and voice data is sent to the server using the secure communication protocol HTTPS. As input, face and voice data are provided, and as output, encrypted data packets are sent to the server. During this process, the data is encrypted using the AES-256 encryption method to ensure security during transmission.

[1077] Step 3: Data Preprocessing

[1078] Server: Data arriving at the server is first preprocessed. The transmitted face data and audio data are provided as input. For the face data, the image processing library OpenCV is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For example, the distance from one eye corner to the other and the shape of the mouth are output as numerical data. For the audio data, a Fourier transform is performed using the SciPy library to calculate the frequency spectrum and feature quantities (e.g., pitch, volume, and tone). As the output of this processing, a dataset for each feature point is generated.

[1079] Step 4: Determine the emotion

[1080] Server: The preprocessed data is fed into the emotion analysis model. Facial feature points and audio features are provided as input. For facial data, a convolutional neural network (CNN) model is used to identify emotions such as "happiness," "anger," and "sadness" from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) model is used to accurately determine emotions from vocal tones. The output is the detected emotion label and its confidence level.

[1081] Step 5: Save your data

[1082] Server: The determined emotion data is stored in a database and used for analysis. As input, emotion labels and their confidence levels are provided. This data is recorded in the database as time-series data. As output, emotion data including time information is stored in the database.

[1083] Step 6: Generate feedback

[1084] Server: Based on the stored emotion data, the feedback generation means creates service improvement proposals. The most recent emotion data and past emotion data are provided as input. Based on comparative analysis with past data, specific trends and patterns, such as "certain menu items are highly rated at lunchtime," are derived. Specific service improvement measures (e.g., improving the menu, increasing the number of staff, etc.) are generated as output.

[1085] Step 7: Alerts and Reports

[1086] Server: Based on the results of sentiment analysis, an alert is sent to the administrator when certain conditions are met. Real-time sentiment data is provided as input. For example, if a customer's "anger" sentiment exceeds a certain threshold, an alert is sent immediately. As output, a notification is generated for the administrator. In addition, aggregated reports of sentiment data are periodically created and delivered to the administrator. As output, reports are created that include weekly and monthly sentiment trends, customer satisfaction levels, and improvement suggestions.

[1087] The above are the specific processing steps of this system.

[1088] (Application example 1)

[1089] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1090] In traditional restaurants, there were limited ways to understand customer satisfaction and dissatisfaction in real time, which often delayed the timing of service improvements. It was also difficult to quantitatively analyze customer sentiment and derive specific improvement measures. Furthermore, there was a lack of alert functions to respond quickly when specific problems occurred.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1092] In this invention, the server includes a feedback generation means for proposing service improvement measures based on the determined emotional state, an alert means for notifying staff or managers of an alert when a specific emotion is recognized frequently, and a data analysis means for analyzing past customer data and generating periodic reports. This makes it possible to grasp customer emotions in real time and quickly propose appropriate service improvement measures. Furthermore, when a specific problem occurs, an alert is sent immediately, encouraging a prompt response. Furthermore, the periodic reports can be compared with past data to derive more accurate service improvement measures.

[1093] "Data collection means" refers to a device or group of devices for collecting face data and voice data of customers.

[1094] "Facial expression recognition means" refers to an algorithm or device for recognizing customer facial expressions from collected facial data.

[1095] The "voice analysis means" refers to an algorithm or device for analyzing the customer's tone of voice from collected voice data.

[1096] The "emotion analysis means" refers to an algorithm or device for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means.

[1097] A "feedback generator" is an algorithm or device that suggests service improvements based on the determined emotional state.

[1098] An "alert means" is a device or group of devices that alerts staff or managers when a specific emotion is recognized frequently.

[1099] "Data analysis tools" are algorithms or devices used to analyze historical customer data and generate periodic reports.

[1100] The "transmission means" refers to a device or group of devices for transmitting the collected face data and voice data using a secure communication protocol.

[1101] The "data storage means" refers to a device or group of devices for storing the emotional states obtained from the facial expression recognition means and the voice analysis means in a database.

[1102] The "analysis means" refers to an algorithm or device for comparing and analyzing new data with past data stored in the data storage means.

[1103] To implement this invention, a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, an alert means, and a data analysis means are mainly required, as well as a transmission means for securely transmitting the collected data, and a data storage means and analysis means for storing and analyzing the data.

[1104] Data collection

[1105] The device uses high-resolution cameras and noise-canceling microphones installed in the store to collect customer facial and voice data in real time, allowing for accurate capture of customer facial expressions and tone of voice.

[1106] Data transmission

[1107] Facial and audio data collected on the device is transmitted to a server using a secure communication protocol (e.g., HTTPS), which prevents unauthorized access or tampering of the data and protects privacy.

[1108] Data Preprocessing and Analysis

[1109] The data that arrives at the server is first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points. For audio data, an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data and analyze the frequency spectrum and features.

[1110] Emotion analysis

[1111] The preprocessed data is then passed to an emotion analysis tool. For facial data, a deep learning model (e.g., a convolutional neural network (CNN)) is used to determine emotions from facial expressions. Similarly, for audio data, a recurrent neural network (RNN) is used to determine emotions. This analysis allows emotions such as "happiness," "anger," and "sadness" to be identified with high accuracy.

[1112] Feedback and Alerts

[1113] The server then uses a feedback generator to generate optimal service improvement measures based on the determined emotional state. For example, specific suggestions such as "We need to hire more staff" or "We should provide more detailed explanations for certain menu items" are generated. Additionally, an alerting mechanism immediately notifies staff and managers when a specific emotion is detected frequently.

[1114] Data Storage and Analysis

[1115] The emotional data is stored in a database and compared with past data using data analysis tools. This allows trends and patterns, such as customer reactions to specific time periods or menu items, to be visualized. Furthermore, aggregated reports of the emotional data are periodically created and distributed to managers.

[1116] Specific examples

[1117] For example, suppose a customer enters a restaurant during lunchtime, and the camera captures their smile and the microphone records their positive conversation. This data is sent to a server, where a deep learning model determines their happiness. Based on this information, feedback is generated that says "This customer is satisfied," and the information is shared with staff. This data is also stored in a database and used to improve service in the future.

[1118] Here are some example prompts to input to a generative AI model:

[1119] "The camera captures a smiling customer entering your restaurant during lunchtime, and the microphone records his positive conversation. Based on this situation, generate feedback and improvement suggestions if the analysis determines that he is happy."

[1120] This invention makes it possible to grasp customer sentiment in real time and quickly propose appropriate service improvement measures. It also sends an immediate alert when a specific problem occurs, encouraging a prompt response. Furthermore, periodic reports can be compared with past data to derive more accurate service improvement measures.

[1121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1122] Step 1:

[1123] The terminal uses a high-resolution camera and a noise-canceling microphone installed in the store to collect customer facial and voice data in real time. The inputs for collection are camera images and microphone audio. To obtain this data clearly, the camera recognizes the customer's face and the microphone removes background noise. The output is the customer's facial image data and voice data.

[1124] Step 2:

[1125] The device sends the collected face data and voice data to the server using a secure communication protocol (e.g., HTTPS). The input is the face image data and voice data collected in step 1. The data is encrypted and sent to the server over the network. The output is the face image data and voice data that arrives at the server.

[1126] Step 3:

[1127] The server preprocesses the received facial image data using an image processing library (e.g., OpenCV). The input is the facial image data received by the server. Facial feature points (e.g., the positions of the eyes, nose, and mouth) are extracted through image processing. The output is the extracted facial feature points.

[1128] Step 4:

[1129] The server preprocesses the incoming audio data using an audio processing library (e.g., SciPy). The input is the audio data arriving at the server. The audio data is Fourier transformed and the frequency spectrum and features (e.g., pitch, volume, and tone) are analyzed. The output is the analyzed audio features.

[1130] Step 5:

[1131] The server analyzes emotions from the preprocessed facial data using a convolutional neural network (CNN) model. The input is the facial feature points extracted in step 3. The CNN model determines emotions such as "happiness," "anger," and "sadness" with high accuracy from these feature points. The output is the analyzed emotion data.

[1132] Step 6:

[1133] The server analyzes the preprocessed speech data for emotion using a recurrent neural network (RNN) model. The input is the speech features analyzed in step 4. The RNN model determines the emotion from the speech data. The output is the analyzed emotion data.

[1134] Step 7:

[1135] The server uses the feedback generation means to propose optimal service improvement measures based on the emotional state of the face data and voice data obtained from the emotion analysis means. The input is the emotional data obtained in steps 5 and 6. The algorithm generates specific service improvement proposals. The output is the proposed service improvement measures.

[1136] Step 8:

[1137] The server uses an alerting means to notify staff or administrators when a specific emotion is recognized frequently. The input is emotion data from the emotion analysis means. An alert is generated based on specific conditions and a notification is sent. The output is an alert notification to staff or administrators.

[1138] Step 9:

[1139] The server stores the emotion data in a database and performs comparative analysis with past data using data analysis means. The input is the emotion data obtained in steps 7 and 8. This allows analysis of customer reactions to specific time periods and menu items, and visualizes trends. The output is the analysis results and data stored in the database.

[1140] Step 10:

[1141] The server periodically generates an aggregate report of the emotion data and delivers it to the administrator. The input is the emotion data saved in step 9. The aggregate report includes weekly and monthly emotion trends, customer satisfaction trends, improvement suggestions, etc. The output is the emotion data report delivered to the administrator.

[1142] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1143] The present invention is a system for understanding customer emotions in real time at restaurants, helping to improve service and feedback accuracy. This system is composed of a data collection means, a facial expression recognition means, a voice analysis means, an emotion analysis means, a feedback generation means, a transmission means, a data storage means, and an emotion engine.

[1144] Collection of customer data

[1145] Device: Cameras installed in the store collect customer facial data in real time, and microphones record customer conversations. The cameras capture high-resolution footage, and the microphones are equipped with background noise reduction.

[1146] Sending data

[1147] On the device: The collected face and voice data is sent to the server in batches (e.g., every 10 seconds). The data is encrypted using a secure communication protocol (e.g., HTTPS).

[1148] Data preprocessing and analysis

[1149] Server: When the face and audio data arrive at the server, they are first preprocessed. For face data, an image processing library (e.g., OpenCV) is used to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). For audio data, a Fourier transform is performed using an audio processing library (e.g., SciPy) to analyze the frequency spectrum and features (e.g., pitch, volume, and tone).

[1150] Emotion assessment and integration

[1151] Server: The pre-processed data is first processed using facial expression recognition means to determine customer emotions using a deep learning model (e.g., CNN). Similarly, audio analysis means is used to determine emotions from audio data using a machine learning model (e.g., RNN). These data are then integrated by the emotion engine to determine the overall customer emotional state.

[1152] Data storage and analysis

[1153] Server: The determined emotion data is stored in a database and compared with past data for analysis. Here, customer reactions to specific times of day or menu items are analyzed, and trends and patterns are visualized.

[1154] Generate feedback

[1155] Server: Based on the sentiment data, the feedback generator automatically suggests optimal service improvement measures, such as "increase staffing during certain times" or "add explanations for certain menu items."

[1156] Alerts and Reports

[1157] Server: Based on the results of the emotion data analysis, real-time alerts are sent to administrators when certain conditions are met. Aggregated reports of emotion data are also periodically created and distributed to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement.

[1158] Specific examples

[1159] Device: The in-store camera captures the customer's smile, and the microphone simultaneously records the customer's positive conversation. These data are sent to the server, and feature points of the video and audio are extracted.

[1160] Server: Using CNN and RNN models, the emotion of "happiness" is determined from smiles and positive vocal tones. The recognized emotion data is stored in a database, and analysis derives trends such as "certain lunchtime menu items are popular."

[1161] Server: Feedback is generated suggesting that "serve this menu item at dinner time as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[1162] In this way, the system can grasp customer sentiment in real time and provide accurate feedback and solutions to improve the dining experience.

[1163] The processing flow will be explained below.

[1164] Step 1:

[1165] Terminal: A camera installed in the store captures customer facial data in real time. Specifically, the camera takes a certain number of frames per second (e.g., 30 frames per second) and detects facial regions using a face detection algorithm (e.g., Haar cascade). The detected facial regions are segmented and passed to subsequent processing steps.

[1166] Step 2:

[1167] Device: A microphone installed in the store collects customer voice data. Specifically, the microphone records the voice at a certain sample rate (e.g., 44.1 kHz), and the voice data is processed using noise cancellation and then saved in an audio file format (e.g., WAV file).

[1168] Step 3:

[1169] Device: Sends face and voice data to the server in batches of a certain size (e.g., every 10 seconds). Specifically, when the data reaches a certain size, it encrypts the data using a secure communication protocol (e.g., HTTPS) and sends it to the server.

[1170] Step 4:

[1171] Server: Preprocesses the face data. Specifically, it uses an image processing library (e.g., OpenCV) to extract facial feature points (e.g., the positions of the eyes, nose, and mouth). This process converts the data into a format that can be used by the deep learning model. The extracted feature points are also normalized and passed on to the next step.

[1172] Step 5:

[1173] Server: Preprocesses the audio data. Specifically, it uses an audio processing library (e.g., SciPy) to perform a Fourier transform on the audio data and analyze the frequency spectrum. It also extracts features such as pitch, tone, and speed, and converts them into input data for emotion analysis.

[1174] Step 6:

[1175] Server: Using the preprocessed facial feature data, a deep learning model (e.g., CNN) is used to analyze customer emotions. Specifically, the facial feature data is input into the model to determine emotions such as "happiness," "anger," and "sadness."

[1176] Step 7:

[1177] Server: Using the preprocessed voice feature data, a machine learning model (e.g., RNN) is used to analyze customer emotions. Specifically, the voice feature data is input into the model to determine emotions such as "excitement" or "calmness."

[1178] Step 8:

[1179] Server: The emotion data obtained from facial expression recognition and voice analysis is integrated into the emotion engine to determine the overall emotional state of the customer. Specifically, the emotion engine integrates the facial expression data and voice data to calculate a single integrated emotion score. This information is stored in a database.

[1180] Step 9:

[1181] Server: Analyzes the integrated sentiment data and extracts trends and patterns. For example, analyzes customer reactions to specific times or menu items and generates statistical information. Data analysis tools (e.g., Pandas, NumPy) are used for the analysis.

[1182] Step 10:

[1183] Server: Generates feedback based on emotion data. Specifically, the feedback generator automatically creates specific improvement suggestions, such as "increase staff during this time period" or "add explanations for specific menu items."

[1184] Step 11:

[1185] Server: Sends real-time alerts to administrators when certain conditions are met. For example, if customer satisfaction is low or if there is a high level of dissatisfaction with a particular menu item, an alert will be sent to the administrator via email or SMS.

[1186] Step 12:

[1187] Server: Periodically creates aggregated reports of emotion data and delivers them to administrators. The reports include emotion trends, changes in satisfaction levels, and suggestions for improvement. The reports are generated in PDF format and sent via email.

[1188] Example 2

[1189] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] Restaurants are required to understand the emotional state of customers in real time and use that information to improve and optimize the quality of their services. However, conventional systems collect, analyze, store, and generate feedback from facial and voice data separately, making efficient and integrated operation difficult.

[1191] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image capture means for collecting customer facial data, an audio collection means for collecting customer voice data, a transmission means for transmitting the collected facial data, a transmission means for transmitting the collected voice data, an image processing means for preprocessing the collected facial data, an audio processing means for preprocessing the collected voice data, an expression analysis means for recognizing the customer's facial expression from the preprocessed facial data, an audio analysis means for analyzing the customer's tone of voice from the preprocessed audio data, an emotion analysis means for integrating data obtained from the expression analysis means and the audio analysis means to determine the customer's emotional state, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to efficiently integrate the customer's emotional state in real time and quickly suggest optimal service improvement measures.

[1192] "Image Capture Device" means a device used to collect customer facial data in real time.

[1193] An "audio collection means" is a device used to collect customer audio data.

[1194] "Transmission means" refers to a communication device or protocol for transmitting collected data to a server.

[1195] "Image processing means" refers to software or hardware for preprocessing collected face data and extracting feature points.

[1196] The "acoustic processing means" refers to software or hardware for preprocessing collected audio data and analyzing its features.

[1197] "Facial expression analysis means" refers to algorithms or models used to recognize customer emotions from pre-processed facial data.

[1198] "Voice analysis means" refers to algorithms or models for analyzing the customer's tone of voice from pre-processed voice data.

[1199] The "emotion analysis means" is a system for integrating data obtained from the facial expression analysis means and the voice analysis means to determine the emotional state of the customer.

[1200] The "feedback generation means" is a mechanism for suggesting service improvements based on the determined emotional state.

[1201] The present invention is a system for understanding customer emotions in real time at restaurants and optimizing and improving service quality based on that information. This system is composed of an image capture means, an audio collection means, a transmission means, an image processing means, an audio processing means, a facial expression analysis means, a voice analysis means, an emotion analysis means, and a feedback generation means.

[1202] First, cameras and microphones installed in the store function as image capture and audio collection devices. The high-resolution cameras collect customer facial data in real time, while the microphones record customer conversations. The cameras capture details from the moment customers enter the store to the moment they smile, while the microphones record the conversations while filtering out surrounding noise.

[1203] This data is then sent from the device to the server in batches (e.g., every 10 seconds) using a secure communication protocol (e.g., HTTPS) to encrypt and transmit the data safely. This process reduces the risk of data eavesdropping or tampering.

[1204] The data that arrives at the server is first preprocessed using image processing and audio processing means. Specifically, an image processing library (e.g., OpenCV) is used to extract facial feature points (the positions of the eyes, nose, and mouth, etc.), and an audio processing library (e.g., SciPy) is used to perform a Fourier transform on the audio data, analyzing the frequency spectrum and features (pitch, volume, tone, etc.).

[1205] The preprocessed data is then subjected to emotion recognition using facial expression analysis means within the server. This analysis uses a deep learning model (e.g., CNN) to determine the customer's emotion from the image data. Similarly, emotion is determined from the voice data using a machine learning model (e.g., RNN) through the voice analysis means. The results of both analyses are integrated by the emotion analysis means to comprehensively determine the customer's final emotional state.

[1206] The emotional state determined is stored in a database and compared with past data using analytical tools. This analysis visualizes customer reactions to specific time periods and menu items, highlighting trends and patterns. For example, a trend such as "a particular pasta menu item is popular during lunchtime" can be analyzed.

[1207] Finally, the feedback generation means automatically proposes service improvement measures based on the emotional data. For example, a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well" may be automatically generated. Furthermore, when certain conditions are met, an alert is sent to the manager in real time, and a summary report of the emotional data is periodically created and distributed to the manager.

[1208] As a concrete example, suppose an in-store camera captures a customer's smile, while a microphone simultaneously records a customer's positive conversation. This data is sent to a server, where feature points from the video and audio are extracted. Using CNN and RNN models, the smile and positive tone of voice are used to determine the emotion of "happiness." The recognized emotion data is then stored in a database, and analysis derives a trend, such as "a particular lunchtime menu item is popular." As feedback, a suggestion is generated to "serve this menu item at dinnertime as well." Based on this trend, a report is sent to the manager stating that "customer satisfaction is high at lunchtime."

[1209] The system allows for real-time understanding of customer sentiment and provides targeted feedback and improvements to improve the dining experience.

[1210] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1211] Step 1:

[1212] Terminal: A camera installed in the store captures customers' faces in real time. Specifically, the camera captures high-resolution video and outputs each frame as image data. The input is a video of customers in the store, and the output is image data containing the customer's face.

[1213] Step 2:

[1214] Terminal: A microphone records customer conversations in real time. Noise-canceling technology is used to remove background noise and obtain clean audio data. The input is the in-store audio environment, and the output is audio data with the customer's conversations emphasized.

[1215] Step 3:

[1216] Terminal: The collected face data and audio data are batch processed (e.g., every 10 seconds) and sent to the server. When sent, the data is encrypted using a secure communication protocol (e.g., HTTPS). The input is image data and audio data, and the output is the encrypted transmission data.

[1217] Step 4:

[1218] Server: Receives the transmitted face data and performs preprocessing using an image processing library (e.g., OpenCV). Specifically, the processing involves extracting facial feature points (e.g., the positions of the eyes, nose, and mouth). The input is encrypted image data, and the output is preprocessed image data with feature points extracted.

[1219] Step 5:

[1220] Server: Receives the transmitted audio data and performs preprocessing using an audio processing library (e.g., SciPy). Performs a Fourier transform on the audio data and analyzes the frequency spectrum and features (e.g., pitch, volume, tone). The input is encrypted audio data, and the output is preprocessed audio data with the frequency spectrum and features analyzed.

[1221] Step 6:

[1222] Server: Using the preprocessed image data, a deep learning model (e.g., CNN) is run to analyze facial expressions and determine the customer's emotions. Specifically, the model outputs emotion labels such as "joy," "anger," and "sadness" from the input image data. The input is the preprocessed image data, and the output is the emotion label.

[1223] Step 7:

[1224] Server: Using the preprocessed voice data, a machine learning model (e.g., RNN) is run to analyze the voice and determine the emotion from the customer's tone of voice. Specifically, the model outputs an emotion label such as "satisfied" or "dissatisfied" from the input voice data. The input is the preprocessed voice data, and the output is the emotion label.

[1225] Step 8:

[1226] Server: The emotion analysis means integrates the emotion labels obtained from the facial expression analysis means and the voice analysis means to determine the overall emotional state of the customer. For example, if the facial expression of "happiness" matches the tone of voice of "satisfied," the overall judgment is "very satisfied." The input is the emotion labels of facial expressions and voice, and the output is the integrated emotional state.

[1227] Step 9:

[1228] Server: Stores the determined emotional state in a database. It then performs comparative analysis with past data, analyzes customer reactions to specific time periods and menu items, and visualizes trends and patterns. For example, it analyzes the trend that "a specific lunchtime menu item is popular." The input is the integrated emotional state data, and the output is the visualized results of trends and patterns.

[1229] Step 10:

[1230] Server: The feedback generation means automatically proposes optimal service improvement measures based on the emotion data. For example, it generates a specific service improvement measure such as "offering popular lunchtime menu items at dinnertime as well." The input is the analysis results and emotion data, and the output is a service improvement proposal.

[1231] Step 11:

[1232] Server: When a specific condition is met (for example, when customer satisfaction declines), it sends an alert to the administrator in real time. It also periodically creates aggregate reports of emotion data and delivers them to the administrator. The input is emotion data that meets the condition, and the output is the alert notification and aggregate reports.

[1233] (Application example 2)

[1234] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] Traditional customer service in restaurants relies heavily on staff experience and intuition, making it difficult to accurately grasp customer emotions and satisfaction. Furthermore, there are limited means to grasp customers' real-time emotional states and provide appropriate feedback based on them, resulting in a lack of speed in service improvement. Therefore, in order to improve customer satisfaction and service quality, real-time emotion analysis and immediate feedback based on the results are required.

[1236] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a data collection means for collecting customer facial data, a data collection means for collecting customer voice data, a facial expression recognition means for recognizing the customer's facial expression from the collected facial data, a voice analysis means for analyzing the customer's tone of voice from the collected voice data, an emotion analysis means for determining the customer's emotional state based on the data obtained by the facial expression recognition means and the voice analysis means, a display means for displaying the determined emotional state in real time, a feedback presentation means for providing feedback to the staff based on the emotional state displayed in real time, and a feedback generation means for suggesting service improvement measures based on the determined emotional state. This makes it possible to accurately grasp the customer's emotional state in real time and provide immediate feedback based on that.

[1237] The "data collection means" is a mechanism for collecting face data and voice data of customers.

[1238] The "facial expression recognition means" is a mechanism that analyzes the facial expression of the customer from the collected facial data and recognizes that expression.

[1239] The "voice analysis means" is a mechanism that analyzes collected voice data to analyze the customer's voice tone and other characteristics.

[1240] The "emotion analysis means" is a mechanism that determines the emotional state of the customer using data obtained from the facial expression recognition means and the voice analysis means.

[1241] The "display means" is a mechanism for displaying the determined emotional state in real time.

[1242] A "feedback presentation means" is a mechanism that provides feedback to staff based on their emotional state displayed in real time.

[1243] A "feedback generator" is a mechanism that automatically suggests service improvements based on the determined emotional state.

[1244] The "transmission means" is a mechanism for transmitting the collected face data and voice data to a server using a secure communication protocol.

[1245] The "data storage means" is a mechanism used to store the emotional states obtained from the facial expression recognition means and the voice analysis means.

[1246] The "analysis means" is a mechanism for comparing and analyzing stored data with past data.

[1247] System configuration

[1248] The present invention is a system that analyzes customer sentiment in real time and provides feedback to staff. This system is broadly composed of data collection means, data analysis means, feedback means, transmission means, data storage means, and analysis means.

[1249] Hardware and Software Use

[1250] The present invention is implemented using the following hardware and software:

[1251] Hardware: smart glasses, camera, microphone

[1252] Software: OpenCV (face recognition), TensorFlow (sentiment analysis), SciPy (voice analysis), HTTP request library (data transmission)

[1253] Data collection methods

[1254] The device collects customer facial and voice data in real time using the smart glasses' built-in camera and microphone. The camera captures high-resolution footage, and the microphone has the ability to filter out background noise.

[1255] Data Analysis Methods

[1256] To analyze the collected data, the server first extracts facial feature points using an image processing library (e.g., OpenCV) for face data, performs a Fourier transform on the voice data using a voice processing library (e.g., SciPy) to analyze the frequency spectrum and features, and then uses a deep learning model (e.g., TensorFlow) to determine the customer's emotional state.

[1257] Feedback Methods

[1258] The server then displays the determined emotional state on the staff member's smart glasses in real time, allowing the staff member to receive real-time feedback to provide optimal service based on the customer's emotions. Feedback is provided using audio guidance and visual alerts.

[1259] Means of transmission and data storage

[1260] The server sends the collected facial and voice data to a central server using a secure communication protocol (e.g., HTTPS), where it is stored in a database and used for analysis.

[1261] Data analysis and feedback generation

[1262] The emotion data stored in the data storage means is periodically analyzed by the analysis means. Here, customer responses to specific time periods and services are compared and analyzed, and trends and patterns are visualized. The feedback generation means then proposes optimal service improvement measures based on the analysis results.

[1263] Specific examples

[1264] For example, suppose a staff member wearing smart glasses captures a customer's smile while serving them. The system analyzes the video and recognizes the emotion of "happiness." It then displays feedback to the staff member's smart glasses, saying, "This customer is enjoying themselves. Please consider providing additional services." This feedback allows the staff member to immediately provide the optimal service to the customer.

[1265] Prompt Sentence Examples

[1266] "The following Python program implements a system that analyzes customer emotions in real time. The system uses OpenCV for facial recognition from camera footage and TensorFlow for emotion recognition. Audio data is analyzed using SciPy, and the data is sent to a server using HTTPS. Please complete the following code to enhance the emotion recognition functionality."

[1267] In this way, the present invention is a system that increases customer satisfaction by grasping customer emotions in real time and providing immediate feedback based on the emotions.

[1268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1269] Step 1:

[1270] The device uses a camera and microphone to collect customer facial and audio data. The camera captures high-resolution video, and the microphone filters out background noise. The input is real-time facial video and audio, and the output is raw video and audio data.

[1271] Step 2:

[1272] The server receives the video data sent from the device and extracts facial feature points using an image processing library (e.g., OpenCV). The input is raw video data, and the output is processed image data containing facial feature points (e.g., the positions of the eyes, nose, and mouth).

[1273] Step 3:

[1274] The server simultaneously receives the audio data sent from the device and performs a Fourier transform using an audio processing library (e.g., SciPy). This analyzes the frequency spectrum and features (e.g., pitch, volume, and tone) of the audio data. The input is raw audio data, and the output is audio data with extracted features.

[1275] Step 4:

[1276] The server inputs the processed image data into a deep learning model (e.g., TensorFlow) and analyzes the customer's facial expression using facial expression recognition. The input is image data containing facial feature points, and the output is the analyzed facial expression (e.g., "happiness," "sadness," etc.).

[1277] Step 5:

[1278] The server inputs the feature-extracted voice data into a machine learning model (e.g., RNN) and analyzes the customer's voice tone using a voice analysis method. The input is the feature-extracted voice data, and the output is the analyzed emotional state (e.g., "joy," "anger," etc.).

[1279] Step 6:

[1280] The server integrates the data obtained from the facial expression recognition means and the voice analysis means and determines the overall emotional state of the customer using the emotion analysis means, where the input is the emotion result from the facial expression data and the voice data, and the output is the integrated emotional state.

[1281] Step 7:

[1282] The server displays the determined emotional state in real time on the staff member's smart glasses using a display means, where the input is the integrated emotional state and the output is feedback information that the staff member can see through the glasses.

[1283] Step 8:

[1284] The server provides feedback to staff using a feedback presentation tool based on the emotional state displayed in real time. The input is the displayed emotional state, and the output is a specific action or suggestion (e.g., "The customer is enjoying themselves. Please consider additional services.").

[1285] Step 9:

[1286] The server transmits the collected face and audio data to a central server using a secure communication protocol (e.g., HTTPS). The input is raw video and audio data, and the output is encrypted data communication.

[1287] Step 10:

[1288] The server stores the transmitted data in a database and compares it with past data. The input is the stored emotion data, and the output is the analysis results and visualization of trends and patterns.

[1289] Step 11:

[1290] The server uses a feedback generation means to propose optimal service improvement measures based on the analysis results. The input is the analysis results, and the output is automatically generated service improvement proposals.

[1291] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1292] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1293] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1294] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1295] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1296] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1297] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1298] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1299] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1300] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1301] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1302] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1303] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1304] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1305] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1306] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1307] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1308] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1309] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1310] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1311] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1312] The following is further disclosed regarding the above embodiment.

[1313] (Claim 1)

[1314] A data collection means for collecting customer face data;

[1315] a data collection means for collecting customer voice data;

[1316] facial expression recognition means for recognizing customer facial expressions from the collected facial data;

[1317] a voice analysis means for analyzing the tone of the customer's voice from the collected voice data;

[1318] emotion analysis means for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means;

[1319] a feedback generating means for suggesting service improvements based on the determined emotional state;

[1320] A system including:

[1321] (Claim 2)

[1322] 10. The system of claim 1, further comprising a transmitting means for transmitting the collected face and voice data using a secure communication protocol.

[1323] (Claim 3)

[1324] 2. The system according to claim 1, further comprising data storage means and analysis means for storing the emotional state obtained from the facial expression recognition means and the voice analysis means in a database and for comparing and analyzing it with past data.

[1325] "Example 1"

[1326] (Claim 1)

[1327] a means for collecting customer facial data;

[1328] a means for collecting customer voice data;

[1329] means for recognizing customer facial expressions from the collected facial data;

[1330] means for analyzing the customer's tone of voice from the collected voice data;

[1331] a means for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means;

[1332] means for suggesting service improvements based on the determined emotional state;

[1333] A means for sending an alert to an administrator when customer sentiment data meets certain conditions;

[1334] A method for periodically creating aggregate reports of emotion data and distributing them to administrators;

[1335] A system including:

[1336] (Claim 2)

[1337] 10. The system of claim 1, further comprising means for transmitting the collected facial and audio data using a secure communication protocol.

[1338] (Claim 3)

[1339] 2. The system according to claim 1, further comprising means for storing the emotional state obtained from the facial expression recognition means and the voice analysis means in a database and for comparing and analyzing it with past data.

[1340] "Application Example 1"

[1341] (Claim 1)

[1342] A data collection means for collecting customer face data;

[1343] a data collection means for collecting customer voice data;

[1344] facial expression recognition means for recognizing customer facial expressions from the collected facial data;

[1345] a voice analysis means for analyzing the tone of the customer's voice from the collected voice data;

[1346] emotion analysis means for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means;

[1347] a feedback generating means for suggesting service improvements based on the determined emotional state;

[1348] An alerting method to notify staff and managers when a specific emotion is recognized frequently.

[1349] a data analysis means for analyzing past customer data and generating periodic reports;

[1350] A system including:

[1351] (Claim 2)

[1352] 10. The system of claim 1, further comprising a transmitting means for transmitting the collected face and voice data using a secure communication protocol.

[1353] (Claim 3)

[1354] 2. The system according to claim 1, further comprising data storage means and analysis means for storing the emotional state obtained from the facial expression recognition means and the voice analysis means in a database and for comparing and analyzing it with past data.

[1355] "Example 2: Combining Emotion Engines"

[1356] (Claim 1)

[1357] image capture means for collecting customer facial data;

[1358] an audio collection means for collecting audio data of the customer;

[1359] a transmitting means for transmitting the collected face data;

[1360] a transmitting means for transmitting the collected voice data;

[1361] image processing means for pre-processing the collected face data;

[1362] an acoustic processing means for pre-processing the collected audio data;

[1363] facial expression analysis means for recognizing a customer's facial expression from the preprocessed facial data;

[1364] a voice analysis means for analyzing the tone of the customer's voice from the preprocessed voice data;

[1365] emotion analysis means for integrating data obtained from the facial expression analysis means and the voice analysis means to determine the emotional state of the customer;

[1366] a feedback generating means for suggesting service improvements based on the determined emotional state;

[1367] A system including:

[1368] (Claim 2)

[1369] 10. The system of claim 1, further comprising a transmitting means for transmitting the collected face and voice data using a secure communication protocol.

[1370] (Claim 3)

[1371] 2. The system according to claim 1, further comprising data storage means and analysis means for storing the emotional states obtained from the facial expression analysis means and the voice analysis means in a database and for comparing and analyzing them with past data.

[1372] "Application example 2 when combining emotion engines"

[1373] (Claim 1)

[1374] A data collection means for collecting customer face data;

[1375] a data collection means for collecting customer voice data;

[1376] facial expression recognition means for recognizing customer facial expressions from the collected facial data;

[1377] a voice analysis means for analyzing the tone of the customer's voice from the collected voice data;

[1378] emotion analysis means for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means;

[1379] display means for displaying the determined emotional state in real time;

[1380] a feedback presentation means for providing feedback to the staff member based on the real-time displayed emotional state;

[1381] a feedback generating means for suggesting service improvements based on the determined emotional state;

[1382] A system including:

[1383] (Claim 2)

[1384] 10. The system of claim 1, further comprising a transmitting means for transmitting the collected face and voice data using a secure communication protocol.

[1385] (Claim 3)

[1386] 2. The system according to claim 1, further comprising data storage means and analysis means for storing the emotional state obtained from the facial expression recognition means and the voice analysis means in a database and for comparing and analyzing it with past data. [Explanation of symbols]

[1387] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A data collection means for collecting customer face data; a data collection means for collecting customer voice data; facial expression recognition means for recognizing customer facial expressions from the collected facial data; a voice analysis means for analyzing the tone of the customer's voice from the collected voice data; emotion analysis means for determining the emotional state of a customer based on data obtained by the facial expression recognition means and the voice analysis means; a feedback generating means for suggesting service improvements based on the determined emotional state; A system including:

2. 10. The system of claim 1, further comprising a transmitting means for transmitting the collected face and voice data using a secure communication protocol.

3. 2. The system according to claim 1, further comprising data storage means and analysis means for storing the emotional state obtained from the facial expression recognition means and the voice analysis means in a database and for comparing and analyzing it with past data.

Images