system

Abstract

We provide the system. [Solution] A means of obtaining image data to predict customer dwell time, An image recognition device means that predicts the customer's dwell time based on acquired image data, A means for obtaining the customer's location information and calculating the start time of travel, Means for notifying the customer of the calculated departure time and estimated waiting time, A means for optimizing seating arrangements based on predicted dwell time, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that the customer satisfaction decreases and business opportunities are lost due to the long waiting queue of customers in a restaurant. In particular, when customers have to wait for a long time in the store, their dissatisfaction is abundant, which will reduce their willingness to use it next time. Also, there is a problem that it is difficult to maximize sales due to the slow rotation of seats.

Means for Solving the Problems

[0005] This invention solves the above problems by providing a system that predicts customer dwell time and enables highly accurate prediction using an image recognition device. This system predicts dwell time based on acquired image data, obtains customer location information, and calculates the start time of movement. It also notifies the customer of the information obtained and optimizes waiting times. Furthermore, it improves prediction accuracy through reinforcement learning using past data and optimizes seating arrangements. Through these means, it is possible to improve customer satisfaction, achieve efficient seating management, and maximize sales.

[0006] "Image data" refers to video information obtained using in-store cameras, which is used to predict customer dwell time.

[0007] An "image recognition device" is a computer system or program that analyzes acquired image data to predict the customer's dwell time.

[0008] "Customer location information" refers to data used to identify where a customer is currently located, such as GPS information used to calculate the start time of travel.

[0009] "Departure start time" refers to the optimal time for customers to head to the restaurant, calculated to minimize expected waiting times.

[0010] "Optimizing seating arrangement" is the process of calculating the best seating arrangement in a restaurant to maximize seating use, reduce customer waiting times, and increase sales.

[0011] Reinforcement learning is a method that improves system performance by repeatedly using past data to increase the accuracy of predictions. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, a storage with a reference numeral is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, a communication I / F (Interface) with a reference numeral is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system that minimizes customer waiting times in restaurants and enables efficient seating management. At its core, it utilizes image recognition technology and customer location information to provide customers with optimal information, thereby improving customer satisfaction.

[0034] 1. Server Role

[0035] The server acquires video data from cameras installed in the restaurant and uses image recognition technology to predict customer dwell times in real time. Furthermore, it improves prediction accuracy through reinforcement learning using past data. Based on these predictions, it calculates customer waiting times and determines the optimal seating arrangement.

[0036] 2. The role of the terminal

[0037] The terminal is a device that displays information provided by the server to restaurant staff. Through the terminal, staff can understand customer waiting times and predicted seat availability, allowing them to guide customers efficiently. This maximizes the seating occupancy rate in the restaurant.

[0038] 3. User Roles

[0039] Users receive real-time notifications from this system via their smartphones. Notifications of departure time and estimated waiting times allow users to spend their time productively elsewhere, and they are immediately seated upon arrival.

[0040] Specific example

[0041] For example, consider a scenario where a family visits a restaurant. Minimizing waiting times is especially important when children are present. The server identifies the family as customers and, based on their estimated stay time at their table, notifies the user of suggestions for other places to spend time. The system also calculates the optimal time for the customer to begin their journey, taking into account the travel time to the restaurant. Upon arrival, a smooth entry process is facilitated through a terminal, with staff guiding the customer. This allows customers to enjoy high-quality service without unnecessary waiting.

[0042] The introduction of this system is expected to improve the overall customer experience at the restaurant, enable more efficient store operations, and increase sales.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server acquires video data from in-store cameras and uses image recognition technology to analyze the time customers spend at each table in real time. It measures the time customers spend at each table and uses reinforcement learning based on past data to improve the accuracy of predictions.

[0046] Step 2:

[0047] The server uses information obtained through image recognition to predict the length of time customers currently seated will stay. It calculates the time when seats are expected to become vacant, calculates the optimal seating arrangement based on the projected number of customers, and sends this information to the terminal.

[0048] Step 3:

[0049] Based on the information it receives, the terminal provides restaurant staff with specific instructions on which tables will become available and which customers should be seated at which tables. This allows staff to efficiently seat customers and optimize table turnover.

[0050] Step 4:

[0051] The server obtains the customer's location information and uses it to calculate the optimal time for the customer to begin their journey. The calculation result is then sent from the server to the user's smartphone to reduce waiting times.

[0052] Step 5:

[0053] Users receive notifications on their smartphones and prepare to head to the restaurant based on the suggested departure time. They can also check detailed information such as waiting times and estimated arrival times on their smartphones.

[0054] Step 6:

[0055] When the server detects that a user has started moving, it uses that information to update the wait time for greater accuracy. After arriving at the restaurant, the terminal prepares to smoothly guide the customer to the appropriate seat.

[0056] Step 7:

[0057] When a user arrives at the restaurant, the terminal instructs staff to guide them according to pre-calculated results, and the staff smoothly leads the customer to their designated seat. This allows customers to enjoy a seamless dining experience without experiencing unnecessary waiting times.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] In current store operations, it is difficult to quickly and accurately predict customer dwell times and achieve optimal seating arrangements. This can lead to unnecessary waiting times for customers, resulting in decreased customer satisfaction and insufficient store efficiency. Furthermore, the lack of mechanisms to learn from past data prevents improvements in prediction accuracy.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for acquiring image data using a sensing device for acquiring customer behavior data, an image analysis unit means for analyzing the acquired image data and predicting the customer's stay time, and means for acquiring the customer's geographical information and calculating the optimal start time for travel. This enables efficient seat allocation in line with the customer's arrival and notification of waiting times with high prediction accuracy.

[0063] A "sensing device" is a device that detects a customer's movements and location and acquires image data based on that information.

[0064] An "image analysis unit" is a system that processes acquired image data and performs analysis to predict customer movements and dwell time.

[0065] "Geographic information" refers to data that shows location information related to travel, such as a customer's current location or destination.

[0066] "Reinforcement learning" is a machine learning technique that uses historical data to repeatedly learn and make predictions about the future and optimize actions.

[0067] "Communication equipment" refers to devices used to transmit calculated information or notifications to customers, and includes smartphones and other mobile devices.

[0068] "Staff" refers to the personnel who guide customers within the store and are responsible for providing smooth service.

[0069] "Seat allocation" is the process of determining the optimal seating arrangement based on the predicted length of customer stay.

[0070] This invention relates to a system for minimizing customer waiting times in restaurants and efficiently managing seating. Specific embodiments are described below.

[0071] Server Role

[0072] The server uses image data acquired in real time from sensing devices within the restaurant. This includes cameras and other image sensors used to accurately track customer movements. The server analyzes the data using image processing software such as OpenCV to predict customer dwell time. A mechanism is in place to improve prediction accuracy by utilizing reinforcement learning algorithms and learning from past data. Furthermore, the server acquires customer geographical information and calculates the optimal time to start moving.

[0073] Terminal role

[0074] The terminal is a device that displays information provided by the server to restaurant staff. A special application is installed on the terminal so that staff can easily check customer waiting times and predicted seat availability. This allows staff to guide customers efficiently and optimize seating allocation within the restaurant.

[0075] User roles

[0076] Users receive notifications sent from the server via their smartphones. For example, information about the start time of travel and the estimated waiting time is sent to the user's smartphone as a push notification. This notification allows users to spend their time productively elsewhere and arrive at the restaurant at the optimal time.

[0077] Specific example

[0078] For example, when visiting a restaurant with family, the server notifies the user with suggestions for the best way to minimize waiting time and avoid crowds. This is especially beneficial for families with children. Upon arrival, staff guide the customer to their seats via a terminal to ensure a smooth entry.

[0079] Example of a prompt

[0080] Examples of prompts for the generating AI model include, "Please explain how to analyze image data to predict customer dwell time," and "Please describe an example of how the system can be used to optimize seating management in a restaurant."

[0081] The introduction of this system is expected to improve the overall operational efficiency of the restaurant, increase customer satisfaction, and boost sales.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server acquires image data in real time from sensing devices within the restaurant. Camera images are taken as input, and image analysis is performed based on this data. Specifically, frame extraction is performed to record customer movements and postures. The output is data related to customer movement patterns and dwell time.

[0085] Step 2:

[0086] The server analyzes the acquired image data using image processing software such as OpenCV. The input is the movement information acquired in step 1, and based on this, it predicts the customer's dwell time. Specifically, it recognizes the customer's time spent in the store and their behavior patterns, and refines the prediction using a reinforcement learning algorithm that utilizes past data. The output is the predicted dwell time data.

[0087] Step 3:

[0088] The server calculates the optimal seating arrangement based on predicted dwell time data. The input is the dwell time data obtained in step 2, and the optimization algorithm is executed by taking into account the current seating situation. Specifically, it considers seat type and customer needs to perform efficient seating assignment. The output is the assigned seating information.

[0089] Step 4:

[0090] The terminal displays seating arrangement information provided by the server to the staff. The input is the seating information generated in step 3, which is made intuitively viewable via a GUI. Specifically, an interface is provided that allows detailed information to be viewed via touch operation. The output is clear seating arrangement instructions that staff can use to guide customers.

[0091] Step 5:

[0092] The user receives notifications from the server on their smartphone. The input is the result of the dwell time prediction in step 2 and seat assignment in step 3, and notifications of departure time and waiting time are provided based on this. Specifically, the travel instructions are optimized based on the user's current location. The output is real-time travel guidance and waiting time information displayed on the user's device.

[0093] In this way, each step works in conjunction to create a system that manages restaurant seating and improves customer satisfaction.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In modern commercial facilities and restaurants, efficient queue management that minimizes customer waiting times is essential. However, traditional systems struggle to accurately predict waiting times, leading to decreased customer satisfaction. Furthermore, there is a lack of means to provide customers with the optimal timing for arriving at the store, resulting in a failure to offer customers a meaningful experience.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for acquiring electronic information to predict the customer's stay time, visual recognition device means for predicting the customer's stay time based on the acquired electronic information, and means for displaying the store's congestion status and the customer's place in line in real time on an information terminal. This allows customers to accurately understand their waiting time and efficiently plan their time in the store.

[0099] "Electronic information" refers to information that includes data necessary to identify customer behavior and characteristics.

[0100] A "visual recognition device" is a device that uses cameras and sensors to analyze customer characteristics from image and video data and predict their behavior.

[0101] An "information terminal" is a device that displays necessary information to the user and accepts their input, and includes smartphones and tablets.

[0102] "Duration of stay" refers to the time a customer spends in a store from the moment they enter until they leave.

[0103] "Crowding status" refers to information indicating the number and placement of customers within a store, and is used to understand the usage status of the facility.

[0104] "Queue management" is a method of properly managing the order in which customers enter and leave a store in order to minimize waiting times.

[0105] The system for implementing this invention aims to efficiently manage customer waiting times in restaurants and commercial facilities and improve customer satisfaction. The detailed configuration of the system is shown below.

[0106] The server first uses cameras and sensors to acquire electronic information about customer movements and dwell times within the facility. This information is analyzed by a visual recognition device, and an AI model is used to predict customer dwell times. Software used includes OpenCV for image analysis and TENSORFLOW® for machine learning models. MySQL® is used as the database software to store customer visit history and prediction results.

[0107] The device is a medium that receives data sent from the server. This device is a smartphone or tablet, acting as an information terminal, and displays customer dwell time predictions and current congestion status. The information is sent in real time via Firebase Cloud Messaging and displayed on the device in the form of a React Native application.

[0108] Based on the information obtained through this device, users can make efficient use of their time. Specifically, if there is a long wait, they can receive suggestions for spending time shopping or taking a walk nearby. Also, as the reserved time approaches, they will be notified of the optimal departure time, allowing them to enter the store at the specified time.

[0109] As a concrete example, this system can be used when dining out with family on weekends to determine the optimal arrival time in real time based on the restaurant's congestion, allowing the whole family to enjoy their meal without stress. Prompt messages such as, "Check the current waiting time and spend your time productively at home. You will be notified when your table is ready," are issued, providing customers with useful information.

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The server acquires electronic information about customers in real time from cameras and sensors installed within the facility. At this point, the input is camera footage, and the output is image data. The image data is pre-processed for use in visual recognition devices. Specifically, this includes noise reduction and resolution adjustment.

[0113] Step 2:

[0114] The server analyzes image data using a visual recognition device to predict customer dwell time. The input is pre-processed image data obtained in step 1. Using the generative AI model TensorFlow, data processing and pattern recognition are performed, and the predicted dwell time is output. This process includes facial recognition and motion analysis.

[0115] Step 3:

[0116] The server references historical customer data and improves the accuracy of dwell time predictions through reinforcement learning. In this step, historical visit history data and the prediction data from step 2 are used as input. The output is a new prediction model that reflects the improvement in prediction accuracy. This model is continuously learned and adjusted to minimize prediction errors.

[0117] Step 4:

[0118] The server obtains the customer's location information and calculates the optimal departure time. The input here is the current location information obtained from a smartphone or GPS device. The output is information notified to the customer in the form of a departure start time. An algorithm is used to perform the calculation, taking into account traffic conditions and distance.

[0119] Step 5:

[0120] The device receives data on estimated dwell time and departure time from the server and displays the information on the customer's smartphone. The input is notification information from the server, and the output is the waiting time and arrival time information displayed on the device. Push notifications are sent via Firebase Cloud Messaging.

[0121] Step 6:

[0122] Based on the information received, the user heads to the facility according to the optimal departure time. The input is the notification information displayed on the terminal, and the output is the user's actions. The user uses this information to spend their time productively and arrive at the facility at the appropriate time.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention aims to improve customer service by providing a system that combines an emotion engine to streamline restaurant waiting time management and seating guidance. The emotion engine analyzes changes in the customer's facial expressions and voice to understand their emotions, thereby enabling more personalized service.

[0125] 1. Server Role

[0126] The server uses cameras and an emotion engine to collect customer facial expression data and analyze their emotional state based on this data. The emotion engine combines facial expression analysis and voice analysis to understand how customers feel about their waiting time.

[0127] 2. The role of the terminal

[0128] The terminal provides instructions to restaurant staff based on emotional state and predicted length of stay information transmitted from the server. This allows staff to understand the customer's emotional state and provide optimal service. For example, if a customer's emotional state is determined to be negative, the system can prioritize seating them.

[0129] 3. User Roles

[0130] Users can receive not only the waiting time and estimated departure time provided by the server, but also a less stressful waiting experience based on their own emotions. Since notifications are adjusted according to their emotions, users can make appropriate decisions about when and what actions to take.

[0131] Specific example

[0132] For example, when customers visit a restaurant with their family, children often get bored while waiting. When the emotion engine detects a negative change in the child's mood, the server uses that information to send instructions to the staff via a terminal. The staff then quickly prepares a table and adjusts the schedule so that the family is seated sooner. As a result, customers can enjoy a smooth and comfortable dining experience, which enhances the restaurant's reputation.

[0133] The introduction of this system will lead to higher levels of customer satisfaction, more efficient store operations, and flexible responses based on data.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] The server collects video and audio data in real time from multiple cameras and microphones installed in the store. Using an emotion engine, it analyzes the customer's facial expressions and voice characteristics from this data to determine the customer's emotional state.

[0137] Step 2:

[0138] The server uses image recognition to predict customer dwell time. Combined with the determined emotional state, it dynamically adjusts the acceptable waiting time range and priority for each customer. Customers with positive emotions are given a normal waiting time, while those with negative emotions are given a shorter waiting time.

[0139] Step 3:

[0140] The terminal receives each customer's emotional state and predicted length of stay from the server and displays this information to restaurant staff as visual instructions. Based on this information, staff determine the order in which to seat customers and optimize the customer experience.

[0141] Step 4:

[0142] The server tracks the customer's location and, based on the calculated waiting time, notifies the user's smartphone of the estimated start time of their journey. The notification is delivered in an appropriate tone and timing based on the customer's emotional state.

[0143] Step 5:

[0144] Users receive notifications on their smartphones and prepare to head to the restaurant according to the displayed departure time. A more comfortable waiting environment is provided thanks to the emotionally responsive waiting time.

[0145] Step 6:

[0146] When a user arrives at the restaurant, the terminal uses pre-prepared information to instruct staff, ensuring they are promptly seated at the most suitable table. This enables smooth customer service that takes emotions into consideration.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] In modern restaurants, the decline in customer satisfaction due to long wait times is a significant problem. In particular, when customer emotional states are not considered, the overall quality of service deteriorates, impacting the restaurant's reputation. Traditional waiting time management systems lack the ability to analyze customer emotions from facial expressions and voice, making it difficult to provide more personalized service. Therefore, there is a need for effective waiting time management and seating guidance that takes customer emotions into account.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes means for providing an input device for collecting customer facial expressions and voice data, processing means for analyzing the collected facial expressions and voice data and inferring the customer's emotional state, and means for generating service instructions for the customer based on the analyzed emotional state and transmitting them to a staff terminal. This enables quick and effective waiting time management and seating guidance that takes into account the customer's emotional state.

[0152] "Customer facial expression and voice data" refers to information related to a customer's facial expressions and voice, and is used to infer the customer's emotional state by analyzing this data.

[0153] An "input device" is a device, such as a camera or microphone, installed in a restaurant to collect facial expressions and voice data from customers.

[0154] A "processing device" is a computer system that analyzes collected facial and voice data to infer the customer's emotional state.

[0155] "Emotional state" refers to the psychological and emotional condition exhibited by a customer, as inferred from their facial expressions and voice.

[0156] "Generating service instructions" means determining specific actions and services that restaurant staff should take based on the customer's emotional state, and creating instructions to display on a terminal.

[0157] A "staff terminal" is a device used by restaurant staff to receive and display information transmitted from the server.

[0158] A description of embodiments for carrying out this invention will be given.

[0159] The server collects customer facial expressions and audio data through cameras and microphones installed in the restaurant. Video data acquired from the cameras is processed in real time using an image processing library to extract facial features and infer the customer's emotional state. Audio data is transcribed using speech analysis software, and features such as voice tone and speaking speed are analyzed. Based on this, the server infers the customer's overall emotional state and stores it in a database.

[0160] The terminal receives customer emotional states transmitted from the server and displays appropriate service instructions on the screen for store staff. The terminal is notified of the optimal response for the customer based on the analysis results, enabling staff to quickly provide situation-appropriate service. For example, if a customer is a family and the children are bored during a long wait, the terminal will prompt staff to quickly seat them. This allows staff to effectively perform their duties to ensure customer comfort.

[0161] Users can receive notifications from the server via their smartphones regarding waiting times and estimated service times. The user application displays actionable guidelines based on the data provided by the server, enabling users to make the waiting experience less stressful. This allows users to use their time efficiently, such as returning to the restaurant at the appropriate time.

[0162] As a concrete example, an example of a prompt message is shown below.

[0163] "Please describe in detail how you analyze customers' emotions in real time while they are waiting and provide the best possible service. Include examples of how to handle situations where customers with children are bored."

[0164] As described above, this invention aims to improve the efficiency of restaurant operations and enhance customer satisfaction by providing personalized services through the analysis of customers' emotional states.

[0165] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0166] Step 1:

[0167] The server collects customer facial expressions and audio data via cameras and microphones. Camera footage and audio from within the store are acquired as input. Specifically, the server captures this data in real time, performs initial processing, and converts it into an analyzable format.

[0168] Step 2:

[0169] The server analyzes the customer's emotional state using the collected data. The input is the facial and audio data collected in step 1, and the output is the estimated result of the customer's emotional state. Specifically, the server uses an image processing library to detect facial feature points and audio processing software to analyze the tone of voice and comprehensively estimate the emotion.

[0170] Step 3:

[0171] The server sends the analyzed emotional state to the terminal and generates instructions for display to store staff. The input is the emotional state data obtained in step 2, and the output is specific service instructions for staff. Specifically, the server creates a service scenario that takes the emotional state into account and pushes it to the terminal.

[0172] Step 4:

[0173] The terminal displays instructions to staff based on the instructions it receives, supporting service delivery. The input is service instructions sent from the server, and the output is the action staff should take. Specifically, the terminal displays alerts on the screen to clearly indicate to staff which customers should be given priority.

[0174] Step 5:

[0175] Users receive notifications from the server via a smartphone app, obtaining information about waiting times and estimated service times. Input is waiting status data provided by the server, while output is recommended information regarding user actions. Specifically, the user's app screen displays notifications such as "You will be served in 5 minutes," allowing users to efficiently manage their time.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] Customers often experience dissatisfaction due to long waiting times at facilities. Furthermore, standard service often struggles to provide individualized care based on customer emotions, making it difficult to improve customer satisfaction. Especially for families and customers whose emotions are easily swayed, there is a need to reduce frustration and stress during waiting times and provide a comfortable experience. An efficient system is needed to improve this customer experience.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes means for analyzing customer emotions and providing personalized services based on that data, means for predicting customer dwell time and optimizing location placement based on the prediction results, and means for acquiring customer location information and providing navigation guidance at the optimal timing. This makes it possible to provide individualized services that take customer emotions into consideration and to optimize dwell time management.

[0181] "Data for predicting customer dwell time" refers to information about customer behavior and movements within the facility.

[0182] A "recognition device" is a device that analyzes acquired information and derives a specific result.

[0183] "Location information" refers to information that indicates where a customer is currently located.

[0184] "Movement start time" refers to the calculated time when the customer is scheduled to begin their next action.

[0185] "Waiting time" refers to the expected time it takes for a customer to receive a particular service.

[0186] "Placement arrangement" refers to the positioning of each area and seating area within a facility.

[0187] "Analyzing customer emotions" means inferring a customer's psychological state and feelings based on their facial expressions and voice.

[0188] "Personalized service" refers to the provision of services that are customized to the individual needs and circumstances of each customer.

[0189] This system is designed to improve customer service in restaurants and other establishments, and primarily consists of three elements: servers, terminals, and users.

[0190] The server uses cameras and microphones installed within the facility to collect customer facial expressions and audio data. This data is processed by emotion analysis AI (a specific example being Microsoft® Azure® Emotion API) to analyze the customer's emotional state in real time. This allows the server to understand what emotions the customer is experiencing.

[0191] Subsequently, the terminal receives emotion data from the server and wait time information calculated based on the customer's predicted stay duration. The terminal then uses this information to send instructions to facility staff. For example, if a customer is deemed unhappy, it can be instructed to prioritize seating them. The terminal also has the function of optimizing guest flow and providing personalized entertainment and information during waiting times.

[0192] Users can receive information from their devices, reduce their own stress, and choose the optimal course of action. These notifications are tailored to the user's emotions, allowing for more flexible responses.

[0193] As a concrete example, when visiting a restaurant with family, robots are used to make waiting time more enjoyable by displaying animations on smartphones to keep children entertained. Another example of a prompt message is, "Please think of actions for a robot that analyzes the customer's voice and facial expressions and proposes the most appropriate service based on those emotions."

[0194] Thus, this invention aims to enhance the waiting experience within a facility and increase customer satisfaction by utilizing emotion analysis technology to provide personalized services to customers.

[0195] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0196] Step 1:

[0197] The server acquires real-time data from cameras and microphones installed within the facility. This data includes customer facial expressions and voice. Image and audio data are provided as input, forming the basis for subsequent processing.

[0198] Step 2:

[0199] The server uses the acquired data to analyze customer emotions using an emotion analysis AI model (e.g., Microsoft Azure Emotion API). This process extracts facial and vocal characteristics and outputs emotional states such as joy and anger. This output is a different emotion label for each customer.

[0200] Step 3:

[0201] The server sends the predicted dwell time and customer emotional state to the terminal based on the emotion analysis results. Here, the predicted emotional state and dwell time are input, and appropriate action instructions are output to each terminal accordingly.

[0202] Step 4:

[0203] The terminal uses emotional state and dwell time information received from the server to issue instructions to staff. These instructions may include, for example, promptly seating an unhappy customer or providing entertainment. Here, the input is emotional and time information, and the output is specific action instructions.

[0204] Step 5:

[0205] Based on information provided by the device, users adjust their travel timing and how they spend their waiting time. In this step, information that responds to emotions as input leads to the output of optimized customer behavior.

[0206] Step 6:

[0207] The server collects and stores data on each customer's emotional state and dwell time, and uses this data for reinforcement learning of the model. This aims to improve the accuracy of predictions and the quality of service. The input is the customer data history, and the output is the improved accuracy of emotion prediction.

[0208] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0209] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0210] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0211] [Second Embodiment]

[0212] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0213] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0214] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0215] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0216] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0217] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0218] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0219] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0220] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0221] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0222] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0223] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0224] This invention is a system that minimizes customer waiting times in restaurants and enables efficient seating management. At its core, it utilizes image recognition technology and customer location information to provide customers with optimal information, thereby improving customer satisfaction.

[0225] 1. Server Role

[0226] The server acquires video data from cameras installed in the restaurant and uses image recognition technology to predict customer dwell times in real time. Furthermore, it improves prediction accuracy through reinforcement learning using past data. Based on these predictions, it calculates customer waiting times and determines the optimal seating arrangement.

[0227] 2. The role of the terminal

[0228] The terminal is a device that displays information provided by the server to restaurant staff. Through the terminal, staff can understand customer waiting times and predicted seat availability, allowing them to guide customers efficiently. This maximizes the seating occupancy rate in the restaurant.

[0229] 3. User Roles

[0230] Users receive real-time notifications from this system via their smartphones. Notifications of departure time and estimated waiting times allow users to spend their time productively elsewhere, and they are immediately seated upon arrival.

[0231] Specific example

[0232] For example, consider a scenario where a family visits a restaurant. Minimizing waiting times is especially important when children are present. The server identifies the family as customers and, based on their estimated stay time at their table, notifies the user of suggestions for other places to spend time. The system also calculates the optimal time for the customer to begin their journey, taking into account the travel time to the restaurant. Upon arrival, a smooth entry process is facilitated through a terminal, with staff guiding the customer. This allows customers to enjoy high-quality service without unnecessary waiting.

[0233] The introduction of this system is expected to improve the overall customer experience at the restaurant, enable more efficient store operations, and increase sales.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The server acquires video data from in-store cameras and uses image recognition technology to analyze the time customers spend at each table in real time. It measures the time customers spend at each table and uses reinforcement learning based on past data to improve the accuracy of predictions.

[0237] Step 2:

[0238] The server uses information obtained through image recognition to predict the length of time customers currently seated will stay. It calculates the time when seats are expected to become vacant, calculates the optimal seating arrangement based on the projected number of customers, and sends this information to the terminal.

[0239] Step 3:

[0240] Based on the information it receives, the terminal provides restaurant staff with specific instructions on which tables will become available and which customers should be seated at which tables. This allows staff to efficiently seat customers and optimize table turnover.

[0241] Step 4:

[0242] The server obtains the customer's location information and uses it to calculate the optimal time for the customer to begin their journey. The calculation result is then sent from the server to the user's smartphone to reduce waiting times.

[0243] Step 5:

[0244] Users receive notifications on their smartphones and prepare to head to the restaurant based on the suggested departure time. They can also check detailed information such as waiting times and estimated arrival times on their smartphones.

[0245] Step 6:

[0246] When the server detects that a user has started moving, it uses that information to update the wait time for greater accuracy. After arriving at the restaurant, the terminal prepares to smoothly guide the customer to the appropriate seat.

[0247] Step 7:

[0248] When a user arrives at the restaurant, the terminal instructs staff to guide them according to pre-calculated results, and the staff smoothly leads the customer to their designated seat. This allows customers to enjoy a seamless dining experience without experiencing unnecessary waiting times.

[0249] (Example 1)

[0250] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0251] In current store operations, it is difficult to quickly and accurately predict customer dwell times and achieve optimal seating arrangements. This can lead to unnecessary waiting times for customers, resulting in decreased customer satisfaction and insufficient store efficiency. Furthermore, the lack of mechanisms to learn from past data prevents improvements in prediction accuracy.

[0252] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0253] In this invention, the server includes means for acquiring image data using a sensing device for acquiring customer behavior data, an image analysis unit means for analyzing the acquired image data and predicting the customer's stay time, and means for acquiring the customer's geographical information and calculating the optimal start time for travel. This enables efficient seat allocation in line with the customer's arrival and notification of waiting times with high prediction accuracy.

[0254] A "sensing device" is a device that detects a customer's movements and location and acquires image data based on that information.

[0255] An "image analysis unit" is a system that processes acquired image data and performs analysis to predict customer movements and dwell time.

[0256] "Geographic information" refers to data that shows location information related to travel, such as a customer's current location or destination.

[0257] "Reinforcement learning" is a machine learning technique that uses historical data to repeatedly learn and make predictions about the future and optimize actions.

[0258] "Communication equipment" refers to devices used to transmit calculated information or notifications to customers, and includes smartphones and other mobile devices.

[0259] "Staff" refers to the personnel who guide customers within the store and are responsible for providing smooth service.

[0260] "Seat allocation" is the process of determining the optimal seating arrangement based on the predicted length of customer stay.

[0261] This invention relates to a system for minimizing customer waiting times in restaurants and efficiently managing seating. Specific embodiments are described below.

[0262] Server Role

[0263] The server uses image data acquired in real time from sensing devices within the restaurant. This includes cameras and other image sensors used to accurately track customer movements. The server analyzes the data using image processing software such as OpenCV to predict customer dwell time. A mechanism is in place to improve prediction accuracy by utilizing reinforcement learning algorithms and learning from past data. Furthermore, the server acquires customer geographical information and calculates the optimal time to start moving.

[0264] Terminal role

[0265] The terminal is a device that displays information provided by the server to restaurant staff. A special application is installed on the terminal so that staff can easily check customer waiting times and predicted seat availability. This allows staff to guide customers efficiently and optimize seating allocation within the restaurant.

[0266] User roles

[0267] Users receive notifications sent from the server via their smartphones. For example, information about the start time of travel and the estimated waiting time is sent to the user's smartphone as a push notification. This notification allows users to spend their time productively elsewhere and arrive at the restaurant at the optimal time.

[0268] Specific example

[0269] For example, when visiting a restaurant with family, the server notifies the user with suggestions for the best way to minimize waiting time and avoid crowds. This is especially beneficial for families with children. Upon arrival, staff guide the customer to their seats via a terminal to ensure a smooth entry.

[0270] Example of a prompt

[0271] Examples of prompts for the generating AI model include, "Please explain how to analyze image data to predict customer dwell time," and "Please describe an example of how the system can be used to optimize seating management in a restaurant."

[0272] The introduction of this system is expected to improve the overall operational efficiency of the restaurant, increase customer satisfaction, and boost sales.

[0273] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0274] Step 1:

[0275] The server acquires image data in real time from sensing devices within the restaurant. Camera images are taken as input, and image analysis is performed based on this data. Specifically, frame extraction is performed to record customer movements and postures. The output is data related to customer movement patterns and dwell time.

[0276] Step 2:

[0277] The server analyzes the acquired image data using image processing software such as OpenCV. The input is the movement information acquired in step 1, and based on this, it predicts the customer's dwell time. Specifically, it recognizes the customer's time spent in the store and their behavior patterns, and refines the prediction using a reinforcement learning algorithm that utilizes past data. The output is the predicted dwell time data.

[0278] Step 3:

[0279] The server calculates the optimal seating arrangement based on predicted dwell time data. The input is the dwell time data obtained in step 2, and the optimization algorithm is executed by taking into account the current seating situation. Specifically, it considers seat type and customer needs to perform efficient seating assignment. The output is the assigned seating information.

[0280] Step 4:

[0281] The terminal displays the seat arrangement information provided by the server to the staff. The input is the seat information generated in Step 3, and it is made possible to intuitively confirm this via the GUI. As a specific operation, an interface is provided that allows detailed information to be confirmed by touch operation. The output is a clear seat arrangement instruction for customer guidance that can be executed by the staff.

[0282] Step 5:

[0283] The user receives a notification from the server on the smartphone. The input is the stay time prediction in Step 2 and the result of the seat arrangement in Step 3, and based on this, notifications of the movement start time and waiting time are made. Specifically, the movement instruction is optimized based on the user's current location. The output is real-time movement guidance and waiting time information displayed on the user's device.

[0284] In this way, by the coordinated operation of each step, a system is completed that realizes restaurant seat management and improvement of customer satisfaction.

[0285] (Application Example 1)

[0286] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0287] In modern commercial facilities and restaurants, efficient queuing management that minimizes customer waiting time is required. However, in conventional systems, it is difficult to accurately predict the waiting time, which is a problem that leads to a decrease in customer satisfaction. Also, there is a lack of means to provide the optimal timing until the customer arrives at the store, and the problem is that it is not possible to provide a meaningful time for the customer.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0289] In this invention, the server includes means for acquiring electronic information to predict the customer's stay time, visual recognition device means for predicting the customer's stay time based on the acquired electronic information, and means for displaying the store's congestion status and the customer's place in line in real time on an information terminal. This allows customers to accurately understand their waiting time and efficiently plan their time in the store.

[0290] "Electronic information" refers to information that includes data necessary to identify customer behavior and characteristics.

[0291] A "visual recognition device" is a device that uses cameras and sensors to analyze customer characteristics from image and video data and predict their behavior.

[0292] An "information terminal" is a device that displays necessary information to the user and accepts their input, and includes smartphones and tablets.

[0293] "Duration of stay" refers to the time a customer spends in a store from the moment they enter until they leave.

[0294] "Crowding status" refers to information indicating the number and placement of customers within a store, and is used to understand the usage status of the facility.

[0295] "Queue management" is a method of properly managing the order in which customers enter and leave a store in order to minimize waiting times.

[0296] The system for implementing this invention aims to efficiently manage customer waiting times in restaurants and commercial facilities and improve customer satisfaction. The detailed configuration of the system is shown below.

[0297] The server first uses cameras and sensors to acquire electronic information about customer movements and dwell times within the facility. This information is analyzed by a visual recognition device, and an AI model is used to predict customer dwell times. Software used includes OpenCV for image analysis and TensorFlow for machine learning models. MySQL is used as the database software to store customer visit history and prediction results.

[0298] The device is a medium that receives data sent from the server. This device is a smartphone or tablet, acting as an information terminal, and displays customer dwell time predictions and current congestion status. The information is sent in real time via Firebase Cloud Messaging and displayed on the device in the form of a React Native application.

[0299] Based on the information obtained through this device, users can make efficient use of their time. Specifically, if there is a long wait, they can receive suggestions for spending time shopping or taking a walk nearby. Also, as the reserved time approaches, they will be notified of the optimal departure time, allowing them to enter the store at the specified time.

[0300] As a concrete example, this system can be used when dining out with family on weekends to determine the optimal arrival time in real time based on the restaurant's congestion, allowing the whole family to enjoy their meal without stress. Prompt messages such as, "Check the current waiting time and spend your time productively at home. You will be notified when your table is ready," are issued, providing customers with useful information.

[0301] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0302] Step 1:

[0303] The server acquires electronic information about customers in real time from cameras and sensors installed within the facility. At this point, the input is camera footage, and the output is image data. The image data undergoes preprocessing for use in the visual recognition device. Specifically, this includes noise removal and resolution adjustment, etc.

[0304] Step 2:

[0305] The server analyzes the image data using a visual recognition device and predicts the customer's stay time. The input is the preprocessed image data obtained in Step 1. Utilizing the generative AI model TensorFlow, data processing and pattern recognition are performed, and the output is the predicted stay time. This process includes face recognition and motion analysis.

[0306] Step 3:

[0307] The server refers to past customer data and improves the accuracy of stay time prediction through reinforcement learning. In this step, past visit history data and the prediction data from Step 2 are used as inputs. The output is a new prediction model that reflects the improvement in prediction accuracy. This model continuously learns and is adjusted to minimize prediction errors.

[0308] Step 4:

[0309] The server acquires the customer's location information and calculates the optimal start time for movement. The input here is the current location information obtained from a smartphone or GPS device. The output is the information notified to the customer in the form of the start time for movement. Calculations considering traffic conditions and distance are performed using an algorithm.

[0310] Step 5:

[0311] The device receives data on estimated dwell time and departure time from the server and displays the information on the customer's smartphone. The input is notification information from the server, and the output is the waiting time and arrival time information displayed on the device. Push notifications are sent via Firebase Cloud Messaging.

[0312] Step 6:

[0313] Based on the information received, the user heads to the facility according to the optimal departure time. The input is the notification information displayed on the terminal, and the output is the user's actions. The user uses this information to spend their time productively and arrive at the facility at the appropriate time.

[0314] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0315] This invention aims to improve customer service by providing a system that combines an emotion engine to streamline restaurant waiting time management and seating guidance. The emotion engine analyzes changes in the customer's facial expressions and voice to understand their emotions, thereby enabling more personalized service.

[0316] 1. Server Role

[0317] The server uses cameras and an emotion engine to collect customer facial expression data and analyze their emotional state based on this data. The emotion engine combines facial expression analysis and voice analysis to understand how customers feel about their waiting time.

[0318] 2. The role of the terminal

[0319] The terminal provides instructions to restaurant staff based on emotional state and predicted length of stay information transmitted from the server. This allows staff to understand the customer's emotional state and provide optimal service. For example, if a customer's emotional state is determined to be negative, the system can prioritize seating them.

[0320] 3. User Roles

[0321] Users can receive not only the waiting time and estimated departure time provided by the server, but also a less stressful waiting experience based on their own emotions. Since notifications are adjusted according to their emotions, users can make appropriate decisions about when and what actions to take.

[0322] Specific example

[0323] For example, when customers visit a restaurant with their family, children often get bored while waiting. When the emotion engine detects a negative change in the child's mood, the server uses that information to send instructions to the staff via a terminal. The staff then quickly prepares a table and adjusts the schedule so that the family is seated sooner. As a result, customers can enjoy a smooth and comfortable dining experience, which enhances the restaurant's reputation.

[0324] The introduction of this system will lead to higher levels of customer satisfaction, more efficient store operations, and flexible responses based on data.

[0325] The following describes the processing flow.

[0326] Step 1:

[0327] The server collects video and audio data in real time from multiple cameras and microphones installed in the store. Using an emotion engine, it analyzes the customer's facial expressions and voice characteristics from this data to determine the customer's emotional state.

[0328] Step 2:

[0329] The server uses image recognition to predict customer dwell time. Combined with the determined emotional state, it dynamically adjusts the acceptable waiting time range and priority for each customer. Customers with positive emotions are given a normal waiting time, while those with negative emotions are given a shorter waiting time.

[0330] Step 3:

[0331] The terminal receives each customer's emotional state and predicted length of stay from the server and displays this information to restaurant staff as visual instructions. Based on this information, staff determine the order in which to seat customers and optimize the customer experience.

[0332] Step 4:

[0333] The server tracks the customer's location and, based on the calculated waiting time, notifies the user's smartphone of the estimated start time of their journey. The notification is delivered in an appropriate tone and timing based on the customer's emotional state.

[0334] Step 5:

[0335] Users receive notifications on their smartphones and prepare to head to the restaurant according to the displayed departure time. A more comfortable waiting environment is provided thanks to the emotionally responsive waiting time.

[0336] Step 6:

[0337] When a user arrives at the restaurant, the terminal uses pre-prepared information to instruct staff, ensuring they are promptly seated at the most suitable table. This enables smooth customer service that takes emotions into consideration.

[0338] (Example 2)

[0339] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0340] In modern restaurants, the decline in customer satisfaction due to long wait times is a significant problem. In particular, when customer emotional states are not considered, the overall quality of service deteriorates, impacting the restaurant's reputation. Traditional waiting time management systems lack the ability to analyze customer emotions from facial expressions and voice, making it difficult to provide more personalized service. Therefore, there is a need for effective waiting time management and seating guidance that takes customer emotions into account.

[0341] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0342] In this invention, the server includes means for providing an input device for collecting customer facial expressions and voice data, processing means for analyzing the collected facial expressions and voice data and inferring the customer's emotional state, and means for generating service instructions for the customer based on the analyzed emotional state and transmitting them to a staff terminal. This enables quick and effective waiting time management and seating guidance that takes into account the customer's emotional state.

[0343] "Customer facial expression and voice data" refers to information related to a customer's facial expressions and voice, and is used to infer the customer's emotional state by analyzing this data.

[0344] An "input device" is a device, such as a camera or microphone, installed in a restaurant to collect facial expressions and voice data from customers.

[0345] A "processing device" is a computer system that analyzes collected facial and voice data to infer the customer's emotional state.

[0346] "Emotional state" refers to the psychological and emotional condition exhibited by a customer, as inferred from their facial expressions and voice.

[0347] "Generating service instructions" means determining specific actions and services that restaurant staff should take based on the customer's emotional state, and creating instructions to display on a terminal.

[0348] A "staff terminal" is a device used by restaurant staff to receive and display information transmitted from the server.

[0349] A description of embodiments for carrying out this invention will be given.

[0350] The server collects customer facial expressions and audio data through cameras and microphones installed in the restaurant. Video data acquired from the cameras is processed in real time using an image processing library to extract facial features and infer the customer's emotional state. Audio data is transcribed using speech analysis software, and features such as voice tone and speaking speed are analyzed. Based on this, the server infers the customer's overall emotional state and stores it in a database.

[0351] The terminal receives customer emotional states transmitted from the server and displays appropriate service instructions on the screen for store staff. The terminal is notified of the optimal response for the customer based on the analysis results, enabling staff to quickly provide situation-appropriate service. For example, if a customer is a family and the children are bored during a long wait, the terminal will prompt staff to quickly seat them. This allows staff to effectively perform their duties to ensure customer comfort.

[0352] Users can receive notifications from the server via their smartphones regarding waiting times and estimated service times. The user application displays actionable guidelines based on the data provided by the server, enabling users to make the waiting experience less stressful. This allows users to use their time efficiently, such as returning to the restaurant at the appropriate time.

[0353] As a concrete example, an example of a prompt message is shown below.

[0354] "Please describe in detail how you analyze customers' emotions in real time while they are waiting and provide the best possible service. Include examples of how to handle situations where customers with children are bored."

[0355] As described above, this invention aims to improve the efficiency of restaurant operations and enhance customer satisfaction by providing personalized services through the analysis of customers' emotional states.

[0356] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0357] Step 1:

[0358] The server collects customer facial expressions and audio data via cameras and microphones. Camera footage and audio from within the store are acquired as input. Specifically, the server captures this data in real time, performs initial processing, and converts it into an analyzable format.

[0359] Step 2:

[0360] The server analyzes the customer's emotional state using the collected data. The input is the facial and audio data collected in step 1, and the output is the estimated result of the customer's emotional state. Specifically, the server uses an image processing library to detect facial feature points and audio processing software to analyze the tone of voice and comprehensively estimate the emotion.

[0361] Step 3:

[0362] The server sends the analyzed emotional state to the terminal and generates instructions for display to store staff. The input is the emotional state data obtained in step 2, and the output is specific service instructions for staff. Specifically, the server creates a service scenario that takes the emotional state into account and pushes it to the terminal.

[0363] Step 4:

[0364] The terminal displays instructions to staff based on the instructions it receives, supporting service delivery. The input is service instructions sent from the server, and the output is the action staff should take. Specifically, the terminal displays alerts on the screen to clearly indicate to staff which customers should be given priority.

[0365] Step 5:

[0366] Users receive notifications from the server via a smartphone app, obtaining information about waiting times and estimated service times. Input is waiting status data provided by the server, while output is recommended information regarding user actions. Specifically, the user's app screen displays notifications such as "You will be served in 5 minutes," allowing users to efficiently manage their time.

[0367] (Application Example 2)

[0368] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0369] Customers often experience dissatisfaction due to long waiting times at facilities. Furthermore, standard service often struggles to provide individualized care based on customer emotions, making it difficult to improve customer satisfaction. Especially for families and customers whose emotions are easily swayed, there is a need to reduce frustration and stress during waiting times and provide a comfortable experience. An efficient system is needed to improve this customer experience.

[0370] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0371] In this invention, the server includes means for analyzing customer emotions and providing personalized services based on that data, means for predicting customer dwell time and optimizing location placement based on the prediction results, and means for acquiring customer location information and providing navigation guidance at the optimal timing. This makes it possible to provide individualized services that take customer emotions into consideration and to optimize dwell time management.

[0372] "Data for predicting customer dwell time" refers to information about customer behavior and movements within the facility.

[0373] A "recognition device" is a device that analyzes acquired information and derives a specific result.

[0374] "Location information" refers to information that indicates where a customer is currently located.

[0375] "Movement start time" refers to the calculated time when the customer is scheduled to begin their next action.

[0376] "Waiting time" refers to the expected time it takes for a customer to receive a particular service.

[0377] "Placement arrangement" refers to the positioning of each area and seating area within a facility.

[0378] "Analyzing customer emotions" means inferring a customer's psychological state and feelings based on their facial expressions and voice.

[0379] "Personalized service" refers to the provision of services that are customized to the individual needs and circumstances of each customer.

[0380] This system is designed to improve customer service in restaurants and other establishments, and primarily consists of three elements: servers, terminals, and users.

[0381] The server uses cameras and microphones installed within the facility to collect customer facial expressions and audio data. This data is processed by emotion analysis AI (Microsoft Azure Emotion API being a specific example) to analyze the customer's emotional state in real time. This allows the server to understand what emotions the customer is experiencing.

[0382] Subsequently, the terminal receives emotion data from the server and wait time information calculated based on the customer's predicted stay duration. The terminal then uses this information to send instructions to facility staff. For example, if a customer is deemed unhappy, it can be instructed to prioritize seating them. The terminal also has the function of optimizing guest flow and providing personalized entertainment and information during waiting times.

[0383] Users can receive information from their devices, reduce their own stress, and choose the optimal course of action. These notifications are tailored to the user's emotions, allowing for more flexible responses.

[0384] As a concrete example, when visiting a restaurant with family, robots are used to make waiting time more enjoyable by displaying animations on smartphones to keep children entertained. Another example of a prompt message is, "Please think of actions for a robot that analyzes the customer's voice and facial expressions and proposes the most appropriate service based on those emotions."

[0385] Thus, this invention aims to enhance the waiting experience within a facility and increase customer satisfaction by utilizing emotion analysis technology to provide personalized services to customers.

[0386] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0387] Step 1:

[0388] The server acquires real-time data from cameras and microphones installed within the facility. This data includes customer facial expressions and voice. Image and audio data are provided as input, forming the basis for subsequent processing.

[0389] Step 2:

[0390] The server uses the acquired data to analyze customer emotions using an emotion analysis AI model (e.g., Microsoft Azure Emotion API). This process extracts facial and vocal characteristics and outputs emotional states such as joy and anger. This output is a different emotion label for each customer.

[0391] Step 3:

[0392] The server sends the predicted dwell time and customer emotional state to the terminal based on the emotion analysis results. Here, the predicted emotional state and dwell time are input, and appropriate action instructions are output to each terminal accordingly.

[0393] Step 4:

[0394] The terminal uses emotional state and dwell time information received from the server to issue instructions to staff. These instructions may include, for example, promptly seating an unhappy customer or providing entertainment. Here, the input is emotional and time information, and the output is specific action instructions.

[0395] Step 5:

[0396] Based on information provided by the device, users adjust their travel timing and how they spend their waiting time. In this step, information that responds to emotions as input leads to the output of optimized customer behavior.

[0397] Step 6:

[0398] The server collects and stores data on each customer's emotional state and dwell time, and uses this data for reinforcement learning of the model. This aims to improve the accuracy of predictions and the quality of service. The input is the customer data history, and the output is the improved accuracy of emotion prediction.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0400] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0402] [Third Embodiment]

[0403] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0404] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0405] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0407] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0409] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0410] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0411] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0414] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0415] This invention is a system that minimizes customer waiting times in restaurants and enables efficient seating management. At its core, it utilizes image recognition technology and customer location information to provide customers with optimal information, thereby improving customer satisfaction.

[0416] 1. Server Role

[0417] The server acquires video data from cameras installed in the restaurant and uses image recognition technology to predict customer dwell times in real time. Furthermore, it improves prediction accuracy through reinforcement learning using past data. Based on these predictions, it calculates customer waiting times and determines the optimal seating arrangement.

[0418] 2. The role of the terminal

[0419] The terminal is a device that displays information provided by the server to restaurant staff. Through the terminal, staff can understand customer waiting times and predicted seat availability, allowing them to guide customers efficiently. This maximizes the seating occupancy rate in the restaurant.

[0420] 3. User Roles

[0421] Users receive real-time notifications from this system via their smartphones. Notifications of departure time and estimated waiting times allow users to spend their time productively elsewhere, and they are immediately seated upon arrival.

[0422] Specific example

[0423] For example, consider a scenario where a family visits a restaurant. Minimizing waiting times is especially important when children are present. The server identifies the family as customers and, based on their estimated stay time at their table, notifies the user of suggestions for other places to spend time. The system also calculates the optimal time for the customer to begin their journey, taking into account the travel time to the restaurant. Upon arrival, a smooth entry process is facilitated through a terminal, with staff guiding the customer. This allows customers to enjoy high-quality service without unnecessary waiting.

[0424] The introduction of this system is expected to improve the overall customer experience at the restaurant, enable more efficient store operations, and increase sales.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] The server acquires video data from in-store cameras and uses image recognition technology to analyze the time customers spend at each table in real time. It measures the time customers spend at each table and uses reinforcement learning based on past data to improve the accuracy of predictions.

[0428] Step 2:

[0429] The server uses information obtained through image recognition to predict the length of time customers currently seated will stay. It calculates the time when seats are expected to become vacant, calculates the optimal seating arrangement based on the projected number of customers, and sends this information to the terminal.

[0430] Step 3:

[0431] Based on the information it receives, the terminal provides restaurant staff with specific instructions on which tables will become available and which customers should be seated at which tables. This allows staff to efficiently seat customers and optimize table turnover.

[0432] Step 4:

[0433] The server obtains the customer's location information and uses it to calculate the optimal time for the customer to begin their journey. The calculation result is then sent from the server to the user's smartphone to reduce waiting times.

[0434] Step 5:

[0435] Users receive notifications on their smartphones and prepare to head to the restaurant based on the suggested departure time. They can also check detailed information such as waiting times and estimated arrival times on their smartphones.

[0436] Step 6:

[0437] When the server detects that a user has started moving, it uses that information to update the wait time for greater accuracy. After arriving at the restaurant, the terminal prepares to smoothly guide the customer to the appropriate seat.

[0438] Step 7:

[0439] When a user arrives at the restaurant, the terminal instructs staff to guide them according to pre-calculated results, and the staff smoothly leads the customer to their designated seat. This allows customers to enjoy a seamless dining experience without experiencing unnecessary waiting times.

[0440] (Example 1)

[0441] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0442] In current store operations, it is difficult to quickly and accurately predict customer dwell times and achieve optimal seating arrangements. This can lead to unnecessary waiting times for customers, resulting in decreased customer satisfaction and insufficient store efficiency. Furthermore, the lack of mechanisms to learn from past data prevents improvements in prediction accuracy.

[0443] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0444] In this invention, the server includes means for acquiring image data using a sensing device for acquiring customer behavior data, an image analysis unit means for analyzing the acquired image data and predicting the customer's stay time, and means for acquiring the customer's geographical information and calculating the optimal start time for travel. This enables efficient seat allocation in line with the customer's arrival and notification of waiting times with high prediction accuracy.

[0445] A "sensing device" is a device that detects a customer's movements and location and acquires image data based on that information.

[0446] An "image analysis unit" is a system that processes acquired image data and performs analysis to predict customer movements and dwell time.

[0447] "Geographic information" refers to data that shows location information related to travel, such as a customer's current location or destination.

[0448] "Reinforcement learning" is a machine learning technique that uses historical data to repeatedly learn and make predictions about the future and optimize actions.

[0449] "Communication equipment" refers to devices used to transmit calculated information or notifications to customers, and includes smartphones and other mobile devices.

[0450] "Staff" refers to the personnel who guide customers within the store and are responsible for providing smooth service.

[0451] "Seat allocation" is the process of determining the optimal seating arrangement based on the predicted length of customer stay.

[0452] This invention relates to a system for minimizing customer waiting times in restaurants and efficiently managing seating. Specific embodiments are described below.

[0453] Server Role

[0454] The server uses image data acquired in real time from sensing devices within the restaurant. This includes cameras and other image sensors used to accurately track customer movements. The server analyzes the data using image processing software such as OpenCV to predict customer dwell time. A mechanism is in place to improve prediction accuracy by utilizing reinforcement learning algorithms and learning from past data. Furthermore, the server acquires customer geographical information and calculates the optimal time to start moving.

[0455] Terminal role

[0456] The terminal is a device that displays information provided by the server to restaurant staff. A special application is installed on the terminal so that staff can easily check customer waiting times and predicted seat availability. This allows staff to guide customers efficiently and optimize seating allocation within the restaurant.

[0457] User roles

[0458] Users receive notifications sent from the server via their smartphones. For example, information about the start time of travel and the estimated waiting time is sent to the user's smartphone as a push notification. This notification allows users to spend their time productively elsewhere and arrive at the restaurant at the optimal time.

[0459] Specific example

[0460] For example, when visiting a restaurant with family, the server notifies the user with suggestions for the best way to minimize waiting time and avoid crowds. This is especially beneficial for families with children. Upon arrival, staff guide the customer to their seats via a terminal to ensure a smooth entry.

[0461] Example of a prompt

[0462] Examples of prompts for the generating AI model include, "Please explain how to analyze image data to predict customer dwell time," and "Please describe an example of how the system can be used to optimize seating management in a restaurant."

[0463] The introduction of this system is expected to improve the overall operational efficiency of the restaurant, increase customer satisfaction, and boost sales.

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The server acquires image data in real time from sensing devices within the restaurant. Camera images are taken as input, and image analysis is performed based on this data. Specifically, frame extraction is performed to record customer movements and postures. The output is data related to customer movement patterns and dwell time.

[0467] Step 2:

[0468] The server analyzes the acquired image data using image processing software such as OpenCV. The input is the movement information acquired in step 1, and based on this, it predicts the customer's dwell time. Specifically, it recognizes the customer's time spent in the store and their behavior patterns, and refines the prediction using a reinforcement learning algorithm that utilizes past data. The output is the predicted dwell time data.

[0469] Step 3:

[0470] The server calculates the optimal seating arrangement based on predicted dwell time data. The input is the dwell time data obtained in step 2, and the optimization algorithm is executed by taking into account the current seating situation. Specifically, it considers seat type and customer needs to perform efficient seating assignment. The output is the assigned seating information.

[0471] Step 4:

[0472] The terminal displays seating arrangement information provided by the server to the staff. The input is the seating information generated in step 3, which is made intuitively viewable via a GUI. Specifically, an interface is provided that allows detailed information to be viewed via touch operation. The output is clear seating arrangement instructions that staff can use to guide customers.

[0473] Step 5:

[0474] The user receives notifications from the server on their smartphone. The input is the result of the dwell time prediction in step 2 and seat assignment in step 3, and notifications of departure time and waiting time are provided based on this. Specifically, the travel instructions are optimized based on the user's current location. The output is real-time travel guidance and waiting time information displayed on the user's device.

[0475] In this way, each step works in conjunction to create a system that manages restaurant seating and improves customer satisfaction.

[0476] (Application Example 1)

[0477] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] In modern commercial facilities and restaurants, efficient queue management that minimizes customer waiting times is essential. However, traditional systems struggle to accurately predict waiting times, leading to decreased customer satisfaction. Furthermore, there is a lack of means to provide customers with the optimal timing for arriving at the store, resulting in a failure to offer customers a meaningful experience.

[0479] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0480] In this invention, the server includes means for acquiring electronic information to predict the customer's stay time, visual recognition device means for predicting the customer's stay time based on the acquired electronic information, and means for displaying the store's congestion status and the customer's place in line in real time on an information terminal. This allows customers to accurately understand their waiting time and efficiently plan their time in the store.

[0481] "Electronic information" refers to information that includes data necessary to identify customer behavior and characteristics.

[0482] A "visual recognition device" is a device that uses cameras and sensors to analyze customer characteristics from image and video data and predict their behavior.

[0483] An "information terminal" is a device that displays necessary information to the user and accepts their input, and includes smartphones and tablets.

[0484] "Duration of stay" refers to the time a customer spends in a store from the moment they enter until they leave.

[0485] "Crowding status" refers to information indicating the number and placement of customers within a store, and is used to understand the usage status of the facility.

[0486] "Queue management" is a method of properly managing the order in which customers enter and leave a store in order to minimize waiting times.

[0487] The system for implementing this invention aims to efficiently manage customer waiting times in restaurants and commercial facilities and improve customer satisfaction. The detailed configuration of the system is shown below.

[0488] The server first uses cameras and sensors to acquire electronic information about customer movements and dwell times within the facility. This information is analyzed by a visual recognition device, and an AI model is used to predict customer dwell times. Software used includes OpenCV for image analysis and TensorFlow for machine learning models. MySQL is used as the database software to store customer visit history and prediction results.

[0489] The device is a medium that receives data sent from the server. This device is a smartphone or tablet, acting as an information terminal, and displays customer dwell time predictions and current congestion status. The information is sent in real time via Firebase Cloud Messaging and displayed on the device in the form of a React Native application.

[0490] Based on the information obtained through this device, users can make efficient use of their time. Specifically, if there is a long wait, they can receive suggestions for spending time shopping or taking a walk nearby. Also, as the reserved time approaches, they will be notified of the optimal departure time, allowing them to enter the store at the specified time.

[0491] As a concrete example, this system can be used when dining out with family on weekends to determine the optimal arrival time in real time based on the restaurant's congestion, allowing the whole family to enjoy their meal without stress. Prompt messages such as, "Check the current waiting time and spend your time productively at home. You will be notified when your table is ready," are issued, providing customers with useful information.

[0492] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0493] Step 1:

[0494] The server acquires electronic information about customers in real time from cameras and sensors installed within the facility. At this point, the input is camera footage, and the output is image data. The image data is pre-processed for use in visual recognition devices. Specifically, this includes noise reduction and resolution adjustment.

[0495] Step 2:

[0496] The server analyzes image data using a visual recognition device to predict customer dwell time. The input is pre-processed image data obtained in step 1. Using the generative AI model TensorFlow, data processing and pattern recognition are performed, and the predicted dwell time is output. This process includes facial recognition and motion analysis.

[0497] Step 3:

[0498] The server references historical customer data and improves the accuracy of dwell time predictions through reinforcement learning. In this step, historical visit history data and the prediction data from step 2 are used as input. The output is a new prediction model that reflects the improvement in prediction accuracy. This model is continuously learned and adjusted to minimize prediction errors.

[0499] Step 4:

[0500] The server obtains the customer's location information and calculates the optimal departure time. The input here is the current location information obtained from a smartphone or GPS device. The output is information notified to the customer in the form of a departure start time. An algorithm is used to perform the calculation, taking into account traffic conditions and distance.

[0501] Step 5:

[0502] The device receives data on estimated dwell time and departure time from the server and displays the information on the customer's smartphone. The input is notification information from the server, and the output is the waiting time and arrival time information displayed on the device. Push notifications are sent via Firebase Cloud Messaging.

[0503] Step 6:

[0504] Based on the information received, the user heads to the facility according to the optimal departure time. The input is the notification information displayed on the terminal, and the output is the user's actions. The user uses this information to spend their time productively and arrive at the facility at the appropriate time.

[0505] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0506] This invention aims to improve customer service by providing a system that combines an emotion engine to streamline restaurant waiting time management and seating guidance. The emotion engine analyzes changes in the customer's facial expressions and voice to understand their emotions, thereby enabling more personalized service.

[0507] 1. Server Role

[0508] The server uses cameras and an emotion engine to collect customer facial expression data and analyze their emotional state based on this data. The emotion engine combines facial expression analysis and voice analysis to understand how customers feel about their waiting time.

[0509] 2. The role of the terminal

[0510] The terminal provides instructions to restaurant staff based on emotional state and predicted length of stay information transmitted from the server. This allows staff to understand the customer's emotional state and provide optimal service. For example, if a customer's emotional state is determined to be negative, the system can prioritize seating them.

[0511] 3. User Roles

[0512] Users can receive not only the waiting time and estimated departure time provided by the server, but also a less stressful waiting experience based on their own emotions. Since notifications are adjusted according to their emotions, users can make appropriate decisions about when and what actions to take.

[0513] Specific example

[0514] For example, when customers visit a restaurant with their family, children often get bored while waiting. When the emotion engine detects a negative change in the child's mood, the server uses that information to send instructions to the staff via a terminal. The staff then quickly prepares a table and adjusts the schedule so that the family is seated sooner. As a result, customers can enjoy a smooth and comfortable dining experience, which enhances the restaurant's reputation.

[0515] The introduction of this system will lead to higher levels of customer satisfaction, more efficient store operations, and flexible responses based on data.

[0516] The following describes the processing flow.

[0517] Step 1:

[0518] The server collects video and audio data in real time from multiple cameras and microphones installed in the store. Using an emotion engine, it analyzes the customer's facial expressions and voice characteristics from this data to determine the customer's emotional state.

[0519] Step 2:

[0520] The server uses image recognition to predict customer dwell time. Combined with the determined emotional state, it dynamically adjusts the acceptable waiting time range and priority for each customer. Customers with positive emotions are given a normal waiting time, while those with negative emotions are given a shorter waiting time.

[0521] Step 3:

[0522] The terminal receives each customer's emotional state and predicted length of stay from the server and displays this information to restaurant staff as visual instructions. Based on this information, staff determine the order in which to seat customers and optimize the customer experience.

[0523] Step 4:

[0524] The server tracks the customer's location and, based on the calculated waiting time, notifies the user's smartphone of the estimated start time of their journey. The notification is delivered in an appropriate tone and timing based on the customer's emotional state.

[0525] Step 5:

[0526] Users receive notifications on their smartphones and prepare to head to the restaurant according to the displayed departure time. A more comfortable waiting environment is provided thanks to the emotionally responsive waiting time.

[0527] Step 6:

[0528] When a user arrives at the restaurant, the terminal uses pre-prepared information to instruct staff, ensuring they are promptly seated at the most suitable table. This enables smooth customer service that takes emotions into consideration.

[0529] (Example 2)

[0530] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0531] In modern restaurants, the decline in customer satisfaction due to long wait times is a significant problem. In particular, when customer emotional states are not considered, the overall quality of service deteriorates, impacting the restaurant's reputation. Traditional waiting time management systems lack the ability to analyze customer emotions from facial expressions and voice, making it difficult to provide more personalized service. Therefore, there is a need for effective waiting time management and seating guidance that takes customer emotions into account.

[0532] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0533] In this invention, the server includes means for providing an input device for collecting customer facial expressions and voice data, processing means for analyzing the collected facial expressions and voice data and inferring the customer's emotional state, and means for generating service instructions for the customer based on the analyzed emotional state and transmitting them to a staff terminal. This enables quick and effective waiting time management and seating guidance that takes into account the customer's emotional state.

[0534] "Customer facial expression and voice data" refers to information related to a customer's facial expressions and voice, and is used to infer the customer's emotional state by analyzing this data.

[0535] An "input device" is a device, such as a camera or microphone, installed in a restaurant to collect facial expressions and voice data from customers.

[0536] A "processing device" is a computer system that analyzes collected facial and voice data to infer the customer's emotional state.

[0537] "Emotional state" refers to the psychological and emotional condition exhibited by a customer, as inferred from their facial expressions and voice.

[0538] "Generating service instructions" means determining specific actions and services that restaurant staff should take based on the customer's emotional state, and creating instructions to display on a terminal.

[0539] A "staff terminal" is a device used by restaurant staff to receive and display information transmitted from the server.

[0540] A description of embodiments for carrying out this invention will be given.

[0541] The server collects customer facial expressions and audio data through cameras and microphones installed in the restaurant. Video data acquired from the cameras is processed in real time using an image processing library to extract facial features and infer the customer's emotional state. Audio data is transcribed using speech analysis software, and features such as voice tone and speaking speed are analyzed. Based on this, the server infers the customer's overall emotional state and stores it in a database.

[0542] The terminal receives customer emotional states transmitted from the server and displays appropriate service instructions on the screen for store staff. The terminal is notified of the optimal response for the customer based on the analysis results, enabling staff to quickly provide situation-appropriate service. For example, if a customer is a family and the children are bored during a long wait, the terminal will prompt staff to quickly seat them. This allows staff to effectively perform their duties to ensure customer comfort.

[0543] Users can receive notifications from the server via their smartphones regarding waiting times and estimated service times. The user application displays actionable guidelines based on the data provided by the server, enabling users to make the waiting experience less stressful. This allows users to use their time efficiently, such as returning to the restaurant at the appropriate time.

[0544] As a concrete example, an example of a prompt message is shown below.

[0545] "Please describe in detail how you analyze customers' emotions in real time while they are waiting and provide the best possible service. Include examples of how to handle situations where customers with children are bored."

[0546] As described above, this invention aims to improve the efficiency of restaurant operations and enhance customer satisfaction by providing personalized services through the analysis of customers' emotional states.

[0547] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0548] Step 1:

[0549] The server collects customer facial expressions and audio data via cameras and microphones. Camera footage and audio from within the store are acquired as input. Specifically, the server captures this data in real time, performs initial processing, and converts it into an analyzable format.

[0550] Step 2:

[0551] The server analyzes the customer's emotional state using the collected data. The input is the facial and audio data collected in step 1, and the output is the estimated result of the customer's emotional state. Specifically, the server uses an image processing library to detect facial feature points and audio processing software to analyze the tone of voice and comprehensively estimate the emotion.

[0552] Step 3:

[0553] The server sends the analyzed emotional state to the terminal and generates instructions for display to store staff. The input is the emotional state data obtained in step 2, and the output is specific service instructions for staff. Specifically, the server creates a service scenario that takes the emotional state into account and pushes it to the terminal.

[0554] Step 4:

[0555] The terminal displays instructions to staff based on the instructions it receives, supporting service delivery. The input is service instructions sent from the server, and the output is the action staff should take. Specifically, the terminal displays alerts on the screen to clearly indicate to staff which customers should be given priority.

[0556] Step 5:

[0557] Users receive notifications from the server via a smartphone app, obtaining information about waiting times and estimated service times. Input is waiting status data provided by the server, while output is recommended information regarding user actions. Specifically, the user's app screen displays notifications such as "You will be served in 5 minutes," allowing users to efficiently manage their time.

[0558] (Application Example 2)

[0559] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] Customers often experience dissatisfaction due to long waiting times at facilities. Furthermore, standard service often struggles to provide individualized care based on customer emotions, making it difficult to improve customer satisfaction. Especially for families and customers whose emotions are easily swayed, there is a need to reduce frustration and stress during waiting times and provide a comfortable experience. An efficient system is needed to improve this customer experience.

[0561] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0562] In this invention, the server includes means for analyzing customer emotions and providing personalized services based on that data, means for predicting customer dwell time and optimizing location placement based on the prediction results, and means for acquiring customer location information and providing navigation guidance at the optimal timing. This makes it possible to provide individualized services that take customer emotions into consideration and to optimize dwell time management.

[0563] "Data for predicting customer dwell time" refers to information about customer behavior and movements within the facility.

[0564] A "recognition device" is a device that analyzes acquired information and derives a specific result.

[0565] "Location information" refers to information that indicates where a customer is currently located.

[0566] "Movement start time" refers to the calculated time when the customer is scheduled to begin their next action.

[0567] "Waiting time" refers to the expected time it takes for a customer to receive a particular service.

[0568] "Placement arrangement" refers to the positioning of each area and seating area within a facility.

[0569] "Analyzing customer emotions" means inferring a customer's psychological state and feelings based on their facial expressions and voice.

[0570] "Personalized service" refers to the provision of services that are customized to the individual needs and circumstances of each customer.

[0571] This system is designed to improve customer service in restaurants and other establishments, and primarily consists of three elements: servers, terminals, and users.

[0572] The server uses cameras and microphones installed within the facility to collect customer facial expressions and audio data. This data is processed by emotion analysis AI (Microsoft Azure Emotion API being a specific example) to analyze the customer's emotional state in real time. This allows the server to understand what emotions the customer is experiencing.

[0573] Subsequently, the terminal receives emotion data from the server and wait time information calculated based on the customer's predicted stay duration. The terminal then uses this information to send instructions to facility staff. For example, if a customer is deemed unhappy, it can be instructed to prioritize seating them. The terminal also has the function of optimizing guest flow and providing personalized entertainment and information during waiting times.

[0574] Users can receive information from their devices, reduce their own stress, and choose the optimal course of action. These notifications are tailored to the user's emotions, allowing for more flexible responses.

[0575] As a concrete example, when visiting a restaurant with family, robots are used to make waiting time more enjoyable by displaying animations on smartphones to keep children entertained. Another example of a prompt message is, "Please think of actions for a robot that analyzes the customer's voice and facial expressions and proposes the most appropriate service based on those emotions."

[0576] Thus, this invention aims to enhance the waiting experience within a facility and increase customer satisfaction by utilizing emotion analysis technology to provide personalized services to customers.

[0577] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0578] Step 1:

[0579] The server acquires real-time data from cameras and microphones installed within the facility. This data includes customer facial expressions and voice. Image and audio data are provided as input, forming the basis for subsequent processing.

[0580] Step 2:

[0581] The server uses the acquired data to analyze customer emotions using an emotion analysis AI model (e.g., Microsoft Azure Emotion API). This process extracts facial and vocal characteristics and outputs emotional states such as joy and anger. This output is a different emotion label for each customer.

[0582] Step 3:

[0583] The server sends the predicted dwell time and customer emotional state to the terminal based on the emotion analysis results. Here, the predicted emotional state and dwell time are input, and appropriate action instructions are output to each terminal accordingly.

[0584] Step 4:

[0585] The terminal uses emotional state and dwell time information received from the server to issue instructions to staff. These instructions may include, for example, promptly seating an unhappy customer or providing entertainment. Here, the input is emotional and time information, and the output is specific action instructions.

[0586] Step 5:

[0587] Based on information provided by the device, users adjust their travel timing and how they spend their waiting time. In this step, information that responds to emotions as input leads to the output of optimized customer behavior.

[0588] Step 6:

[0589] The server collects and stores data on each customer's emotional state and dwell time, and uses this data for reinforcement learning of the model. This aims to improve the accuracy of predictions and the quality of service. The input is the customer data history, and the output is the improved accuracy of emotion prediction.

[0590] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0591] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0592] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0593] [Fourth Embodiment]

[0594] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0595] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0596] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0597] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0598] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0599] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0600] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0601] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0602] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0603] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0604] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0605] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0606] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0607] This invention is a system that minimizes customer waiting times in restaurants and enables efficient seating management. At its core, it utilizes image recognition technology and customer location information to provide customers with optimal information, thereby improving customer satisfaction.

[0608] 1. Server Role

[0609] The server acquires video data from cameras installed in the restaurant and uses image recognition technology to predict customer dwell times in real time. Furthermore, it improves prediction accuracy through reinforcement learning using past data. Based on these predictions, it calculates customer waiting times and determines the optimal seating arrangement.

[0610] 2. The role of the terminal

[0611] The terminal is a device that displays information provided by the server to restaurant staff. Through the terminal, staff can understand customer waiting times and predicted seat availability, allowing them to guide customers efficiently. This maximizes the seating occupancy rate in the restaurant.

[0612] 3. User Roles

[0613] Users receive real-time notifications from this system via their smartphones. Notifications of departure time and estimated waiting times allow users to spend their time productively elsewhere, and they are immediately seated upon arrival.

[0614] Specific example

[0615] For example, consider a scenario where a family visits a restaurant. Minimizing waiting times is especially important when children are present. The server identifies the family as customers and, based on their estimated stay time at their table, notifies the user of suggestions for other places to spend time. The system also calculates the optimal time for the customer to begin their journey, taking into account the travel time to the restaurant. Upon arrival, a smooth entry process is facilitated through a terminal, with staff guiding the customer. This allows customers to enjoy high-quality service without unnecessary waiting.

[0616] The introduction of this system is expected to improve the overall customer experience at the restaurant, enable more efficient store operations, and increase sales.

[0617] The following describes the processing flow.

[0618] Step 1:

[0619] The server acquires video data from in-store cameras and uses image recognition technology to analyze the time customers spend at each table in real time. It measures the time customers spend at each table and uses reinforcement learning based on past data to improve the accuracy of predictions.

[0620] Step 2:

[0621] The server uses information obtained through image recognition to predict the length of time customers currently seated will stay. It calculates the time when seats are expected to become vacant, calculates the optimal seating arrangement based on the projected number of customers, and sends this information to the terminal.

[0622] Step 3:

[0623] Based on the information it receives, the terminal provides restaurant staff with specific instructions on which tables will become available and which customers should be seated at which tables. This allows staff to efficiently seat customers and optimize table turnover.

[0624] Step 4:

[0625] The server obtains the customer's location information and uses it to calculate the optimal time for the customer to begin their journey. The calculation result is then sent from the server to the user's smartphone to reduce waiting times.

[0626] Step 5:

[0627] Users receive notifications on their smartphones and prepare to head to the restaurant based on the suggested departure time. They can also check detailed information such as waiting times and estimated arrival times on their smartphones.

[0628] Step 6:

[0629] When the server detects that a user has started moving, it uses that information to update the wait time for greater accuracy. After arriving at the restaurant, the terminal prepares to smoothly guide the customer to the appropriate seat.

[0630] Step 7:

[0631] When a user arrives at the restaurant, the terminal instructs staff to guide them according to pre-calculated results, and the staff smoothly leads the customer to their designated seat. This allows customers to enjoy a seamless dining experience without experiencing unnecessary waiting times.

[0632] (Example 1)

[0633] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] In current store operations, it is difficult to quickly and accurately predict customer dwell times and achieve optimal seating arrangements. This can lead to unnecessary waiting times for customers, resulting in decreased customer satisfaction and insufficient store efficiency. Furthermore, the lack of mechanisms to learn from past data prevents improvements in prediction accuracy.

[0635] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0636] In this invention, the server includes means for acquiring image data using a sensing device for acquiring customer behavior data, an image analysis unit means for analyzing the acquired image data and predicting the customer's stay time, and means for acquiring the customer's geographical information and calculating the optimal start time for travel. This enables efficient seat allocation in line with the customer's arrival and notification of waiting times with high prediction accuracy.

[0637] A "sensing device" is a device that detects a customer's movements and location and acquires image data based on that information.

[0638] An "image analysis unit" is a system that processes acquired image data and performs analysis to predict customer movements and dwell time.

[0639] "Geographic information" refers to data that shows location information related to travel, such as a customer's current location or destination.

[0640] "Reinforcement learning" is a machine learning technique that uses historical data to repeatedly learn and make predictions about the future and optimize actions.

[0641] "Communication equipment" refers to devices used to transmit calculated information or notifications to customers, and includes smartphones and other mobile devices.

[0642] "Staff" refers to the personnel who guide customers within the store and are responsible for providing smooth service.

[0643] "Seat allocation" is the process of determining the optimal seating arrangement based on the predicted length of customer stay.

[0644] This invention relates to a system for minimizing customer waiting times in restaurants and efficiently managing seating. Specific embodiments are described below.

[0645] Server Role

[0646] The server uses image data acquired in real time from sensing devices within the restaurant. This includes cameras and other image sensors used to accurately track customer movements. The server analyzes the data using image processing software such as OpenCV to predict customer dwell time. A mechanism is in place to improve prediction accuracy by utilizing reinforcement learning algorithms and learning from past data. Furthermore, the server acquires customer geographical information and calculates the optimal time to start moving.

[0647] Terminal role

[0648] The terminal is a device that displays information provided by the server to restaurant staff. A special application is installed on the terminal so that staff can easily check customer waiting times and predicted seat availability. This allows staff to guide customers efficiently and optimize seating allocation within the restaurant.

[0649] User roles

[0650] Users receive notifications sent from the server via their smartphones. For example, information about the start time of travel and the estimated waiting time is sent to the user's smartphone as a push notification. This notification allows users to spend their time productively elsewhere and arrive at the restaurant at the optimal time.

[0651] Specific example

[0652] For example, when visiting a restaurant with family, the server notifies the user with suggestions for the best way to minimize waiting time and avoid crowds. This is especially beneficial for families with children. Upon arrival, staff guide the customer to their seats via a terminal to ensure a smooth entry.

[0653] Example of a prompt

[0654] Examples of prompts for the generating AI model include, "Please explain how to analyze image data to predict customer dwell time," and "Please describe an example of how the system can be used to optimize seating management in a restaurant."

[0655] The introduction of this system is expected to improve the overall operational efficiency of the restaurant, increase customer satisfaction, and boost sales.

[0656] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0657] Step 1:

[0658] The server acquires image data in real time from sensing devices within the restaurant. Camera images are taken as input, and image analysis is performed based on this data. Specifically, frame extraction is performed to record customer movements and postures. The output is data related to customer movement patterns and dwell time.

[0659] Step 2:

[0660] The server analyzes the acquired image data using image processing software such as OpenCV. The input is the movement information acquired in step 1, and based on this, it predicts the customer's dwell time. Specifically, it recognizes the customer's time spent in the store and their behavior patterns, and refines the prediction using a reinforcement learning algorithm that utilizes past data. The output is the predicted dwell time data.

[0661] Step 3:

[0662] The server calculates the optimal seating arrangement based on predicted dwell time data. The input is the dwell time data obtained in step 2, and the optimization algorithm is executed by taking into account the current seating situation. Specifically, it considers seat type and customer needs to perform efficient seating assignment. The output is the assigned seating information.

[0663] Step 4:

[0664] The terminal displays seating arrangement information provided by the server to the staff. The input is the seating information generated in step 3, which is made intuitively viewable via a GUI. Specifically, an interface is provided that allows detailed information to be viewed via touch operation. The output is clear seating arrangement instructions that staff can use to guide customers.

[0665] Step 5:

[0666] The user receives notifications from the server on their smartphone. The input is the result of the dwell time prediction in step 2 and seat assignment in step 3, and notifications of departure time and waiting time are provided based on this. Specifically, the travel instructions are optimized based on the user's current location. The output is real-time travel guidance and waiting time information displayed on the user's device.

[0667] In this way, each step works in conjunction to create a system that manages restaurant seating and improves customer satisfaction.

[0668] (Application Example 1)

[0669] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] In modern commercial facilities and restaurants, efficient queue management that minimizes customer waiting times is essential. However, traditional systems struggle to accurately predict waiting times, leading to decreased customer satisfaction. Furthermore, there is a lack of means to provide customers with the optimal timing for arriving at the store, resulting in a failure to offer customers a meaningful experience.

[0671] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0672] In this invention, the server includes means for acquiring electronic information to predict the customer's stay time, visual recognition device means for predicting the customer's stay time based on the acquired electronic information, and means for displaying the store's congestion status and the customer's place in line in real time on an information terminal. This allows customers to accurately understand their waiting time and efficiently plan their time in the store.

[0673] "Electronic information" refers to information that includes data necessary to identify customer behavior and characteristics.

[0674] A "visual recognition device" is a device that uses cameras and sensors to analyze customer characteristics from image and video data and predict their behavior.

[0675] An "information terminal" is a device that displays necessary information to the user and accepts their input, and includes smartphones and tablets.

[0676] "Duration of stay" refers to the time a customer spends in a store from the moment they enter until they leave.

[0677] "Crowding status" refers to information indicating the number and placement of customers within a store, and is used to understand the usage status of the facility.

[0678] "Queue management" is a method of properly managing the order in which customers enter and leave a store in order to minimize waiting times.

[0679] The system for implementing this invention aims to efficiently manage customer waiting times in restaurants and commercial facilities and improve customer satisfaction. The detailed configuration of the system is shown below.

[0680] The server first uses cameras and sensors to acquire electronic information about customer movements and dwell times within the facility. This information is analyzed by a visual recognition device, and an AI model is used to predict customer dwell times. Software used includes OpenCV for image analysis and TensorFlow for machine learning models. MySQL is used as the database software to store customer visit history and prediction results.

[0681] The device is a medium that receives data sent from the server. This device is a smartphone or tablet, acting as an information terminal, and displays customer dwell time predictions and current congestion status. The information is sent in real time via Firebase Cloud Messaging and displayed on the device in the form of a React Native application.

[0682] Based on the information obtained through this device, users can make efficient use of their time. Specifically, if there is a long wait, they can receive suggestions for spending time shopping or taking a walk nearby. Also, as the reserved time approaches, they will be notified of the optimal departure time, allowing them to enter the store at the specified time.

[0683] As a concrete example, this system can be used when dining out with family on weekends to determine the optimal arrival time in real time based on the restaurant's congestion, allowing the whole family to enjoy their meal without stress. Prompt messages such as, "Check the current waiting time and spend your time productively at home. You will be notified when your table is ready," are issued, providing customers with useful information.

[0684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0685] Step 1:

[0686] The server acquires electronic information about customers in real time from cameras and sensors installed within the facility. At this point, the input is camera footage, and the output is image data. The image data is pre-processed for use in visual recognition devices. Specifically, this includes noise reduction and resolution adjustment.

[0687] Step 2:

[0688] The server analyzes image data using a visual recognition device to predict customer dwell time. The input is pre-processed image data obtained in step 1. Using the generative AI model TensorFlow, data processing and pattern recognition are performed, and the predicted dwell time is output. This process includes facial recognition and motion analysis.

[0689] Step 3:

[0690] The server references historical customer data and improves the accuracy of dwell time predictions through reinforcement learning. In this step, historical visit history data and the prediction data from step 2 are used as input. The output is a new prediction model that reflects the improvement in prediction accuracy. This model is continuously learned and adjusted to minimize prediction errors.

[0691] Step 4:

[0692] The server obtains the customer's location information and calculates the optimal departure time. The input here is the current location information obtained from a smartphone or GPS device. The output is information notified to the customer in the form of a departure start time. An algorithm is used to perform the calculation, taking into account traffic conditions and distance.

[0693] Step 5:

[0694] The device receives data on estimated dwell time and departure time from the server and displays the information on the customer's smartphone. The input is notification information from the server, and the output is the waiting time and arrival time information displayed on the device. Push notifications are sent via Firebase Cloud Messaging.

[0695] Step 6:

[0696] Based on the information received, the user heads to the facility according to the optimal departure time. The input is the notification information displayed on the terminal, and the output is the user's actions. The user uses this information to spend their time productively and arrive at the facility at the appropriate time.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] This invention aims to improve customer service by providing a system that combines an emotion engine to streamline restaurant waiting time management and seating guidance. The emotion engine analyzes changes in the customer's facial expressions and voice to understand their emotions, thereby enabling more personalized service.

[0699] 1. Server Role

[0700] The server uses cameras and an emotion engine to collect customer facial expression data and analyze their emotional state based on this data. The emotion engine combines facial expression analysis and voice analysis to understand how customers feel about their waiting time.

[0701] 2. The role of the terminal

[0702] The terminal provides instructions to restaurant staff based on emotional state and predicted length of stay information transmitted from the server. This allows staff to understand the customer's emotional state and provide optimal service. For example, if a customer's emotional state is determined to be negative, the system can prioritize seating them.

[0703] 3. User Roles

[0704] Users can receive not only the waiting time and estimated departure time provided by the server, but also a less stressful waiting experience based on their own emotions. Since notifications are adjusted according to their emotions, users can make appropriate decisions about when and what actions to take.

[0705] Specific example

[0706] For example, when customers visit a restaurant with their family, children often get bored while waiting. When the emotion engine detects a negative change in the child's mood, the server uses that information to send instructions to the staff via a terminal. The staff then quickly prepares a table and adjusts the schedule so that the family is seated sooner. As a result, customers can enjoy a smooth and comfortable dining experience, which enhances the restaurant's reputation.

[0707] The introduction of this system will lead to higher levels of customer satisfaction, more efficient store operations, and flexible responses based on data.

[0708] The following describes the processing flow.

[0709] Step 1:

[0710] The server collects video and audio data in real time from multiple cameras and microphones installed in the store. Using an emotion engine, it analyzes the customer's facial expressions and voice characteristics from this data to determine the customer's emotional state.

[0711] Step 2:

[0712] The server uses image recognition to predict customer dwell time. Combined with the determined emotional state, it dynamically adjusts the acceptable waiting time range and priority for each customer. Customers with positive emotions are given a normal waiting time, while those with negative emotions are given a shorter waiting time.

[0713] Step 3:

[0714] The terminal receives each customer's emotional state and predicted length of stay from the server and displays this information to restaurant staff as visual instructions. Based on this information, staff determine the order in which to seat customers and optimize the customer experience.

[0715] Step 4:

[0716] The server tracks the customer's location and, based on the calculated waiting time, notifies the user's smartphone of the estimated start time of their journey. The notification is delivered in an appropriate tone and timing based on the customer's emotional state.

[0717] Step 5:

[0718] Users receive notifications on their smartphones and prepare to head to the restaurant according to the displayed departure time. A more comfortable waiting environment is provided thanks to the emotionally responsive waiting time.

[0719] Step 6:

[0720] When a user arrives at the restaurant, the terminal uses pre-prepared information to instruct staff, ensuring they are promptly seated at the most suitable table. This enables smooth customer service that takes emotions into consideration.

[0721] (Example 2)

[0722] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0723] In modern restaurants, the decline in customer satisfaction due to long wait times is a significant problem. In particular, when customer emotional states are not considered, the overall quality of service deteriorates, impacting the restaurant's reputation. Traditional waiting time management systems lack the ability to analyze customer emotions from facial expressions and voice, making it difficult to provide more personalized service. Therefore, there is a need for effective waiting time management and seating guidance that takes customer emotions into account.

[0724] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0725] In this invention, the server includes means for providing an input device for collecting customer facial expressions and voice data, processing means for analyzing the collected facial expressions and voice data and inferring the customer's emotional state, and means for generating service instructions for the customer based on the analyzed emotional state and transmitting them to a staff terminal. This enables quick and effective waiting time management and seating guidance that takes into account the customer's emotional state.

[0726] "Customer facial expression and voice data" refers to information related to a customer's facial expressions and voice, and is used to infer the customer's emotional state by analyzing this data.

[0727] An "input device" is a device, such as a camera or microphone, installed in a restaurant to collect facial expressions and voice data from customers.

[0728] A "processing device" is a computer system that analyzes collected facial and voice data to infer the customer's emotional state.

[0729] "Emotional state" refers to the psychological and emotional condition exhibited by a customer, as inferred from their facial expressions and voice.

[0730] "Generating service instructions" means determining specific actions and services that restaurant staff should take based on the customer's emotional state, and creating instructions to display on a terminal.

[0731] A "staff terminal" is a device used by restaurant staff to receive and display information transmitted from the server.

[0732] A description of embodiments for carrying out this invention will be given.

[0733] The server collects customer facial expressions and audio data through cameras and microphones installed in the restaurant. Video data acquired from the cameras is processed in real time using an image processing library to extract facial features and infer the customer's emotional state. Audio data is transcribed using speech analysis software, and features such as voice tone and speaking speed are analyzed. Based on this, the server infers the customer's overall emotional state and stores it in a database.

[0734] The terminal receives customer emotional states transmitted from the server and displays appropriate service instructions on the screen for store staff. The terminal is notified of the optimal response for the customer based on the analysis results, enabling staff to quickly provide situation-appropriate service. For example, if a customer is a family and the children are bored during a long wait, the terminal will prompt staff to quickly seat them. This allows staff to effectively perform their duties to ensure customer comfort.

[0735] Users can receive notifications from the server via their smartphones regarding waiting times and estimated service times. The user application displays actionable guidelines based on the data provided by the server, enabling users to make the waiting experience less stressful. This allows users to use their time efficiently, such as returning to the restaurant at the appropriate time.

[0736] As a concrete example, an example of a prompt message is shown below.

[0737] "Please describe in detail how you analyze customers' emotions in real time while they are waiting and provide the best possible service. Include examples of how to handle situations where customers with children are bored."

[0738] As described above, this invention aims to improve the efficiency of restaurant operations and enhance customer satisfaction by providing personalized services through the analysis of customers' emotional states.

[0739] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0740] Step 1:

[0741] The server collects customer facial expressions and audio data via cameras and microphones. Camera footage and audio from within the store are acquired as input. Specifically, the server captures this data in real time, performs initial processing, and converts it into an analyzable format.

[0742] Step 2:

[0743] The server analyzes the customer's emotional state using the collected data. The input is the facial and audio data collected in step 1, and the output is the estimated result of the customer's emotional state. Specifically, the server uses an image processing library to detect facial feature points and audio processing software to analyze the tone of voice and comprehensively estimate the emotion.

[0744] Step 3:

[0745] The server sends the analyzed emotional state to the terminal and generates instructions for display to store staff. The input is the emotional state data obtained in step 2, and the output is specific service instructions for staff. Specifically, the server creates a service scenario that takes the emotional state into account and pushes it to the terminal.

[0746] Step 4:

[0747] The terminal displays instructions to staff based on the instructions it receives, supporting service delivery. The input is service instructions sent from the server, and the output is the action staff should take. Specifically, the terminal displays alerts on the screen to clearly indicate to staff which customers should be given priority.

[0748] Step 5:

[0749] Users receive notifications from the server via a smartphone app, obtaining information about waiting times and estimated service times. Input is waiting status data provided by the server, while output is recommended information regarding user actions. Specifically, the user's app screen displays notifications such as "You will be served in 5 minutes," allowing users to efficiently manage their time.

[0750] (Application Example 2)

[0751] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] Customers often experience dissatisfaction due to long waiting times at facilities. Furthermore, standard service often struggles to provide individualized care based on customer emotions, making it difficult to improve customer satisfaction. Especially for families and customers whose emotions are easily swayed, there is a need to reduce frustration and stress during waiting times and provide a comfortable experience. An efficient system is needed to improve this customer experience.

[0753] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0754] In this invention, the server includes means for analyzing customer emotions and providing personalized services based on that data, means for predicting customer dwell time and optimizing location placement based on the prediction results, and means for acquiring customer location information and providing navigation guidance at the optimal timing. This makes it possible to provide individualized services that take customer emotions into consideration and to optimize dwell time management.

[0755] "Data for predicting customer dwell time" refers to information about customer behavior and movements within the facility.

[0756] A "recognition device" is a device that analyzes acquired information and derives a specific result.

[0757] "Location information" refers to information that indicates where a customer is currently located.

[0758] "Movement start time" refers to the calculated time when the customer is scheduled to begin their next action.

[0759] "Waiting time" refers to the expected time it takes for a customer to receive a particular service.

[0760] "Placement arrangement" refers to the positioning of each area and seating area within a facility.

[0761] "Analyzing customer emotions" means inferring a customer's psychological state and feelings based on their facial expressions and voice.

[0762] "Personalized service" refers to the provision of services that are customized to the individual needs and circumstances of each customer.

[0763] This system is designed to improve customer service in restaurants and other establishments, and primarily consists of three elements: servers, terminals, and users.

[0764] The server uses cameras and microphones installed within the facility to collect customer facial expressions and audio data. This data is processed by emotion analysis AI (Microsoft Azure Emotion API being a specific example) to analyze the customer's emotional state in real time. This allows the server to understand what emotions the customer is experiencing.

[0765] Subsequently, the terminal receives emotion data from the server and wait time information calculated based on the customer's predicted stay duration. The terminal then uses this information to send instructions to facility staff. For example, if a customer is deemed unhappy, it can be instructed to prioritize seating them. The terminal also has the function of optimizing guest flow and providing personalized entertainment and information during waiting times.

[0766] Users can receive information from their devices, reduce their own stress, and choose the optimal course of action. These notifications are tailored to the user's emotions, allowing for more flexible responses.

[0767] As a concrete example, when visiting a restaurant with family, robots are used to make waiting time more enjoyable by displaying animations on smartphones to keep children entertained. Another example of a prompt message is, "Please think of actions for a robot that analyzes the customer's voice and facial expressions and proposes the most appropriate service based on those emotions."

[0768] Thus, this invention aims to enhance the waiting experience within a facility and increase customer satisfaction by utilizing emotion analysis technology to provide personalized services to customers.

[0769] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0770] Step 1:

[0771] The server acquires real-time data from cameras and microphones installed within the facility. This data includes customer facial expressions and voice. Image and audio data are provided as input, forming the basis for subsequent processing.

[0772] Step 2:

[0773] The server uses the acquired data to analyze customer emotions using an emotion analysis AI model (e.g., Microsoft Azure Emotion API). This process extracts facial and vocal characteristics and outputs emotional states such as joy and anger. This output is a different emotion label for each customer.

[0774] Step 3:

[0775] The server sends the predicted dwell time and customer emotional state to the terminal based on the emotion analysis results. Here, the predicted emotional state and dwell time are input, and appropriate action instructions are output to each terminal accordingly.

[0776] Step 4:

[0777] The terminal uses emotional state and dwell time information received from the server to issue instructions to staff. These instructions may include, for example, promptly seating an unhappy customer or providing entertainment. Here, the input is emotional and time information, and the output is specific action instructions.

[0778] Step 5:

[0779] Based on information provided by the device, users adjust their travel timing and how they spend their waiting time. In this step, information that responds to emotions as input leads to the output of optimized customer behavior.

[0780] Step 6:

[0781] The server collects and stores data on each customer's emotional state and dwell time, and uses this data for reinforcement learning of the model. This aims to improve the accuracy of predictions and the quality of service. The input is the customer data history, and the output is the improved accuracy of emotion prediction.

[0782] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0783] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0784] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0785] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0786] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0787] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0788] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0789] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0790] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0791] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0792] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0793] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0794] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0795] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0796] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0797] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0798] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0799] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0800] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0801] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0802] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0803] The following is further disclosed regarding the embodiments described above.

[0804] (Claim 1)

[0805] A means of obtaining image data to predict customer dwell time,

[0806] An image recognition device means that predicts the customer's dwell time based on acquired image data,

[0807] A means for obtaining the customer's location information and calculating the start time of travel,

[0808] Means for notifying the customer of the calculated departure time and estimated waiting time,

[0809] A means for optimizing seating arrangements based on predicted dwell time,

[0810] A system that includes this.

[0811] (Claim 2)

[0812] To improve the accuracy of predicting dwell time, we will perform reinforcement learning using historical data.

[0813] The system according to claim 1.

[0814] (Claim 3)

[0815] When a customer arrives at the store, provide staff with information to help them find the best seat.

[0816] The system according to claim 1.

[0817] "Example 1"

[0818] (Claim 1)

[0819] A means of acquiring image data using a sensing device for acquiring customer movement data,

[0820] An image analysis unit means for analyzing acquired image data and predicting customer dwell time,

[0821] A means for obtaining customer geographical information and calculating the optimal start time for travel,

[0822] Means for notifying the customer of the calculated departure time and predicted waiting time via a communication device,

[0823] A means for optimizing seat allocation based on predicted dwell time,

[0824] A system that includes this.

[0825] (Claim 2)

[0826] The system according to claim 1, which performs reinforcement learning using historical information in order to improve the accuracy of predicting the length of stay.

[0827] (Claim 3)

[0828] The system according to claim 1, which provides staff with information to guide customers to the most suitable seat when they arrive at the facility.

[0829] "Application Example 1"

[0830] (Claim 1)

[0831] A means of obtaining electronic information to predict customer dwell time,

[0832] A visual recognition device means that predicts the customer's dwell time based on acquired electronic information,

[0833] A means for obtaining the customer's location information and calculating the start time of travel,

[0834] Means for notifying the customer of the calculated departure time and estimated waiting time,

[0835] A means for optimizing seating arrangements based on predicted dwell time,

[0836] A means of displaying the store's congestion status and customer queue in real time on an information terminal,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, which performs reinforcement learning using past data in order to improve the accuracy of predicting the length of stay.

[0840] (Claim 3)

[0841] The system according to claim 1, which provides staff with information to guide customers to the best way to reduce waiting times when they arrive at the facility.

[0842] "Example 2 of combining an emotion engine"

[0843] (Claim 1)

[0844] Means for providing an input device for collecting customer facial expressions and voice data,

[0845] A processing device that analyzes collected facial and voice data to estimate the customer's emotional state,

[0846] A means for generating service instructions for customers based on their analyzed emotional state and transmitting them to staff terminals,

[0847] A notification device that provides an optimal waiting time experience based on the customer's emotional state and departure time,

[0848] A system that includes this.

[0849] (Claim 2)

[0850] To improve the accuracy of predicting customer emotional states and dwell time, we use historical data and reinforcement learning models.

[0851] The system according to claim 1.

[0852] (Claim 3)

[0853] An information system that provides staff with optimal guidance to ensure customers are seated comfortably, based on their emotional state.

[0854] The system according to claim 1.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] A means of obtaining data to predict customer dwell time,

[0858] A recognition device means that predicts the customer's dwell time based on the acquired data,

[0859] A means for obtaining the customer's location information and calculating the start time of travel,

[0860] Means for notifying the customer of the calculated departure time and estimated waiting time,

[0861] A means for optimizing the placement of locations based on predicted dwell time,

[0862] A means of analyzing customer emotions and providing personalized services based on that data,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, which performs reinforcement learning using past data in order to improve the accuracy of the service provided by predicting the length of stay and analyzing sentiment.

[0866] (Claim 3)

[0867] The system according to claim 1, which provides a person with information to guide them to the optimal location when a customer arrives at a facility. [Explanation of symbols]

[0868] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of obtaining image data to predict customer dwell time, An image recognition device means that predicts the customer's dwell time based on acquired image data, A means for obtaining the customer's location information and calculating the start time of travel, Means for notifying the customer of the calculated departure time and estimated waiting time, A means for optimizing seating arrangements based on predicted dwell time, A system that includes this.

2. To improve the accuracy of predicting dwell time, we will perform reinforcement learning using historical data. The system according to claim 1.

3. When a customer arrives at the store, provide staff with information to help them find the best seat. The system according to claim 1.

Images