System

Abstract

To provide a system that estimates the emotions of a customer in a restaurant and handles an order accordingly.SOLUTION: A system according to an embodiment comprises an acquisition unit, an estimation unit, a determination unit, and a handling unit. The acquisition unit acquires emotion estimation information for estimating the emotions of a customer in a restaurant. The estimation unit estimates the emotions of the customer on the basis of the emotion estimation information acquired by the acquisition unit. The determination unit determines how to handle an order according to the emotions of the customer as estimated by the estimation unit. The handling unit handles the order in a way determined by the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, it was difficult to accurately grasp the emotions of customers in a restaurant and handle orders accordingly.

[0005] The system according to the embodiment aims to estimate the emotions of customers in a restaurant and handle orders accordingly. [Means for solving the problem]

[0006] The system according to the embodiment includes an acquisition unit, an estimation unit, a determination unit, and a response unit. The acquisition unit acquires emotion estimation information for estimating emotions of customers in a restaurant. The estimation unit estimates emotions of the customers based on the emotion estimation information acquired by the acquisition unit. The determination unit determines how to handle an order according to the emotion of the customers estimated by the estimation unit. The response unit performs order handling determined by the determination unit. Effect of the Invention

[0007] The system according to the embodiment can estimate the emotions of customers in a restaurant and handle orders accordingly. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The order robot system according to the embodiment of the present invention is a system that reads the emotions of customers and responds appropriately to orders. This order robot system realizes smoother order acceptance by reading the emotions of customers and providing appropriate methods of taking orders and recommendations. For example, the order robot system acquires information such as the customer's facial expressions, movements, voice, heart rate, blood pressure, pupil response, brain waves, body temperature, breathing, and posted content, and estimates the emotions of the customer based on this information. Depending on the estimated emotions, the order robot system determines how to take orders and suggests recommendations. For example, if the customer is relaxed, it suggests dishes and drinks that suit the relaxed atmosphere. Conversely, if the customer is in a hurry, it suggests menus that can be provided quickly. Furthermore, the order robot system determines the order suggestion order ... For example, if the surrounding customers are quietly enjoying their meals, the order robot will respond to the order in a way that suits the quiet atmosphere. Finally, the order robot system predicts the order content based on the order response that has been determined, realizing efficient order acceptance. In this way, the order robot system can read the customer's emotions and respond appropriately to the order, realizing smoother order acceptance.

[0029] The order robot system according to the embodiment includes an acquisition unit, an estimation unit, a decision unit, and a response unit. The acquisition unit acquires emotion estimation information for estimating the emotion of a customer. The emotion estimation information includes, for example, the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content, but is not limited to such examples. The acquisition unit, for example, captures the customer's facial expression using a camera, and an AI analyzes the facial expression to estimate the emotion. The acquisition unit can also capture the customer's voice using a microphone, and an AI analyzes the voice to estimate the emotion. Furthermore, the acquisition unit can capture the customer's heart rate using a heart rate sensor, and an AI analyzes the heart rate to estimate the emotion. The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit, for example, uses an AI to analyze the acquired facial expression information, voice information, heart rate information, and the like, and estimates the customer's emotion. The estimation unit, for example, uses a facial expression recognition algorithm to estimate the emotion from the customer's facial expression. The estimation unit can also estimate emotions from the customer's voice using a voice analysis algorithm. The estimation unit can also estimate emotions from the customer's heartbeat using a heartbeat analysis algorithm. The determination unit determines how to handle the order according to the customer's emotions estimated by the estimation unit. The determination unit determines how to take the order and make a recommendation based on the estimated emotions, for example, using AI. For example, when the customer is relaxed, the determination unit proposes dishes and drinks that suit the relaxed atmosphere. Furthermore, when the customer is in a hurry, the determination unit can also propose a menu that can be provided quickly. Furthermore, the determination unit can determine a proposal order according to the customer's emotions and propose the order items in that order. The response unit handles the order determined by the determination unit. For example, the response unit uses AI to make an order taking method and a recommendation suggestion according to the customer's emotions. For example, when the customer is relaxed, the response unit makes an order taking method and a recommendation suggestion that suits the relaxed atmosphere. Furthermore, when the customer is in a hurry, the response unit can also propose a menu that can be provided quickly. Furthermore, the response unit can explain the menu according to the customer's emotions.For example, the robot can encourage customers to place orders by providing detailed explanations about dishes that the customer is interested in. This allows the ordering robot system according to the embodiment to read the customer's emotions and handle the order appropriately.

[0030] The acquisition unit acquires emotion estimation information for estimating the emotion of the customer. The emotion estimation information includes, for example, the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content, but is not limited to such examples. For example, the acquisition unit captures the customer's facial expression using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. The acquisition unit can also capture the customer's voice using a microphone, and the AI ​​analyzes the voice to estimate the emotion. Furthermore, the acquisition unit can capture the customer's heart rate using a heart rate sensor, and the AI ​​analyzes the heart rate to estimate the emotion. Specifically, the camera captures the customer's face with high resolution and detects subtle changes in the facial expression using face recognition technology. For example, facial expressions such as smile, surprise, anger, and sadness are analyzed in real time to estimate the emotional state of the customer. The microphone analyzes the tone, pitch, speed, volume, etc. of the customer's voice to detect changes in emotion. For example, a pattern such as a higher voice indicates excitement and a lower voice indicates calmness is identified. The heart rate sensor monitors the customer's heart rate fluctuations in real time and estimates their state of stress or relaxation. This allows the acquisition unit to collect multifaceted data and provide basic information for estimating the customer's emotions with high accuracy. Furthermore, the acquisition unit centrally manages this data and transmits it to the analysis unit in real time, enabling rapid emotion estimation.

[0031] The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit, for example, uses AI to analyze the acquired facial expression information, voice information, heart rate information, etc., and estimates the customer's emotion. The estimation unit, for example, uses a facial expression recognition algorithm to estimate the emotion from the customer's facial expression. The estimation unit can also use a voice analysis algorithm to estimate the emotion from the customer's voice. Furthermore, the estimation unit can also use a heart rate analysis algorithm to estimate the emotion from the customer's heart rate. Specifically, the AI ​​uses deep learning technology to utilize a model learned from a huge data set. For example, the facial expression recognition algorithm extracts facial feature points and analyzes their positional relationship and changes to estimate the emotion. The voice analysis algorithm converts the voice waveform into a spectrogram, extracts the voice features, and estimates the emotion. The heart rate analysis algorithm analyzes the heart rate fluctuation pattern and estimates the stress or relaxation state. As a result, the estimation unit can integrate multiple emotion estimation information to estimate the customer's emotion with high accuracy. Furthermore, the estimation unit can improve the accuracy of emotion estimation by utilizing past data and customer history information. For example, the estimation unit can refer to the emotional state of a particular customer in the past and reflect it in the current emotion estimation. This allows the estimation unit to grasp the customer's emotions more accurately and use it for the next step, which is handling the order.

[0032] The determination unit determines how to handle an order according to the customer's emotions estimated by the estimation unit. The determination unit uses, for example, AI to determine how to take an order and recommend suggestions based on the estimated emotions. For example, if the customer is relaxed, the determination unit suggests dishes and drinks that fit the relaxed atmosphere. In addition, if the customer is in a hurry, the determination unit can also suggest menus that can be served quickly. Furthermore, the determination unit can determine a suggestion order according to the customer's emotions and suggest order items in that order. Specifically, the AI ​​uses a rule-based system or machine learning model to select optimal menus and services based on the customer's emotional state. For example, if the customer is relaxed, herbal tea or light meals with a relaxing effect are suggested. If the customer is in a hurry, menus with short cooking times or items that can be taken out are suggested. Furthermore, the determination unit can make personalized suggestions taking into account the customer's past order history and preferences. For example, for a customer who has preferred to order a particular dish in the past, a new menu related to that dish is suggested. In this way, the determination unit can determine optimal order responses according to the customer's emotions and preferences, thereby improving customer satisfaction. Furthermore, the decision unit can dynamically adjust order handling based on emotion information updated in real time. For example, if the customer's emotion changes, the proposal content is immediately revised in response to the change. This allows the decision unit to always respond flexibly based on the latest information and quickly meet customer needs.

[0033] The response unit handles the order determined by the determination unit. The response unit, for example, uses AI to take orders and make recommendations according to the customer's emotions. For example, if the customer is relaxed, the response unit takes orders and makes recommendations that match the relaxed atmosphere. In addition, if the customer is in a hurry, the response unit can also suggest menus that can be served quickly. Furthermore, the response unit can explain the menu according to the customer's emotions. For example, the customer is encouraged to order by providing detailed explanations about dishes that interest them. Specifically, the AI ​​uses natural language processing technology to smoothly communicate with the customer. For example, if the customer is relaxed, the AI ​​speaks to the customer in a gentle tone and suggests a menu that has a relaxing effect. If the customer is in a hurry, the AI ​​introduces the menu quickly and concisely and accepts the order immediately. Furthermore, the response unit collects customer feedback in real time and uses it to improve the accuracy of order handling. For example, the AI ​​records how the customer responded to the proposed menu and reflects it in the next proposal. This allows the response unit to provide optimal order handling according to the customer's emotions and needs, thereby increasing customer satisfaction. Furthermore, the customer service department can reliably convey information using multiple communication methods. For example, the department can provide menu information and suggestions to customers not only through face-to-face conversations but also through smartphone apps and tablet devices. This allows the customer service department to quickly and reliably handle orders and improve the customer experience.

[0034] The determination unit determines an order taking method or a recommendation based on the customer's emotions, and the response unit can make an order taking method or a recommendation based on the customer's emotions. For example, when a customer is relaxed, the determination unit determines an order taking method or a recommendation that suits the relaxed atmosphere. For example, the determination unit suggests food and drinks that suit the relaxed atmosphere. Also, when a customer is in a hurry, the determination unit can also suggest a menu that can be provided quickly. For example, the determination unit suggests food and drinks that can be provided quickly. The response unit makes the order taking method or recommendation determined by the determination unit. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Also, the response unit can suggest a menu that can be provided quickly. In this way, customer satisfaction can be improved by making an order taking method or recommendation suggestions according to the customer's emotions.

[0035] The determination unit determines a proposal order of the order objects based on the customer's emotions, and the response unit can propose the order objects in the proposal order based on the customer's emotions. For example, when the customer is relaxed, the determination unit determines a proposal order suitable for the relaxed state. For example, the determination unit prioritizes proposal of dishes and drinks that suit a relaxed atmosphere. Also, when the customer is in a hurry, the determination unit can also prioritize proposal of menus that can be provided quickly. For example, the determination unit prioritizes proposal of dishes and drinks that can be provided quickly. The response unit proposes the order objects in the proposal order determined by the determination unit. For example, the response unit prioritizes proposal of dishes and drinks that suit a relaxed atmosphere. Also, the response unit can also prioritize proposal of menus that can be provided quickly. In this way, by proposing the order objects in a proposal order according to the customer's emotions, it is possible to provide a service that meets the needs of the customer.

[0036] The response unit can explain the menu according to the customer's emotions. The response unit, for example, encourages ordering by providing a detailed explanation of the dishes that the customer is interested in. For example, the response unit provides a detailed explanation of the ingredients, cooking methods, allergy information, etc. of the dishes. The response unit can also suggest recommended features of the dishes and how to eat them according to the customer's emotions. For example, the response unit explains the recommended features of the dishes to attract the customer's interest. The response unit can also suggest how to eat the dishes to improve customer satisfaction. In this way, by explaining the menu according to the customer's emotions, it is possible to deepen the customer's understanding and encourage ordering.

[0037] The acquisition unit acquires allergy information of the customer, and the response unit can propose a menu that accommodates the allergy based on the information acquired by the acquisition unit. The acquisition unit, for example, acquires the allergy information of the customer. The allergy information includes, for example, nut allergy, dairy allergy, and the like, but is not limited to such examples. The acquisition unit, for example, acquires allergy information input by the customer. The acquisition unit can also estimate the allergy information from the customer's past order history. The response unit proposes a menu that accommodates the allergy based on the allergy information acquired by the acquisition unit. For example, the response unit proposes a dish that does not contain nuts to a customer who has a nut allergy. The response unit can also propose a dish that does not contain dairy products to a customer who has a dairy allergy. This allows the customer to enjoy a meal with peace of mind by proposing a menu based on the customer's allergy information.

[0038] The determination unit determines an order response based on the emotion of the customer and the emotion of other customers around the customer, and the response unit can respond to the order to the customer based on the emotion of the customer and the emotion of other customers around the customer. The determination unit, for example, determines an order response that suits the overall atmosphere based on the emotion of the customer and the emotion of other customers around the customer. For example, the determination unit determines an order response that suits the quiet atmosphere when surrounding customers are quietly enjoying their meals. Also, the determination unit can determine an order response that suits the lively atmosphere when surrounding customers are lively enjoying their meals. The response unit performs the order response determined by the determination unit. For example, the response unit performs an order response that suits the quiet atmosphere. Also, the response unit can perform an order response that suits the lively atmosphere. In this way, by performing an order response according to the emotion of the customer and other customers around the customer, a service that suits the overall atmosphere can be provided.

[0039] The corresponding unit can display the menu on the digital signage in a manner according to the emotions of the customer and the emotions of other customers around the customer. The corresponding unit adjusts the way the menu is displayed based on, for example, the emotions of the customer and the emotions of the other customers around the customer. For example, the corresponding unit displays a menu that suits a relaxed atmosphere. The corresponding unit can also display a menu that suits a lively atmosphere. Furthermore, the corresponding unit can customize the display content of the menu according to the emotions of the customer. For example, the corresponding unit displays dishes that the customer is interested in with emphasis. The corresponding unit can also display a menu that accommodates allergies based on the allergy information of the customer. In this way, by displaying a menu according to the emotions of the customer and the other customers around the customer, it is possible to provide visually appropriate information.

[0040] The restaurant may include a prediction unit that predicts the order contents based on the order handling decision content by the decision unit. The prediction unit predicts a menu that the customer is likely to order based on the order handling decision content by the decision unit, for example. For example, the prediction unit predicts an optimal menu in consideration of the customer's past order history and current emotional state. The prediction unit can also predict the order contents based on the customer's preferences and allergy information. Furthermore, the prediction unit can adjust the prediction accuracy of the order contents according to the customer's emotions. For example, the prediction unit predicts a menu that suits a relaxed atmosphere for a customer who is in a relaxed state. The prediction unit can also predict a menu that can be quickly provided to a customer who is in a hurry. This makes it possible to efficiently accept orders by predicting the order contents based on the order handling decision content.

[0041] The order robot can function as an order robot having an acquisition unit, an estimation unit, a determination unit, and a response unit. The order robot is used, for example, to automate and streamline order reception in a restaurant. The order robot acquires customer emotion estimation information using the acquisition unit, and estimates the customer emotion using the estimation unit. Next, the order robot determines an order response based on the emotion estimated using the determination unit, and executes the determined order response using the response unit. This allows the order robot to read the customer's emotion and perform an appropriate order response. For example, the order robot captures the customer's facial expression with a camera, and the AI ​​analyzes the facial expression to estimate the emotion. The order robot can also capture the customer's voice with a microphone, and the AI ​​can analyze the voice to estimate the emotion. Furthermore, the order robot can capture the customer's heartbeat with a heartbeat sensor, and the AI ​​can analyze the heartbeat to estimate the emotion. This allows the order robot to read the customer's emotion and perform an appropriate order response, thereby automating and streamlining order reception in a restaurant.

[0042] The acquisition unit can acquire one or more pieces of information among the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content as emotion estimation information. The acquisition unit, for example, captures the customer's facial expression using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. For example, the acquisition unit captures the customer's facial expression while looking at a menu, and the AI ​​analyzes the facial expression to estimate the interest level. The acquisition unit can also periodically capture the customer's facial expression while waiting for an order, and the AI ​​can analyze the change to estimate the emotion change. Furthermore, the acquisition unit can capture the customer's facial expression while eating food, and the AI ​​can analyze the facial expression to estimate the satisfaction level. In this way, the emotion of the customer can be more accurately estimated by acquiring various information as emotion estimation information.

[0043] The acquisition unit can acquire the facial expression of the customer and use it as emotion estimation information. For example, the acquisition unit captures the facial expression of the customer using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. For example, the acquisition unit captures the facial expression of the customer while looking at a menu, and the AI ​​analyzes the facial expression to estimate the interest level. The acquisition unit can also periodically capture the facial expression of the customer while waiting for the order, and the AI ​​can analyze the change to estimate the emotion change. Furthermore, the acquisition unit can capture the facial expression of the customer while eating food, and the AI ​​can analyze the facial expression to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the facial expression of the customer.

[0044] The acquisition unit can acquire the customer's behavior and use it as emotion estimation information. For example, the acquisition unit can capture the customer's behavior using a camera, and the AI ​​can analyze the behavior to estimate the emotion. For example, the acquisition unit can capture the customer's behavior of pointing at a menu, and the AI ​​can analyze the behavior to estimate the dish the customer is interested in. The acquisition unit can also capture the hand movements of the customer while waiting for the order, and the AI ​​can analyze the movements to estimate the customer's impatience or anxiety. Furthermore, the acquisition unit can capture the hand movements of the customer while eating the food, and the AI ​​can analyze the movements to estimate the customer's satisfaction. In this way, the accuracy of emotion estimation is improved by acquiring the customer's behavior.

[0045] The acquisition unit can acquire the voice of the customer and use it as emotion estimation information. For example, the acquisition unit can capture the voice of the customer using a microphone, and the AI ​​can analyze the voice to estimate the emotion. For example, the acquisition unit can capture the tone of the voice when the customer places an order, and the AI ​​can analyze the tone to estimate the emotion. The acquisition unit can also capture the reaction voice of the customer when listening to an explanation of the dish, and the AI ​​can analyze the reaction to estimate the interest level. Furthermore, the acquisition unit can capture the customer's impressions while eating the dish, and the AI ​​can analyze the impressions to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's voice.

[0046] The acquisition unit can acquire the heart rate of the customer and use it as emotion estimation information. For example, the acquisition unit can capture the heart rate of the customer using a heart rate sensor, and the AI ​​can analyze the heart rate to estimate the emotion. For example, the acquisition unit can capture the heart rate of the customer while looking at a menu, and the AI ​​can analyze the heart rate to estimate the excitement level. The acquisition unit can also capture the heart rate of the customer while waiting for an order, and the AI ​​can analyze the heart rate to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the heart rate of the customer while eating food, and the AI ​​can analyze the heart rate to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's heart rate.

[0047] The acquisition unit can acquire the customer's blood pressure and use it as emotion estimation information. The acquisition unit, for example, captures the customer's blood pressure using a blood pressure sensor, and the AI ​​analyzes the blood pressure to estimate the emotion. For example, the acquisition unit captures the customer's blood pressure when looking at a menu, and the AI ​​analyzes the blood pressure to estimate the excitement level. The acquisition unit can also capture the customer's blood pressure while waiting to order, and the AI ​​can analyze the blood pressure to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the customer's blood pressure when eating food, and the AI ​​can analyze the blood pressure to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's blood pressure.

[0048] The acquisition unit can acquire the pupillary response of the customer and use it as emotion estimation information. The acquisition unit, for example, uses a pupil sensor to capture the customer's pupillary response, and the AI ​​analyzes the pupillary response to estimate the emotion. For example, the acquisition unit captures the pupillary response when the customer is looking at a menu, and the AI ​​analyzes the response to estimate the excitement level. The acquisition unit can also capture the pupillary response while the customer is waiting for an order, and the AI ​​can analyze the response to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the pupillary response when the customer is eating food, and the AI ​​can analyze the response to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's pupillary response.

[0049] The estimation unit can estimate the emotion of the customer and adjust the estimation algorithm based on the estimated emotion of the customer. For example, when the customer is relaxed, the estimation unit applies an estimation algorithm suitable for the relaxed state. For example, the estimation unit uses an emotion estimation algorithm suitable for the relaxed state. Furthermore, when the customer is excited, the estimation unit can also apply an estimation algorithm suitable for the excited state. For example, the estimation unit uses an emotion estimation algorithm suitable for the excited state. Furthermore, when the customer is impatient, the estimation unit can also apply an estimation algorithm suitable for the impatient state. For example, the estimation unit uses an emotion estimation algorithm suitable for the impatient state. In this way, by adjusting the estimation algorithm based on the emotion of the customer, the accuracy of emotion estimation is improved.

[0050] The estimation unit can adjust the level of detail of the estimation based on the importance of the emotion estimation information at the time of estimation. For example, when facial expression information of a customer is important, the estimation unit performs detailed emotion estimation based on the facial expression information. For example, the estimation unit prioritizes analysis of the facial expression information and performs detailed emotion estimation. Furthermore, when voice information of a customer is important, the estimation unit can also perform detailed emotion estimation based on voice information. For example, the estimation unit prioritizes analysis of the voice information and performs detailed emotion estimation. Furthermore, when heart rate information of a customer is important, the estimation unit can also perform detailed emotion estimation based on the heart rate information. For example, the estimation unit prioritizes analysis of the heart rate information and performs detailed emotion estimation. Thereby, by adjusting the level of detail of the estimation based on the importance of the emotion estimation information, the accuracy of emotion estimation is improved.

[0051] The estimation unit can apply different estimation algorithms depending on the category of the emotion estimation information at the time of estimation. For example, the estimation unit applies a facial expression recognition algorithm to emotion estimation based on facial expression information. For example, the estimation unit uses the facial expression recognition algorithm to estimate an emotion from a customer's facial expression. The estimation unit can also apply a voice analysis algorithm to emotion estimation based on voice information. For example, the estimation unit uses a voice analysis algorithm to estimate an emotion from a customer's voice. The estimation unit can also apply a heartbeat analysis algorithm to emotion estimation based on heartbeat information. For example, the estimation unit uses a heartbeat analysis algorithm to estimate an emotion from a customer's heartbeat. Thereby, the accuracy of emotion estimation is improved by applying an optimal estimation algorithm depending on the category of the emotion estimation information.

[0052] The estimation unit can estimate the customer's emotions and adjust the display method of the estimation result based on the estimated customer's emotions. For example, when the customer is relaxed, the estimation unit provides a display method suitable for the relaxed state. For example, the estimation unit performs a graph display or text display suitable for the relaxed state. Furthermore, when the customer is excited, the estimation unit can also provide a display method suitable for the excited state. For example, the estimation unit performs a graph display or text display suitable for the excited state. Furthermore, when the customer is impatient, the estimation unit can also provide a display method suitable for the impatient state. For example, the estimation unit performs a graph display or text display suitable for the impatient state. This makes it possible to provide more appropriate information by adjusting the display method of the estimation result based on the customer's emotions.

[0053] The estimation unit can determine the priority of estimation based on the acquisition time of the emotion estimation information at the time of estimation. The estimation unit, for example, prioritizes the latest emotion estimation information to perform estimation. For example, the estimation unit prioritizes the latest facial expression information and voice information to estimate an emotion. The estimation unit can also perform estimation with reference to past emotion estimation information. For example, the estimation unit analyzes with reference to past facial expression information and voice information to estimate an emotion. Furthermore, the estimation unit can also perform estimation with priority on emotion estimation information acquired in a specific time period. For example, the estimation unit prioritizes the analysis of facial expression information and voice information acquired in a specific time period to estimate an emotion. Thereby, by determining the priority of estimation based on the acquisition time of the emotion estimation information, it is possible to prioritize the latest information for estimation.

[0054] The estimation unit can adjust the order of estimation based on the relevance of the emotion estimation information at the time of estimation. For example, when facial expression information of a customer is highly relevant, the estimation unit prioritizes facial expression information in the estimation. For example, the estimation unit prioritizes the analysis of facial expression information and estimates an emotion. Furthermore, when voice information of a customer is highly relevant, the estimation unit can also prioritize voice information in the estimation. For example, the estimation unit prioritizes the analysis of voice information and estimates an emotion. Furthermore, when heart rate information of a customer is highly relevant, the estimation unit can also prioritize heart rate information in the estimation. For example, the estimation unit prioritizes the analysis of heart rate information and estimates an emotion. Thereby, by adjusting the order of estimation based on the relevance of the emotion estimation information, it is possible to prioritize information with higher relevance in the estimation.

[0055] The determination unit can estimate the customer's emotions and adjust the order handling decision method based on the estimated customer's emotions. For example, when the customer is relaxed, the determination unit determines an order handling appropriate for the relaxed state. For example, the determination unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the determination unit can also determine an order handling appropriate for the excited state. For example, the determination unit suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the determination unit can also determine an order handling appropriate for the impatient state. For example, the determination unit suggests a menu that can be provided quickly. In this way, by adjusting the order handling decision method based on the customer's emotions, more appropriate order handling is possible.

[0056] When making a decision, the decision unit can analyze the customer's past order history and select the optimal order response. The decision unit, for example, selects the optimal order response by referring to dishes that the customer has ordered in the past. For example, the decision unit suggests a specific dish from the customer's past order history. The decision unit can also analyze the customer's past order history and select an order response that suits the customer's preferences. For example, the decision unit suggests dishes and drinks that suit the customer's preferences. In this way, by analyzing the customer's past order history, it is possible to select the optimal order response that suits the customer's preferences.

[0057] At the time of decision, the decision unit can customize the means for handling the order based on the customer's current situation. For example, if the customer is in a hurry, the decision unit suggests a menu that can be provided quickly. For example, the decision unit suggests food and drinks that can be provided quickly. Moreover, if the customer is relaxed, the decision unit can also suggest food and drinks that suit the relaxed atmosphere. For example, the decision unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, if the customer has a specific allergy, the decision unit can also suggest a menu that accommodates the allergy. For example, the decision unit suggests food and drinks that accommodate the allergy. This allows for more appropriate order handling by customizing the means for handling the order based on the customer's current situation.

[0058] The determination unit can estimate the emotion of the customer and determine the order of suggestions of the order items based on the estimated emotion of the customer. For example, when the customer is relaxed, the determination unit determines the order of suggestions suitable for the relaxed state. For example, the determination unit prioritizes suggestions of dishes and drinks that suit a relaxed atmosphere. Furthermore, when the customer is excited, the determination unit can also determine the order of suggestions suitable for the excited state. For example, the determination unit prioritizes suggestions of dishes and drinks that suit an excited atmosphere. Furthermore, when the customer is impatient, the determination unit can also determine the order of suggestions suitable for the impatient state. For example, the determination unit prioritizes suggestions of menu items that can be provided quickly. In this way, by determining the order of suggestions of the order items based on the emotion of the customer, more appropriate suggestions are possible.

[0059] When making a decision, the determination unit can determine the suggestion order by referring to the customer's past order history. For example, the determination unit prioritizes and suggests specific dishes from the customer's past order history. The determination unit can also analyze the customer's past order history and determine the suggestion order that suits the customer's preferences. For example, the determination unit prioritizes and suggests dishes and drinks that suit the customer's preferences. In this way, by referring to the customer's past order history, the suggestion order that suits the customer's preferences can be determined.

[0060] At the time of determination, the determination unit can customize the proposal order based on the current situation of the customer. For example, when the customer is in a hurry, the determination unit prioritizes and proposes a menu that can be provided quickly. For example, the determination unit prioritizes and proposes food and drinks that can be provided quickly. In addition, when the customer is relaxed, the determination unit can also prioritize and propose food and drinks that suit the relaxed atmosphere. For example, the determination unit prioritizes and proposes food and drinks that suit the relaxed atmosphere. Furthermore, when the customer has a specific allergy, the determination unit can also prioritize and propose a menu that corresponds to the allergy. For example, the determination unit prioritizes and proposes food and drinks that correspond to the allergy. In this way, by customizing the proposal order based on the current situation of the customer, more appropriate proposals can be made.

[0061] The response unit can estimate the customer's emotions and adjust the method of handling the order based on the estimated customer's emotions. For example, when the customer is relaxed, the response unit handles the order in a way that suits the relaxed atmosphere. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the response unit can also handle the order in a way that suits the excited atmosphere. For example, the response unit suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the response unit can also handle the order quickly. For example, the response unit suggests a menu that can be provided quickly. This makes it possible to handle the order more appropriately by adjusting the method of handling the order based on the customer's emotions.

[0062] When responding, the response unit can select the optimal response method by referring to the customer's past order history. For example, the response unit can suggest a specific dish from the customer's past order history. The response unit can also analyze the customer's past order history and select a response method that suits the customer's preferences. For example, the response unit can suggest a dish or drink that suits the customer's preferences. In this way, by referring to the customer's past order history, the optimal response method that suits the customer's preferences can be selected.

[0063] The response unit can customize the means of response based on the customer's current situation when responding. For example, if the customer is in a hurry, the response unit suggests a menu that can be provided quickly. For example, the response unit suggests food and drinks that can be provided quickly. Furthermore, if the customer is relaxed, the response unit can also suggest food and drinks that suit the relaxed atmosphere. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, if the customer has a specific allergy, the response unit can also suggest a menu that accommodates the allergy. For example, the response unit suggests food and drinks that accommodate the allergy. This makes it possible to provide a more appropriate response by customizing the means of response based on the customer's current situation.

[0064] The response unit can estimate the emotion of the customer and determine the priority of the response based on the estimated emotion of the customer. For example, when the customer is relaxed, the response unit prioritizes a response suitable for the relaxed state. For example, the response unit prioritizes and suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the response unit can also prioritize a response suitable for the excited state. For example, the response unit prioritizes and suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the response unit can also prioritize a response suitable for the impatient state. For example, the response unit prioritizes and suggests a menu that can be provided quickly. In this way, by determining the priority of the response based on the emotion of the customer, more appropriate response is possible.

[0065] When responding, the response unit can select the optimal response method by taking into consideration the geographical location information of the customer. For example, when the customer is in a specific area, the response unit suggests local specialties of the area. For example, the response unit prioritizes suggestions of local specialties of the specific area. Furthermore, when the customer is in a tourist spot, the response unit can also suggest dishes and drinks related to the tourist spot. For example, the response unit prioritizes suggestions of dishes and drinks related to the tourist spot. Furthermore, when the customer is participating in a specific event, the response unit can also suggest a menu related to the event. For example, the response unit prioritizes suggestions of dishes and drinks related to the specific event. In this way, a more appropriate response method can be selected by taking into consideration the geographical location information of the customer.

[0066] When responding, the response unit can analyze the social media activity of the customer and suggest a response measure. For example, if the customer mentions a specific dish on social media, the response unit suggests the dish. For example, the response unit prioritizes and suggests the dish mentioned on social media. Furthermore, if the customer is participating in a specific event on social media, the response unit can also suggest a menu related to the event. For example, the response unit prioritizes and suggests dishes and drinks related to the event mentioned on social media. Furthermore, if the customer mentions a specific allergy on social media, the response unit can also suggest a menu that accommodates the allergy. For example, the response unit prioritizes and suggests dishes and drinks that accommodate the allergy mentioned on social media. In this way, by analyzing the social media activity of the customer, it is possible to suggest a more appropriate response measure.

[0067] The prediction unit can estimate the customer's emotions and adjust the method of predicting the order contents based on the estimated customer's emotions. For example, when the customer is relaxed, the prediction unit predicts the order contents suitable for the relaxed state. For example, the prediction unit predicts food and drinks suitable for a relaxed atmosphere. Furthermore, when the customer is excited, the prediction unit can also predict the order contents suitable for the excited state. For example, the prediction unit predicts food and drinks suitable for an excited atmosphere. Furthermore, when the customer is impatient, the prediction unit can also predict the order contents suitable for the impatient state. For example, the prediction unit predicts a menu that can be provided quickly. This allows for more appropriate prediction by adjusting the method of predicting the order contents based on the customer's emotions.

[0068] When making a prediction, the prediction unit can select the optimal prediction method by referring to the customer's past order history. For example, the prediction unit predicts a specific dish from the customer's past order history. The prediction unit can also analyze the customer's past order history and select a prediction method that suits the customer's preferences. For example, the prediction unit predicts dishes and drinks that suit the customer's preferences. In this way, by referring to the customer's past order history, it is possible to select the optimal prediction method that suits the customer's preferences.

[0069] The prediction unit can estimate the customer's emotions and determine the priority of predictions based on the estimated customer's emotions. For example, when the customer is relaxed, the prediction unit prioritizes predictions suitable for a relaxed state. For example, the prediction unit prioritizes predictions of dishes and drinks suitable for a relaxed atmosphere. In addition, when the customer is excited, the prediction unit can also prioritize predictions suitable for an excited state. For example, the prediction unit prioritizes predictions of dishes and drinks suitable for an excited atmosphere. In addition, when the customer is impatient, the prediction unit can also prioritize predictions suitable for an impatient state. For example, the prediction unit prioritizes predictions of menus that can be provided quickly. In this way, more appropriate predictions can be made by determining the priority of predictions based on the customer's emotions.

[0070] The prediction unit can select the optimal prediction method in consideration of the geographical location information of the customer at the time of prediction. For example, when the customer is in a specific area, the prediction unit predicts the specialty products of the specific area. For example, the prediction unit predicts the specialty products of the specific area with priority. In addition, when the customer is in a tourist spot, the prediction unit can also predict dishes and drinks related to the tourist spot. For example, the prediction unit predicts dishes and drinks related to the tourist spot with priority. Furthermore, when the customer is participating in a specific event, the prediction unit can also predict the menu related to the event. For example, the prediction unit predicts dishes and drinks related to the specific event with priority. In this way, a more appropriate prediction method can be selected by considering the geographical location information of the customer.

[0071] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0072] The order robot system may further include a history analysis unit that analyzes the customer's past order history. The history analysis unit accumulates data on food and drinks that the customer has ordered in the past, and analyzes the customer's preferences and trends based on this data. For example, the history analysis unit may identify food that the customer has frequently ordered in the past, and preferentially suggest those food items. The history analysis unit may also take into consideration food that the customer has avoided in the past or allergy information, and suggest menu items that do not include these. Furthermore, the history analysis unit may suggest special menu items according to seasons or events based on the customer's past order history. This makes it possible to make more personalized suggestions by utilizing the customer's past order history, thereby improving customer satisfaction.

[0073] The order robot system may further include a location information acquisition unit that acquires real-time location information of the customer. The location information acquisition unit identifies the customer's current location and proposes menus and services according to the location. For example, when the customer is in a tourist spot, the location information acquisition unit may propose local specialties or menus that are only available in the tourist spot. In addition, when the customer is at a specific event venue, the location information acquisition unit may propose a special menu related to the event. Furthermore, when the customer is at home, the location information acquisition unit may propose a delivery service. In this way, by utilizing the customer's real-time location information, more appropriate menus and services can be provided.

[0074] The ordering robot system may further include a social media analysis unit that analyzes the social media activities of customers. The social media analysis unit collects information on food, drinks, events, etc. that customers have mentioned on social media, and makes suggestions based on this information. For example, the social media analysis unit may identify food that a customer has recently mentioned on social media, and prioritize the suggestion of that food. The social media analysis unit may also suggest a menu item related to an event that the customer is planning to attend. Furthermore, the social media analysis unit may take into account allergy information shared by the customer on social media and suggest a menu item that accommodates allergies. This allows for more personalized suggestions by utilizing the customer's social media activities.

[0075] The ordering robot system may further include a health monitoring unit that monitors the health condition of the customer. The health monitoring unit obtains health data such as the customer's heart rate, blood pressure, and body temperature in real time and makes suggestions based on this. For example, if the customer's heart rate is high, the health monitoring unit may suggest a menu with a relaxing effect. Also, if the customer's blood pressure is high, the health monitoring unit may suggest a low-salt menu. Furthermore, if the customer's body temperature is high, the health monitoring unit may suggest cold drinks or desserts. This makes it possible to provide healthier meals by making suggestions that take the customer's health condition into consideration.

[0076] The ordering robot system may further include a preference learning unit that learns the food preferences of the customer. The preference learning unit learns the customer's preferences based on data on food and drinks that the customer has ordered in the past, and makes suggestions based on this. For example, the preference learning unit may analyze trends in food that the customer has ordered in the past, and suggest new dishes based on these trends. In addition, if a customer has a preference for a particular ingredient, the preference learning unit may also suggest dishes that use that ingredient. In addition, if a customer has a preference for a particular cooking method, the preference learning unit may also suggest dishes that use that cooking method. In this way, by learning the customer's food preferences, more personalized suggestions are possible.

[0077] The process flow of the first embodiment will be briefly described below.

[0078] Step 1: The acquisition unit acquires emotion estimation information for estimating the emotion of a customer. The emotion estimation information includes the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwaves, body temperature, breathing, and posted content. For example, a camera is used to capture the customer's facial expression, and an AI analyzes the facial expression to estimate the emotion. A microphone can also be used to capture the customer's voice, and an AI can analyze the voice to estimate the emotion. A heart rate sensor can also be used to capture the customer's heart rate, and an AI can analyze the heart rate to estimate the emotion. Step 2: The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit uses AI to analyze the acquired facial expression information, voice information, heart rate information, etc., and estimates the customer's emotion. For example, the estimation unit estimates emotion from the customer's facial expression using a facial expression recognition algorithm, estimates emotion from the customer's voice using a voice analysis algorithm, and estimates emotion from the customer's heart rate using a heart rate analysis algorithm. Step 3: The decision unit decides how to handle the order according to the customer's emotions estimated by the estimation unit. The decision unit uses AI to decide how to take the order and what recommendations to make based on the estimated emotions. For example, if the customer is relaxed, it will suggest dishes and drinks that suit a relaxed atmosphere, and if the customer is in a hurry, it will suggest menu items that can be served quickly. Furthermore, it is also possible to determine the order order based on the customer's emotions and make suggestions for order items in that order. Step 4: The response unit handles the order determined by the determination unit. The response unit uses AI to take the order and make recommendations based on the customer's emotions. For example, if the customer is relaxed, the response unit will take the order in a way that suits the relaxed atmosphere and make recommendations, and if the customer is in a hurry, it will suggest a menu item that can be served quickly. In addition, the response unit can provide an explanation of the menu item based on the customer's emotions. For example, the response unit will encourage the customer to place an order by providing a detailed explanation of the dish that the customer is interested in.

[0079] (Example 2) The order robot system according to the embodiment of the present invention is a system that reads the emotions of customers and responds appropriately to orders. This order robot system realizes smoother order acceptance by reading the emotions of customers and providing appropriate methods of taking orders and recommendations. For example, the order robot system acquires information such as the customer's facial expressions, movements, voice, heart rate, blood pressure, pupil response, brain waves, body temperature, breathing, and posted content, and estimates the emotions of the customer based on this information. Depending on the estimated emotions, the order robot system determines how to take orders and suggests recommendations. For example, if the customer is relaxed, it suggests dishes and drinks that suit the relaxed atmosphere. Conversely, if the customer is in a hurry, it suggests menus that can be provided quickly. Furthermore, the order robot system determines the order suggestion order ... For example, if the surrounding customers are quietly enjoying their meals, the order robot will respond to the order in a way that suits the quiet atmosphere. Finally, the order robot system predicts the order content based on the order response that has been determined, realizing efficient order acceptance. In this way, the order robot system can read the customer's emotions and respond appropriately to the order, realizing smoother order acceptance.

[0080] The order robot system according to the embodiment includes an acquisition unit, an estimation unit, a decision unit, and a response unit. The acquisition unit acquires emotion estimation information for estimating the emotion of a customer. The emotion estimation information includes, for example, the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content, but is not limited to such examples. The acquisition unit, for example, captures the customer's facial expression using a camera, and an AI analyzes the facial expression to estimate the emotion. The acquisition unit can also capture the customer's voice using a microphone, and an AI analyzes the voice to estimate the emotion. Furthermore, the acquisition unit can capture the customer's heart rate using a heart rate sensor, and an AI analyzes the heart rate to estimate the emotion. The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit, for example, uses an AI to analyze the acquired facial expression information, voice information, heart rate information, and the like, and estimates the customer's emotion. The estimation unit, for example, uses a facial expression recognition algorithm to estimate the emotion from the customer's facial expression. The estimation unit can also estimate emotions from the customer's voice using a voice analysis algorithm. The estimation unit can also estimate emotions from the customer's heartbeat using a heartbeat analysis algorithm. The determination unit determines how to handle the order according to the customer's emotions estimated by the estimation unit. The determination unit determines how to take the order and make a recommendation based on the estimated emotions, for example, using AI. For example, when the customer is relaxed, the determination unit proposes dishes and drinks that suit the relaxed atmosphere. Furthermore, when the customer is in a hurry, the determination unit can also propose a menu that can be provided quickly. Furthermore, the determination unit can determine a proposal order according to the customer's emotions and propose the order items in that order. The response unit handles the order determined by the determination unit. For example, the response unit uses AI to make an order taking method and a recommendation suggestion according to the customer's emotions. For example, when the customer is relaxed, the response unit makes an order taking method and a recommendation suggestion that suits the relaxed atmosphere. Furthermore, when the customer is in a hurry, the response unit can also propose a menu that can be provided quickly. Furthermore, the response unit can explain the menu according to the customer's emotions.For example, the robot can encourage customers to place orders by providing detailed explanations about dishes that the customer is interested in. This allows the ordering robot system according to the embodiment to read the customer's emotions and handle the order appropriately.

[0081] The acquisition unit acquires emotion estimation information for estimating the emotion of the customer. The emotion estimation information includes, for example, the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content, but is not limited to such examples. For example, the acquisition unit captures the customer's facial expression using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. The acquisition unit can also capture the customer's voice using a microphone, and the AI ​​analyzes the voice to estimate the emotion. Furthermore, the acquisition unit can capture the customer's heart rate using a heart rate sensor, and the AI ​​analyzes the heart rate to estimate the emotion. Specifically, the camera captures the customer's face with high resolution and detects subtle changes in the facial expression using face recognition technology. For example, facial expressions such as smile, surprise, anger, and sadness are analyzed in real time to estimate the emotional state of the customer. The microphone analyzes the tone, pitch, speed, volume, etc. of the customer's voice to detect changes in emotion. For example, a pattern such as a higher voice indicates excitement and a lower voice indicates calmness is identified. The heart rate sensor monitors the customer's heart rate fluctuations in real time and estimates their state of stress or relaxation. This allows the acquisition unit to collect multifaceted data and provide basic information for estimating the customer's emotions with high accuracy. Furthermore, the acquisition unit centrally manages this data and transmits it to the analysis unit in real time, enabling rapid emotion estimation.

[0082] The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit, for example, uses AI to analyze the acquired facial expression information, voice information, heart rate information, etc., and estimates the customer's emotion. The estimation unit, for example, uses a facial expression recognition algorithm to estimate the emotion from the customer's facial expression. The estimation unit can also use a voice analysis algorithm to estimate the emotion from the customer's voice. Furthermore, the estimation unit can also use a heart rate analysis algorithm to estimate the emotion from the customer's heart rate. Specifically, the AI ​​uses deep learning technology to utilize a model learned from a huge data set. For example, the facial expression recognition algorithm extracts facial feature points and analyzes their positional relationship and changes to estimate the emotion. The voice analysis algorithm converts the voice waveform into a spectrogram, extracts the voice features, and estimates the emotion. The heart rate analysis algorithm analyzes the heart rate fluctuation pattern and estimates the stress or relaxation state. As a result, the estimation unit can integrate multiple emotion estimation information to estimate the customer's emotion with high accuracy. Furthermore, the estimation unit can improve the accuracy of emotion estimation by utilizing past data and customer history information. For example, the estimation unit can refer to the emotional state of a particular customer in the past and reflect it in the current emotion estimation. This allows the estimation unit to grasp the customer's emotions more accurately and use it for the next step, which is handling the order.

[0083] The determination unit determines how to handle an order according to the customer's emotions estimated by the estimation unit. The determination unit uses, for example, AI to determine how to take an order and recommend suggestions based on the estimated emotions. For example, if the customer is relaxed, the determination unit suggests dishes and drinks that fit the relaxed atmosphere. In addition, if the customer is in a hurry, the determination unit can also suggest menus that can be served quickly. Furthermore, the determination unit can determine a suggestion order according to the customer's emotions and suggest order items in that order. Specifically, the AI ​​uses a rule-based system or machine learning model to select optimal menus and services based on the customer's emotional state. For example, if the customer is relaxed, herbal tea or light meals with a relaxing effect are suggested. If the customer is in a hurry, menus with short cooking times or items that can be taken out are suggested. Furthermore, the determination unit can make personalized suggestions taking into account the customer's past order history and preferences. For example, for a customer who has preferred to order a particular dish in the past, a new menu related to that dish is suggested. In this way, the determination unit can determine optimal order responses according to the customer's emotions and preferences, thereby improving customer satisfaction. Furthermore, the decision unit can dynamically adjust order handling based on emotion information updated in real time. For example, if the customer's emotion changes, the proposal content is immediately revised in response to the change. This allows the decision unit to always respond flexibly based on the latest information and quickly meet customer needs.

[0084] The response unit handles the order determined by the determination unit. The response unit, for example, uses AI to take orders and make recommendations according to the customer's emotions. For example, if the customer is relaxed, the response unit takes orders and makes recommendations that match the relaxed atmosphere. In addition, if the customer is in a hurry, the response unit can also suggest menus that can be served quickly. Furthermore, the response unit can explain the menu according to the customer's emotions. For example, the customer is encouraged to order by providing detailed explanations about dishes that interest them. Specifically, the AI ​​uses natural language processing technology to smoothly communicate with the customer. For example, if the customer is relaxed, the AI ​​speaks to the customer in a gentle tone and suggests a menu that has a relaxing effect. If the customer is in a hurry, the AI ​​introduces the menu quickly and concisely and accepts the order immediately. Furthermore, the response unit collects customer feedback in real time and uses it to improve the accuracy of order handling. For example, the AI ​​records how the customer responded to the proposed menu and reflects it in the next proposal. This allows the response unit to provide optimal order handling according to the customer's emotions and needs, thereby increasing customer satisfaction. Furthermore, the customer service department can reliably convey information using multiple communication methods. For example, the department can provide menu information and suggestions to customers not only through face-to-face conversations but also through smartphone apps and tablet devices. This allows the customer service department to quickly and reliably handle orders and improve the customer experience.

[0085] The determination unit determines an order taking method or a recommendation based on the customer's emotions, and the response unit can make an order taking method or a recommendation based on the customer's emotions. For example, when a customer is relaxed, the determination unit determines an order taking method or a recommendation that suits the relaxed atmosphere. For example, the determination unit suggests food and drinks that suit the relaxed atmosphere. Also, when a customer is in a hurry, the determination unit can also suggest a menu that can be provided quickly. For example, the determination unit suggests food and drinks that can be provided quickly. The response unit makes the order taking method or recommendation determined by the determination unit. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Also, the response unit can suggest a menu that can be provided quickly. In this way, customer satisfaction can be improved by making an order taking method or recommendation suggestions according to the customer's emotions.

[0086] The determination unit determines a proposal order of the order objects based on the customer's emotions, and the response unit can propose the order objects in the proposal order based on the customer's emotions. For example, when the customer is relaxed, the determination unit determines a proposal order suitable for the relaxed state. For example, the determination unit prioritizes proposal of dishes and drinks that suit a relaxed atmosphere. Also, when the customer is in a hurry, the determination unit can also prioritize proposal of menus that can be provided quickly. For example, the determination unit prioritizes proposal of dishes and drinks that can be provided quickly. The response unit proposes the order objects in the proposal order determined by the determination unit. For example, the response unit prioritizes proposal of dishes and drinks that suit a relaxed atmosphere. Also, the response unit can also prioritize proposal of menus that can be provided quickly. In this way, by proposing the order objects in a proposal order according to the customer's emotions, it is possible to provide a service that meets the needs of the customer.

[0087] The response unit can explain the menu according to the customer's emotions. The response unit, for example, encourages ordering by providing a detailed explanation of the dishes that the customer is interested in. For example, the response unit provides a detailed explanation of the ingredients, cooking methods, allergy information, etc. of the dishes. The response unit can also suggest recommended features of the dishes and how to eat them according to the customer's emotions. For example, the response unit explains the recommended features of the dishes to attract the customer's interest. The response unit can also suggest how to eat the dishes to improve customer satisfaction. In this way, by explaining the menu according to the customer's emotions, it is possible to deepen the customer's understanding and encourage ordering.

[0088] The acquisition unit acquires allergy information of the customer, and the response unit can propose a menu that accommodates the allergy based on the information acquired by the acquisition unit. The acquisition unit, for example, acquires the allergy information of the customer. The allergy information includes, for example, nut allergy, dairy allergy, and the like, but is not limited to such examples. The acquisition unit, for example, acquires allergy information input by the customer. The acquisition unit can also estimate the allergy information from the customer's past order history. The response unit proposes a menu that accommodates the allergy based on the allergy information acquired by the acquisition unit. For example, the response unit proposes a dish that does not contain nuts to a customer who has a nut allergy. The response unit can also propose a dish that does not contain dairy products to a customer who has a dairy allergy. This allows the customer to enjoy a meal with peace of mind by proposing a menu based on the customer's allergy information.

[0089] The determination unit determines an order response based on the emotion of the customer and the emotion of other customers around the customer, and the response unit can respond to the order to the customer based on the emotion of the customer and the emotion of other customers around the customer. The determination unit, for example, determines an order response that suits the overall atmosphere based on the emotion of the customer and the emotion of other customers around the customer. For example, the determination unit determines an order response that suits the quiet atmosphere when surrounding customers are quietly enjoying their meals. Also, the determination unit can determine an order response that suits the lively atmosphere when surrounding customers are lively enjoying their meals. The response unit performs the order response determined by the determination unit. For example, the response unit performs an order response that suits the quiet atmosphere. Also, the response unit can perform an order response that suits the lively atmosphere. In this way, by performing an order response according to the emotion of the customer and other customers around the customer, a service that suits the overall atmosphere can be provided.

[0090] The corresponding unit can display the menu on the digital signage in a manner according to the emotions of the customer and the emotions of other customers around the customer. The corresponding unit adjusts the way the menu is displayed based on, for example, the emotions of the customer and the emotions of the other customers around the customer. For example, the corresponding unit displays a menu that suits a relaxed atmosphere. The corresponding unit can also display a menu that suits a lively atmosphere. Furthermore, the corresponding unit can customize the display content of the menu according to the emotions of the customer. For example, the corresponding unit displays dishes that the customer is interested in with emphasis. The corresponding unit can also display a menu that accommodates allergies based on the allergy information of the customer. In this way, by displaying a menu according to the emotions of the customer and the other customers around the customer, it is possible to provide visually appropriate information.

[0091] The restaurant may include a prediction unit that predicts the order contents based on the order handling decision content by the decision unit. The prediction unit predicts a menu that the customer is likely to order based on the order handling decision content by the decision unit, for example. For example, the prediction unit predicts an optimal menu in consideration of the customer's past order history and current emotional state. The prediction unit can also predict the order contents based on the customer's preferences and allergy information. Furthermore, the prediction unit can adjust the prediction accuracy of the order contents according to the customer's emotions. For example, the prediction unit predicts a menu that suits a relaxed atmosphere for a customer who is in a relaxed state. The prediction unit can also predict a menu that can be quickly provided to a customer who is in a hurry. This makes it possible to efficiently accept orders by predicting the order contents based on the order handling decision content.

[0092] The order robot can function as an order robot having an acquisition unit, an estimation unit, a determination unit, and a response unit. The order robot is used, for example, to automate and streamline order reception in a restaurant. The order robot acquires customer emotion estimation information using the acquisition unit, and estimates the customer emotion using the estimation unit. Next, the order robot determines an order response based on the emotion estimated using the determination unit, and executes the determined order response using the response unit. This allows the order robot to read the customer's emotion and perform an appropriate order response. For example, the order robot captures the customer's facial expression with a camera, and the AI ​​analyzes the facial expression to estimate the emotion. The order robot can also capture the customer's voice with a microphone, and the AI ​​can analyze the voice to estimate the emotion. Furthermore, the order robot can capture the customer's heartbeat with a heartbeat sensor, and the AI ​​can analyze the heartbeat to estimate the emotion. This allows the order robot to read the customer's emotion and perform an appropriate order response, thereby automating and streamlining order reception in a restaurant.

[0093] The acquisition unit can acquire one or more pieces of information among the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, breathing, and posted content as emotion estimation information. The acquisition unit, for example, captures the customer's facial expression using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. For example, the acquisition unit captures the customer's facial expression while looking at a menu, and the AI ​​analyzes the facial expression to estimate the interest level. The acquisition unit can also periodically capture the customer's facial expression while waiting for an order, and the AI ​​can analyze the change to estimate the emotion change. Furthermore, the acquisition unit can capture the customer's facial expression while eating food, and the AI ​​can analyze the facial expression to estimate the satisfaction level. In this way, the emotion of the customer can be more accurately estimated by acquiring various information as emotion estimation information.

[0094] The acquisition unit can acquire the facial expression of the customer and use it as emotion estimation information. For example, the acquisition unit captures the facial expression of the customer using a camera, and the AI ​​analyzes the facial expression to estimate the emotion. For example, the acquisition unit captures the facial expression of the customer while looking at a menu, and the AI ​​analyzes the facial expression to estimate the interest level. The acquisition unit can also periodically capture the facial expression of the customer while waiting for the order, and the AI ​​can analyze the change to estimate the emotion change. Furthermore, the acquisition unit can capture the facial expression of the customer while eating food, and the AI ​​can analyze the facial expression to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the facial expression of the customer.

[0095] The acquisition unit can acquire the customer's behavior and use it as emotion estimation information. For example, the acquisition unit can capture the customer's behavior using a camera, and the AI ​​can analyze the behavior to estimate the emotion. For example, the acquisition unit can capture the customer's behavior of pointing at a menu, and the AI ​​can analyze the behavior to estimate the dish the customer is interested in. The acquisition unit can also capture the hand movements of the customer while waiting for the order, and the AI ​​can analyze the movements to estimate the customer's impatience or anxiety. Furthermore, the acquisition unit can capture the hand movements of the customer while eating the food, and the AI ​​can analyze the movements to estimate the customer's satisfaction. In this way, the accuracy of emotion estimation is improved by acquiring the customer's behavior.

[0096] The acquisition unit can acquire the voice of the customer and use it as emotion estimation information. For example, the acquisition unit can capture the voice of the customer using a microphone, and the AI ​​can analyze the voice to estimate the emotion. For example, the acquisition unit can capture the tone of the voice when the customer places an order, and the AI ​​can analyze the tone to estimate the emotion. The acquisition unit can also capture the reaction voice of the customer when listening to an explanation of the dish, and the AI ​​can analyze the reaction to estimate the interest level. Furthermore, the acquisition unit can capture the customer's impressions while eating the dish, and the AI ​​can analyze the impressions to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's voice.

[0097] The acquisition unit can acquire the heart rate of the customer and use it as emotion estimation information. For example, the acquisition unit can capture the heart rate of the customer using a heart rate sensor, and the AI ​​can analyze the heart rate to estimate the emotion. For example, the acquisition unit can capture the heart rate of the customer while looking at a menu, and the AI ​​can analyze the heart rate to estimate the excitement level. The acquisition unit can also capture the heart rate of the customer while waiting for an order, and the AI ​​can analyze the heart rate to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the heart rate of the customer while eating food, and the AI ​​can analyze the heart rate to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's heart rate.

[0098] The acquisition unit can acquire the customer's blood pressure and use it as emotion estimation information. The acquisition unit, for example, captures the customer's blood pressure using a blood pressure sensor, and the AI ​​analyzes the blood pressure to estimate the emotion. For example, the acquisition unit captures the customer's blood pressure when looking at a menu, and the AI ​​analyzes the blood pressure to estimate the excitement level. The acquisition unit can also capture the customer's blood pressure while waiting to order, and the AI ​​can analyze the blood pressure to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the customer's blood pressure when eating food, and the AI ​​can analyze the blood pressure to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's blood pressure.

[0099] The acquisition unit can acquire the pupillary response of the customer and use it as emotion estimation information. The acquisition unit, for example, uses a pupil sensor to capture the customer's pupillary response, and the AI ​​analyzes the pupillary response to estimate the emotion. For example, the acquisition unit captures the pupillary response when the customer is looking at a menu, and the AI ​​analyzes the response to estimate the excitement level. The acquisition unit can also capture the pupillary response while the customer is waiting for an order, and the AI ​​can analyze the response to estimate impatience or anxiety. Furthermore, the acquisition unit can capture the pupillary response when the customer is eating food, and the AI ​​can analyze the response to estimate the satisfaction level. In this way, the accuracy of emotion estimation is improved by acquiring the customer's pupillary response.

[0100] The estimation unit can estimate the emotion of the customer and adjust the estimation algorithm based on the estimated emotion of the customer. For example, when the customer is relaxed, the estimation unit applies an estimation algorithm suitable for the relaxed state. For example, the estimation unit uses an emotion estimation algorithm suitable for the relaxed state. Furthermore, when the customer is excited, the estimation unit can also apply an estimation algorithm suitable for the excited state. For example, the estimation unit uses an emotion estimation algorithm suitable for the excited state. Furthermore, when the customer is impatient, the estimation unit can also apply an estimation algorithm suitable for the impatient state. For example, the estimation unit uses an emotion estimation algorithm suitable for the impatient state. In this way, by adjusting the estimation algorithm based on the emotion of the customer, the accuracy of emotion estimation is improved.

[0101] The estimation unit can adjust the level of detail of the estimation based on the importance of the emotion estimation information at the time of estimation. For example, when facial expression information of a customer is important, the estimation unit performs detailed emotion estimation based on the facial expression information. For example, the estimation unit prioritizes analysis of the facial expression information and performs detailed emotion estimation. Furthermore, when voice information of a customer is important, the estimation unit can also perform detailed emotion estimation based on voice information. For example, the estimation unit prioritizes analysis of the voice information and performs detailed emotion estimation. Furthermore, when heart rate information of a customer is important, the estimation unit can also perform detailed emotion estimation based on the heart rate information. For example, the estimation unit prioritizes analysis of the heart rate information and performs detailed emotion estimation. Thereby, by adjusting the level of detail of the estimation based on the importance of the emotion estimation information, the accuracy of emotion estimation is improved.

[0102] The estimation unit can apply different estimation algorithms depending on the category of the emotion estimation information at the time of estimation. For example, the estimation unit applies a facial expression recognition algorithm to emotion estimation based on facial expression information. For example, the estimation unit uses the facial expression recognition algorithm to estimate an emotion from a customer's facial expression. The estimation unit can also apply a voice analysis algorithm to emotion estimation based on voice information. For example, the estimation unit uses a voice analysis algorithm to estimate an emotion from a customer's voice. The estimation unit can also apply a heartbeat analysis algorithm to emotion estimation based on heartbeat information. For example, the estimation unit uses a heartbeat analysis algorithm to estimate an emotion from a customer's heartbeat. Thereby, the accuracy of emotion estimation is improved by applying an optimal estimation algorithm depending on the category of the emotion estimation information.

[0103] The estimation unit can estimate the customer's emotions and adjust the display method of the estimation result based on the estimated customer's emotions. For example, when the customer is relaxed, the estimation unit provides a display method suitable for the relaxed state. For example, the estimation unit performs a graph display or text display suitable for the relaxed state. Furthermore, when the customer is excited, the estimation unit can also provide a display method suitable for the excited state. For example, the estimation unit performs a graph display or text display suitable for the excited state. Furthermore, when the customer is impatient, the estimation unit can also provide a display method suitable for the impatient state. For example, the estimation unit performs a graph display or text display suitable for the impatient state. This makes it possible to provide more appropriate information by adjusting the display method of the estimation result based on the customer's emotions.

[0104] The estimation unit can determine the priority of estimation based on the acquisition time of the emotion estimation information at the time of estimation. The estimation unit, for example, prioritizes the latest emotion estimation information to perform estimation. For example, the estimation unit prioritizes the latest facial expression information and voice information to estimate an emotion. The estimation unit can also perform estimation with reference to past emotion estimation information. For example, the estimation unit analyzes with reference to past facial expression information and voice information to estimate an emotion. Furthermore, the estimation unit can also perform estimation with priority on emotion estimation information acquired in a specific time period. For example, the estimation unit prioritizes the analysis of facial expression information and voice information acquired in a specific time period to estimate an emotion. Thereby, by determining the priority of estimation based on the acquisition time of the emotion estimation information, it is possible to prioritize the latest information for estimation.

[0105] The estimation unit can adjust the order of estimation based on the relevance of the emotion estimation information at the time of estimation. For example, when facial expression information of a customer is highly relevant, the estimation unit prioritizes facial expression information in the estimation. For example, the estimation unit prioritizes the analysis of facial expression information and estimates an emotion. Furthermore, when voice information of a customer is highly relevant, the estimation unit can also prioritize voice information in the estimation. For example, the estimation unit prioritizes the analysis of voice information and estimates an emotion. Furthermore, when heart rate information of a customer is highly relevant, the estimation unit can also prioritize heart rate information in the estimation. For example, the estimation unit prioritizes the analysis of heart rate information and estimates an emotion. Thereby, by adjusting the order of estimation based on the relevance of the emotion estimation information, it is possible to prioritize information with higher relevance in the estimation.

[0106] The determination unit can estimate the customer's emotions and adjust the order handling decision method based on the estimated customer's emotions. For example, when the customer is relaxed, the determination unit determines an order handling appropriate for the relaxed state. For example, the determination unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the determination unit can also determine an order handling appropriate for the excited state. For example, the determination unit suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the determination unit can also determine an order handling appropriate for the impatient state. For example, the determination unit suggests a menu that can be provided quickly. In this way, by adjusting the order handling decision method based on the customer's emotions, more appropriate order handling is possible.

[0107] When making a decision, the decision unit can analyze the customer's past order history and select the optimal order response. The decision unit, for example, selects the optimal order response by referring to dishes that the customer has ordered in the past. For example, the decision unit suggests a specific dish from the customer's past order history. The decision unit can also analyze the customer's past order history and select an order response that suits the customer's preferences. For example, the decision unit suggests dishes and drinks that suit the customer's preferences. In this way, by analyzing the customer's past order history, it is possible to select the optimal order response that suits the customer's preferences.

[0108] At the time of decision, the decision unit can customize the means for handling the order based on the customer's current situation. For example, if the customer is in a hurry, the decision unit suggests a menu that can be provided quickly. For example, the decision unit suggests food and drinks that can be provided quickly. Moreover, if the customer is relaxed, the decision unit can also suggest food and drinks that suit the relaxed atmosphere. For example, the decision unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, if the customer has a specific allergy, the decision unit can also suggest a menu that accommodates the allergy. For example, the decision unit suggests food and drinks that accommodate the allergy. This allows for more appropriate order handling by customizing the means for handling the order based on the customer's current situation.

[0109] The determination unit can estimate the emotion of the customer and determine the order of suggestions of the order items based on the estimated emotion of the customer. For example, when the customer is relaxed, the determination unit determines the order of suggestions suitable for the relaxed state. For example, the determination unit prioritizes suggestions of dishes and drinks that suit a relaxed atmosphere. Furthermore, when the customer is excited, the determination unit can also determine the order of suggestions suitable for the excited state. For example, the determination unit prioritizes suggestions of dishes and drinks that suit an excited atmosphere. Furthermore, when the customer is impatient, the determination unit can also determine the order of suggestions suitable for the impatient state. For example, the determination unit prioritizes suggestions of menu items that can be provided quickly. In this way, by determining the order of suggestions of the order items based on the emotion of the customer, more appropriate suggestions are possible.

[0110] When making a decision, the determination unit can determine the suggestion order by referring to the customer's past order history. For example, the determination unit prioritizes and suggests specific dishes from the customer's past order history. The determination unit can also analyze the customer's past order history and determine the suggestion order that suits the customer's preferences. For example, the determination unit prioritizes and suggests dishes and drinks that suit the customer's preferences. In this way, by referring to the customer's past order history, the suggestion order that suits the customer's preferences can be determined.

[0111] At the time of determination, the determination unit can customize the proposal order based on the current situation of the customer. For example, when the customer is in a hurry, the determination unit prioritizes and proposes a menu that can be provided quickly. For example, the determination unit prioritizes and proposes food and drinks that can be provided quickly. In addition, when the customer is relaxed, the determination unit can also prioritize and propose food and drinks that suit the relaxed atmosphere. For example, the determination unit prioritizes and proposes food and drinks that suit the relaxed atmosphere. Furthermore, when the customer has a specific allergy, the determination unit can also prioritize and propose a menu that corresponds to the allergy. For example, the determination unit prioritizes and proposes food and drinks that correspond to the allergy. In this way, by customizing the proposal order based on the current situation of the customer, more appropriate proposals can be made.

[0112] The response unit can estimate the customer's emotions and adjust the method of handling the order based on the estimated customer's emotions. For example, when the customer is relaxed, the response unit handles the order in a way that suits the relaxed atmosphere. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the response unit can also handle the order in a way that suits the excited atmosphere. For example, the response unit suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the response unit can also handle the order quickly. For example, the response unit suggests a menu that can be provided quickly. This makes it possible to handle the order more appropriately by adjusting the method of handling the order based on the customer's emotions.

[0113] When responding, the response unit can select the optimal response method by referring to the customer's past order history. For example, the response unit can suggest a specific dish from the customer's past order history. The response unit can also analyze the customer's past order history and select a response method that suits the customer's preferences. For example, the response unit can suggest a dish or drink that suits the customer's preferences. In this way, by referring to the customer's past order history, the optimal response method that suits the customer's preferences can be selected.

[0114] The response unit can customize the means of response based on the customer's current situation when responding. For example, if the customer is in a hurry, the response unit suggests a menu that can be provided quickly. For example, the response unit suggests food and drinks that can be provided quickly. Furthermore, if the customer is relaxed, the response unit can also suggest food and drinks that suit the relaxed atmosphere. For example, the response unit suggests food and drinks that suit the relaxed atmosphere. Furthermore, if the customer has a specific allergy, the response unit can also suggest a menu that accommodates the allergy. For example, the response unit suggests food and drinks that accommodate the allergy. This makes it possible to provide a more appropriate response by customizing the means of response based on the customer's current situation.

[0115] The response unit can estimate the emotion of the customer and determine the priority of the response based on the estimated emotion of the customer. For example, when the customer is relaxed, the response unit prioritizes a response suitable for the relaxed state. For example, the response unit prioritizes and suggests food and drinks that suit the relaxed atmosphere. Furthermore, when the customer is excited, the response unit can also prioritize a response suitable for the excited state. For example, the response unit prioritizes and suggests food and drinks that suit the excited atmosphere. Furthermore, when the customer is impatient, the response unit can also prioritize a response suitable for the impatient state. For example, the response unit prioritizes and suggests a menu that can be provided quickly. In this way, by determining the priority of the response based on the emotion of the customer, more appropriate response is possible.

[0116] When responding, the response unit can select the optimal response method by taking into consideration the geographical location information of the customer. For example, when the customer is in a specific area, the response unit suggests local specialties of the area. For example, the response unit prioritizes suggestions of local specialties of the specific area. Furthermore, when the customer is in a tourist spot, the response unit can also suggest dishes and drinks related to the tourist spot. For example, the response unit prioritizes suggestions of dishes and drinks related to the tourist spot. Furthermore, when the customer is participating in a specific event, the response unit can also suggest a menu related to the event. For example, the response unit prioritizes suggestions of dishes and drinks related to the specific event. In this way, a more appropriate response method can be selected by taking into consideration the geographical location information of the customer.

[0117] When responding, the response unit can analyze the social media activity of the customer and suggest a response measure. For example, if the customer mentions a specific dish on social media, the response unit suggests the dish. For example, the response unit prioritizes and suggests the dish mentioned on social media. Furthermore, if the customer is participating in a specific event on social media, the response unit can also suggest a menu related to the event. For example, the response unit prioritizes and suggests dishes and drinks related to the event mentioned on social media. Furthermore, if the customer mentions a specific allergy on social media, the response unit can also suggest a menu that accommodates the allergy. For example, the response unit prioritizes and suggests dishes and drinks that accommodate the allergy mentioned on social media. In this way, by analyzing the social media activity of the customer, it is possible to suggest a more appropriate response measure.

[0118] The prediction unit can estimate the customer's emotions and adjust the method of predicting the order contents based on the estimated customer's emotions. For example, when the customer is relaxed, the prediction unit predicts the order contents suitable for the relaxed state. For example, the prediction unit predicts food and drinks suitable for a relaxed atmosphere. Furthermore, when the customer is excited, the prediction unit can also predict the order contents suitable for the excited state. For example, the prediction unit predicts food and drinks suitable for an excited atmosphere. Furthermore, when the customer is impatient, the prediction unit can also predict the order contents suitable for the impatient state. For example, the prediction unit predicts a menu that can be provided quickly. This allows for more appropriate prediction by adjusting the method of predicting the order contents based on the customer's emotions.

[0119] When making a prediction, the prediction unit can select the optimal prediction method by referring to the customer's past order history. For example, the prediction unit predicts a specific dish from the customer's past order history. The prediction unit can also analyze the customer's past order history and select a prediction method that suits the customer's preferences. For example, the prediction unit predicts dishes and drinks that suit the customer's preferences. In this way, by referring to the customer's past order history, it is possible to select the optimal prediction method that suits the customer's preferences.

[0120] The prediction unit can estimate the customer's emotions and determine the priority of predictions based on the estimated customer's emotions. For example, when the customer is relaxed, the prediction unit prioritizes predictions suitable for a relaxed state. For example, the prediction unit prioritizes predictions of dishes and drinks suitable for a relaxed atmosphere. In addition, when the customer is excited, the prediction unit can also prioritize predictions suitable for an excited state. For example, the prediction unit prioritizes predictions of dishes and drinks suitable for an excited atmosphere. In addition, when the customer is impatient, the prediction unit can also prioritize predictions suitable for an impatient state. For example, the prediction unit prioritizes predictions of menus that can be provided quickly. In this way, more appropriate predictions can be made by determining the priority of predictions based on the customer's emotions.

[0121] The prediction unit can select the optimal prediction method in consideration of the geographical location information of the customer at the time of prediction. For example, when the customer is in a specific area, the prediction unit predicts the specialty products of the specific area. For example, the prediction unit predicts the specialty products of the specific area with priority. In addition, when the customer is in a tourist spot, the prediction unit can also predict dishes and drinks related to the tourist spot. For example, the prediction unit predicts dishes and drinks related to the tourist spot with priority. Furthermore, when the customer is participating in a specific event, the prediction unit can also predict the menu related to the event. For example, the prediction unit predicts dishes and drinks related to the specific event with priority. In this way, a more appropriate prediction method can be selected by considering the geographical location information of the customer.

[0122] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0123] The order robot system may further include a history analysis unit that analyzes the customer's past order history. The history analysis unit accumulates data on food and drinks that the customer has ordered in the past, and analyzes the customer's preferences and trends based on this data. For example, the history analysis unit may identify food that the customer has frequently ordered in the past, and preferentially suggest those food items. The history analysis unit may also take into consideration food that the customer has avoided in the past or allergy information, and suggest menu items that do not include these. Furthermore, the history analysis unit may suggest special menu items according to seasons or events based on the customer's past order history. This makes it possible to make more personalized suggestions by utilizing the customer's past order history, thereby improving customer satisfaction.

[0124] The order robot system may further include a location information acquisition unit that acquires real-time location information of the customer. The location information acquisition unit identifies the customer's current location and proposes menus and services according to the location. For example, when the customer is in a tourist spot, the location information acquisition unit may propose local specialties or menus that are only available in the tourist spot. In addition, when the customer is at a specific event venue, the location information acquisition unit may propose a special menu related to the event. Furthermore, when the customer is at home, the location information acquisition unit may propose a delivery service. In this way, by utilizing the customer's real-time location information, more appropriate menus and services can be provided.

[0125] The ordering robot system may further include a social media analysis unit that analyzes the social media activities of customers. The social media analysis unit collects information on food, drinks, events, etc. that customers have mentioned on social media, and makes suggestions based on this information. For example, the social media analysis unit may identify food that a customer has recently mentioned on social media, and prioritize the suggestion of that food. The social media analysis unit may also suggest a menu item related to an event that the customer is planning to attend. Furthermore, the social media analysis unit may take into account allergy information shared by the customer on social media and suggest a menu item that accommodates allergies. This allows for more personalized suggestions by utilizing the customer's social media activities.

[0126] The ordering robot system may further include a health monitoring unit that monitors the health condition of the customer. The health monitoring unit obtains health data such as the customer's heart rate, blood pressure, and body temperature in real time and makes suggestions based on this. For example, if the customer's heart rate is high, the health monitoring unit may suggest a menu with a relaxing effect. Also, if the customer's blood pressure is high, the health monitoring unit may suggest a low-salt menu. Furthermore, if the customer's body temperature is high, the health monitoring unit may suggest cold drinks or desserts. This makes it possible to provide healthier meals by making suggestions that take the customer's health condition into consideration.

[0127] The ordering robot system may further include a preference learning unit that learns the food preferences of the customer. The preference learning unit learns the customer's preferences based on data on food and drinks that the customer has ordered in the past, and makes suggestions based on this. For example, the preference learning unit may analyze trends in food that the customer has ordered in the past, and suggest new dishes based on these trends. In addition, if a customer has a preference for a particular ingredient, the preference learning unit may also suggest dishes that use that ingredient. In addition, if a customer has a preference for a particular cooking method, the preference learning unit may also suggest dishes that use that cooking method. In this way, by learning the customer's food preferences, more personalized suggestions are possible.

[0128] The ordering robot system can estimate the customer's emotions and evaluate the customer's stress level based on the estimated emotions. For example, if the customer is feeling stressed, it can suggest a menu or drink that has a relaxing effect. Also, if the customer is relaxed, it can suggest a menu to help the customer maintain that state. Furthermore, if the customer is excited, it can suggest a menu to help relieve excitement. In this way, it is possible to support the customer's physical and mental health by evaluating the customer's stress level and making suggestions accordingly.

[0129] The ordering robot system can estimate the customer's emotions and evaluate the customer's appetite based on the estimated emotions. For example, if the customer has no appetite, it can suggest a light menu or a menu that stimulates the appetite. Also, if the customer has an appetite, it can suggest a hearty menu. Furthermore, if the customer shows an interest in a particular dish, it can suggest a menu related to that dish. In this way, customer satisfaction can be improved by evaluating the customer's appetite and making suggestions accordingly.

[0130] The ordering robot system can estimate the customer's emotions and evaluate the customer's curiosity based on the estimated emotions. For example, if the customer is curious, it can suggest new dishes or menus using rare ingredients. Also, if the customer is interested in a particular dish, it can explain the story or background related to that dish. Furthermore, if the customer is looking for a new experience, it can suggest a menu with a special cooking method or presentation. In this way, by evaluating the customer's curiosity and making suggestions accordingly, it is possible to attract the customer's interest and improve their satisfaction.

[0131] The ordering robot system can estimate the customer's emotions and evaluate the customer's level of fatigue based on the estimated emotions. For example, if the customer is tired, it can suggest a menu or drink that has a refreshing effect. Also, if the customer is relaxed, it can suggest a menu to maintain that state. Furthermore, if the customer is energetic, it can also suggest a menu to replenish energy. In this way, it is possible to support the customer's physical condition by evaluating the customer's level of fatigue and making suggestions accordingly.

[0132] The order robot system can estimate the emotions of a customer and evaluate the customer's satisfaction level based on the estimated emotions. For example, if a customer is satisfied, it can suggest menu items and services to maintain that state. Also, if a customer is dissatisfied, it can identify the cause and make suggestions to improve it. Furthermore, if a customer is satisfied with a particular dish, it can also suggest a menu item related to that dish. In this way, it is possible to improve customer satisfaction by evaluating customer satisfaction and making suggestions accordingly.

[0133] The process flow of the second embodiment will be briefly described below.

[0134] Step 1: The acquisition unit acquires emotion estimation information for estimating the emotion of a customer. The emotion estimation information includes the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwaves, body temperature, breathing, and posted content. For example, a camera is used to capture the customer's facial expression, and an AI analyzes the facial expression to estimate the emotion. A microphone can also be used to capture the customer's voice, and an AI can analyze the voice to estimate the emotion. A heart rate sensor can also be used to capture the customer's heart rate, and an AI can analyze the heart rate to estimate the emotion. Step 2: The estimation unit estimates the customer's emotion based on the emotion estimation information acquired by the acquisition unit. The estimation unit uses AI to analyze the acquired facial expression information, voice information, heart rate information, etc., and estimates the customer's emotion. For example, the estimation unit estimates emotion from the customer's facial expression using a facial expression recognition algorithm, estimates emotion from the customer's voice using a voice analysis algorithm, and estimates emotion from the customer's heart rate using a heart rate analysis algorithm. Step 3: The decision unit decides how to handle the order according to the customer's emotions estimated by the estimation unit. The decision unit uses AI to decide how to take the order and what recommendations to make based on the estimated emotions. For example, if the customer is relaxed, it will suggest dishes and drinks that suit a relaxed atmosphere, and if the customer is in a hurry, it will suggest menu items that can be served quickly. Furthermore, it is also possible to determine the order order based on the customer's emotions and make suggestions for order items in that order. Step 4: The response unit handles the order determined by the determination unit. The response unit uses AI to take the order and make recommendations based on the customer's emotions. For example, if the customer is relaxed, the response unit will take the order in a way that suits the relaxed atmosphere and make recommendations, and if the customer is in a hurry, it will suggest a menu item that can be served quickly. In addition, the response unit can provide an explanation of the menu item based on the customer's emotions. For example, the response unit will encourage the customer to place an order by providing a detailed explanation of the dish that the customer is interested in.

[0135] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0136] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0137] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0138] For example, the acquisition unit acquires the facial expression and voice of the customer using the camera 42 and microphone 38B of the smart device 14, and the acquired facial expression and voice are analyzed by the specific processing unit 290 of the data processing device 12. The estimation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and estimates the customer's emotion based on the acquired information. The determination unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and determines the order handling based on the estimated emotion. The handling unit is realized, for example, by the control unit 46A of the smart device 14, and executes the determined order handling. The correspondence relationship between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0139] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0140] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0141] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0142] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0143] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0144] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0145] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0146] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0147] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0149] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0150] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0151] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0152] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0153] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0154] For example, the acquisition unit acquires the customer's facial expression and voice using the camera 42 and microphone 238 of the smart glasses 214, and the acquired facial expression and voice are analyzed by the specific processing unit 290 of the data processing device 12. The estimation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and estimates the customer's emotion based on the acquired information. The determination unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and determines the order handling based on the estimated emotion. The handling unit is realized, for example, by the control unit 46A of the smart glasses 214, and executes the determined order handling. The correspondence relationship between each unit and the device or control unit is not limited to the above example, and various changes are possible.

[0155] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0156] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0157] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0158] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0159] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0160] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0161] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0162] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0163] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0164] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0165] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0166] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0167] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0168] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0169] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0170] For example, the acquisition unit acquires the facial expression and voice of the customer using the camera 42 and microphone 238 of the headset terminal 314, and the acquired facial expression and voice are analyzed by the specific processing unit 290 of the data processing device 12. The estimation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and estimates the customer's emotion based on the acquired information. The decision unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and determines how to handle the order based on the estimated emotion. The response unit is realized, for example, by the control unit 46A of the headset terminal 314, and executes the determined order response. The correspondence between each unit and the device or control unit is not limited to the above example, and various changes are possible.

[0171] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0172] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0173] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0174] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0175] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0176] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0177] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0178] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0179] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0180] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0181] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0182] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0183] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0184] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0185] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0186] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0187] For example, the acquisition unit acquires the facial expression and voice of the customer using the camera 42 and microphone 238 of the robot 414, and the acquired facial expression and voice are analyzed by the specific processing unit 290 of the data processing device 12. The estimation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and estimates the customer's emotions based on the acquired information. The decision unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and determines how to handle the order based on the estimated emotions. The response unit is realized, for example, by the control unit 46A of the robot 414, and executes the determined order response. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0188] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0189] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0190] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0191] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[0192] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0193] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0194] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0195] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0196] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0197] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0198] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0199] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0200] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0201] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0202] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0203] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0204] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0205] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0206] (Appendix 1) An acquisition unit that acquires emotion estimation information for estimating emotions of customers in a restaurant; an estimation unit that estimates an emotion of the customer based on the emotion estimation information acquired by the acquisition unit; A determination unit that determines an order response in accordance with the emotion of the customer estimated by the estimation unit; a response unit that responds to the order determined by the determination unit. A system characterized by: (Appendix 2) The determination unit is determining an order taking method or a recommendation based on the customer's sentiment; The corresponding portion is Taking orders or making recommendations based on the customer's emotions 2. The system of claim 1, (Appendix 3) The determination unit is determining a proposal order for order items based on the customer's sentiment; The corresponding portion is Suggesting order items in a suggested order based on the customer's sentiment 2. The system of claim 1, (Appendix 4) The corresponding portion is Explain the menu according to the customer's feelings 2. The system of claim 1, (Appendix 5) The acquisition unit is Obtaining allergy information of the customer; The corresponding portion is Suggesting a menu for allergies based on the information acquired by the acquisition unit 2. The system of claim 1, (Appendix 6) The determination unit is determining order handling based on the customer's emotions and the emotions of other customers around the customer; The corresponding portion is Handle orders for the customer based on the customer's emotions and the emotions of other customers around the customer 2. The system of claim 1, (Appendix 7) The corresponding portion is A menu is displayed on a digital sign in a manner corresponding to the emotion of the customer and the emotion of other customers around the customer. 2. The system of claim 1, (Appendix 8) Based on the order response decision content by the decision unit, Equipped with a prediction unit that predicts order contents 2. The system of claim 1, (Appendix 9) The acquisition unit, The estimation unit, The determination unit, and functioning as an order robot having the corresponding unit. 2. The system of claim 1, (Appendix 10) The acquisition unit is One or more pieces of information among the customer's facial expression, movement, voice, heart rate, blood pressure, pupil response, brainwave, body temperature, respiration, and posted content are acquired as the emotion estimation information. 2. The system of claim 1, (Appendix 11) The acquisition unit is Acquire customer facial expressions and use them as emotion estimation information 2. The system of claim 1, (Appendix 12) The acquisition unit is Obtaining customer behavior and using it as emotion estimation information 2. The system of claim 1, (Appendix 13) The acquisition unit is Acquire customer voice and use it as information for estimating emotions 2. The system of claim 1, (Appendix 14) The acquisition unit is Obtaining the customer's heart rate and using it as information for estimating emotions 2. The system of claim 1, (Appendix 15) The acquisition unit is Obtaining the customer's blood pressure and using it as information for estimating emotions 2. The system of claim 1, (Appendix 16) The acquisition unit is Obtaining the customer's pupil response and using it as information for estimating emotions 2. The system of claim 1, (Appendix 17) The estimation unit is Infer customer sentiment and adjust the inference algorithm based on the inferred customer sentiment 2. The system of claim 1, (Appendix 18) The estimation unit is During estimation, the level of detail of the estimation is adjusted based on the importance of the emotion estimation information. 2. The system of claim 1, (Appendix 19) The estimation unit is During estimation, different estimation algorithms are applied depending on the category of emotion estimation information. 2. The system of claim 1, (Appendix 20) The estimation unit is Estimate customer sentiment and adjust how the results are displayed based on the estimated customer sentiment 2. The system of claim 1, (Appendix 21) The estimation unit is During estimation, the priority of estimation is determined based on the time when emotion estimation information was acquired. 2. The system of claim 1, (Appendix 22) The estimation unit is During estimation, the order of estimation is adjusted based on the relevance of emotion estimation information. 2. The system of claim 1, (Appendix 23) The determination unit is Infer customer sentiment and adjust order fulfillment decisions based on the inferred customer sentiment 2. The system of claim 1, (Appendix 24) The determination unit is When making a decision, analyze the customer's past order history to select the optimal order action 2. The system of claim 1, (Appendix 25) The determination unit is At the point of decision, customize order fulfillment based on the customer's current situation 2. The system of claim 1, (Appendix 26) The determination unit is To estimate customer sentiment and determine the order of proposed orders based on the estimated customer sentiment 2. The system of claim 1, (Appendix 27) The determination unit is When making a decision, the order of proposals is determined based on the customer's past order history. 2. The system of claim 1, (Appendix 28) The determination unit is At decision time, customize the order of suggestions based on the customer's current situation 2. The system of claim 1, (Appendix 29) The corresponding portion is Infer customer sentiment and adjust order fulfillment based on inferred customer sentiment 2. The system of claim 1, (Appendix 30) The corresponding portion is When responding, refer to the customer's past order history to determine the best way to respond 2. The system of claim 1, (Appendix 31) The corresponding portion is When interacting, tailor your interactions based on the customer's current situation 2. The system of claim 1, (Appendix 32) The corresponding portion is Estimate customer sentiment and prioritize interactions based on the estimated customer sentiment 2. The system of claim 1, (Appendix 33) The corresponding portion is When responding, consider the customer's geographic location to determine the best way to respond 2. The system of claim 1, (Appendix 34) The corresponding portion is When responding, analyze the customer's social media activity to suggest ways to respond 2. The system of claim 1, (Appendix 35) The prediction unit is Infer customer sentiment and adjust order prediction methods based on inferred customer sentiment 2. The system of claim 1, (Appendix 36) The prediction unit is When making forecasts, the system uses the customer's past order history to select the best forecasting method. 2. The system of claim 1, (Appendix 37) The prediction unit is Estimate customer sentiment and prioritize predictions based on estimated customer sentiment 2. The system of claim 1, (Appendix 38) The prediction unit is When forecasting, consider the geographic location of the customer to select the most appropriate forecasting method 2. The system of claim 1, [Explanation of symbols]

[0207] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. An acquisition unit that acquires emotion estimation information for estimating emotions of customers in a restaurant; an estimation unit that estimates an emotion of the customer based on the emotion estimation information acquired by the acquisition unit; A determination unit that determines an order response in accordance with the emotion of the customer estimated by the estimation unit; A response unit that responds to the order determined by the determination unit. A system characterized by:

2. The determination unit is determining an order taking method or a recommendation based on the customer's sentiment; The corresponding portion is Taking orders or making recommendations based on the customer's emotions 2. The system of claim 1 .

3. The determination unit is determining a proposal order for order items based on the customer's sentiment; The corresponding portion is Suggesting order items in a suggested order based on the customer's sentiment 2. The system of claim 1 .

4. The corresponding portion is Explain the menu according to the customer's feelings 2. The system of claim 1 .

5. The acquisition unit is Obtaining allergy information of the customer; The corresponding portion is Suggesting a menu for allergies based on the information acquired by the acquisition unit 2. The system of claim 1 .

6. The determination unit is determining order handling based on the customer's emotions and the emotions of other customers around the customer; The corresponding portion is Handle orders for the customer based on the customer's emotions and the emotions of other customers around the customer 2. The system of claim 1 .

7. The corresponding portion is A menu is displayed on a digital sign in a manner corresponding to the emotion of the customer and the emotion of other customers around the customer.

2. The system of claim 1 .

8. Based on the order response decision content by the decision unit, Equipped with a prediction unit that predicts order contents 2. The system of claim 1 .

Images