system

Abstract

Provide a system. [Solution] A means of customer recognition; A means of interacting with customers and proposing menus, a means of receiving and confirming customer orders; a means for transmitting order information to a kitchen; a means for delivering the ordered food to the customer; A means of receiving and responding to additional customer requests; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional restaurants, when a customer visits a restaurant, the wait staff guides the customer, suggests menu items, takes orders, serves the food, and handles any additional requests. This places a heavy workload on the staff, which can lead to a decline in the quality of customer service, especially during busy times. It also poses the risk of ordering errors and increased waiting times, leading to lower customer satisfaction. Therefore, there was a need for a system that could improve the quality of customer service and reduce the workload on staff. [Means for solving the problem]

[0005] The present invention provides a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. This system includes the following elements: a means for recognizing customers, a means for interacting with customers to suggest menu items, a means for receiving and confirming customer orders, a means for transmitting order information to the kitchen, a means for delivering ordered dishes to customers, and a means for accepting and responding to additional customer requests. This reduces the workload on staff while improving the quality of service provided to customers. Furthermore, by providing a means for suggesting menu items by referencing ingredient inventory information and past order history, it is possible to make optimal suggestions based on customer needs. Furthermore, by providing a means for interacting with customers and accepting orders and requests using chat generation AI, natural dialogue can be achieved, improving customer satisfaction.

[0006] "Customer" refers to a person who visits a service providing location such as a restaurant and receives a service.

[0007] "Means of recognition" refers to a mechanism for detecting and identifying the presence of customers using cameras, sensors, etc.

[0008] "Means of dialogue" refers to a system for conducting natural conversations with customers using chat generation AI, etc.

[0009] "Means for suggesting menus" refers to a system that has the function of recommending appropriate dishes and drinks to customers.

[0010] "Means for receiving and confirming orders" refers to the system for receiving and reconfirming orders from customers.

[0011] "Means of sending to the kitchen" refers to a mechanism that has the function of transmitting received order information to the cook or cooking system.

[0012] "Means for serving food" refers to a system that has the function of transporting cooked food to the customer's table.

[0013] "Means for accepting and responding to additional requests" refers to a system for accepting and responding to additional requests from customers.

[0014] "Ingredient inventory information" refers to data regarding the quantity and condition of ingredients currently available for cooking.

[0015] "Past order history" refers to data that records the orders that a customer has previously placed.

[0016] "Chat generation AI" refers to artificial intelligence technology that uses natural language processing to generate dialogue with customers. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing customers, a means for interacting with customers to suggest menu items, a means for receiving and confirming customer orders, a means for transmitting order information to the kitchen, a means for serving ordered dishes to customers, and a means for receiving and responding to additional requests from customers. The specific program processing is explained below in natural language.

[0039] Program processing

[0040] 1. User Awareness

[0041] The device uses a camera and sensors to recognize users sitting at the table.

[0042] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0043] 2. Menu suggestions

[0044] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0045] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0046] 3. Order Receipt and Confirmation

[0047] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0048] The terminal will reconfirm "Would you like a Japanese-style hamburger?" and if the user answers "Yes", it will send the order information to the server.

[0049] 4. Kitchen collaboration

[0050] The server sends the order information to the kitchen system and issues an instruction to start cooking.

[0051] 5. Serving the food

[0052] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[0053] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0054] 6. Response to requests

[0055] The user requests "additional water please."

[0056] The terminal responds, "Understood. We'll bring it to you right away," and sends additional order information to the server.

[0057] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0058] The terminal delivers the water to the user's table.

[0059] Specific examples

[0060] Example of what happens when a user communicates a specific request

[0061] The device recognizes the user and says, "Hello, welcome."

[0062] A user asks, "What do you recommend?"

[0063] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0064] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0065] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0066] The terminal transmits the order information to the server.

[0067] The server sends the order information to the kitchen system.

[0068] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0069] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0070] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The terminal uses a camera and sensors to recognize the user sitting at a table, allowing the terminal to know that a customer has arrived at the restaurant.

[0074] Step 2:

[0075] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0076] Step 3:

[0077] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[0078] Step 4:

[0079] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0080] Step 5:

[0081] When the user responds, "Please tell me your recommendation," the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0082] Step 6:

[0083] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0084] Step 7:

[0085] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[0086] Step 8:

[0087] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[0088] Step 9:

[0089] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[0090] Step 10:

[0091] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0092] Step 11:

[0093] The user requests "additional water please."

[0094] Step 12:

[0095] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0096] Step 13:

[0097] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0098] Step 14:

[0099] The terminal will bring the water to the user's table and fulfill their request.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] In conventional restaurants, the process of receiving an order from a customer, transmitting it to the kitchen, and serving the food required a great deal of time and effort. It was also difficult to respond quickly to additional customer requests, which could lead to a decline in the quality of service. This led to issues such as lower customer satisfaction and a deterioration in store operational efficiency.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes a means for recognizing a user, a means for interacting with the user to provide guidance, a means for receiving and confirming a user's request, a means for transmitting the request information to the cooking department, a means for providing the user with the requested food, and a means for receiving and responding to additional requests from the user. This automates the entire process from when the user enters the store to when the user completes their order, enabling fast and efficient service. Furthermore, by referencing past request history and ingredient inventory information, guidance suggestions based on the user's preferences can be made, and by using a dialogue generation AI, orders and requests can be accepted through natural dialogue. This increases user satisfaction and improves the efficiency of store operations.

[0105] "User" refers to a customer who receives services using the system.

[0106] "Means for recognizing" refers to means for identifying a user using a photographing device or a detecting device.

[0107] "Means for suggesting guidance through dialogue" refers to means for communicating with users via voice or text and suggesting appropriate menus or services.

[0108] "Means for receiving and confirming requests" refers to the means for receiving and confirming orders or requests from users.

[0109] "Means for transmitting request information to the cooking department" refers to means for transmitting received orders and requests to the cooking department.

[0110] "Means for providing requested food" refers to the means for delivering cooked food to the customer.

[0111] "Means for accepting and responding to additional requests" refers to means for accepting additional requests from users and providing services in response to those requests.

[0112] "Raw material inventory information" refers to data showing the inventory status of food ingredients, beverages, etc.

[0113] "Past request history" refers to a record of orders or requests made by a user in the past.

[0114] "Dialogue generation artificial intelligence" refers to AI technology that allows for natural dialogue with users.

[0115] "Photography device" refers to equipment for capturing images or videos, such as cameras and video equipment.

[0116] "Sensing equipment" refers to a sensor, device, or other equipment used to detect a particular condition.

[0117] "Data protocol" refers to the rules and procedures for exchanging information.

[0118] A "generative artificial intelligence model" refers to an AI model that automatically generates appropriate guidance and responses based on user input.

[0119] An "automated serving system" refers to a system that automatically delivers food, drinks, etc.

[0120] This invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing a customer, a means for interacting with the customer to suggest directions, a means for receiving and confirming the customer's request, a means for transmitting the request information to a cooking department, a means for providing the customer with the completed food, and a means for receiving and responding to the customer's additional requests.

[0121] First, to recognize the user, the device activates a camera (e.g., a general-purpose camera) and a detector (e.g., a multi-use detector) to scan for users sitting at nearby tables. Specifically, the device uses facial recognition software (e.g., general-purpose recognition software) to identify the user and initiates a dialogue with them, saying, "Hello, welcome. Would you like to see the menu?" This dialogue uses voice recognition technology (e.g., general-purpose voice recognition technology).

[0122] Next, if the user asks "What do you recommend?" without selecting a menu item, the device sends that information to the server. The server references raw material inventory information (e.g., a general-purpose inventory management system) and past request history (e.g., a general-purpose database) and generates appropriate guidance using a generative AI model (e.g., a general-purpose AI model). For example, it might generate a suggestion such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp" and send it to the device, which then relays that information to the user.

[0123] When a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal asks, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information is sent to the server. The server then sends the received order information to the cooking department's system (e.g., a general-purpose cooking management system) and issues a command to start cooking.

[0124] When cooking is complete, the kitchen department sends a completion notification to the server, which then sends that information to the terminal. The terminal then delivers the food to the designated table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." A serving robot (e.g., a general-purpose serving robot) is used for this.

[0125] Furthermore, if the customer requests "an additional water, please," the terminal responds, "Understood. We will bring it to you immediately," and sends the additional order information to the server. The server again instructs the kitchen department to respond, or handles the request using an automated service system (e.g., a general-purpose automated service system). The terminal then uses the automated service system to bring the water to the customer's table.

[0126] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[0127] Examples:

[0128] An example of processing when a user conveys a specific request is shown below.

[0129] The device recognizes the user and says, "Hello, welcome."

[0130] A user asks, "What do you recommend?"

[0131] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0132] A customer orders, "I'd like a Japanese-style hamburger, please."

[0133] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0134] The terminal transmits the order information to the server.

[0135] The server sends the order information to the cooking department's system.

[0136] The cooking department system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0137] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0138] Example prompt sentence:

[0139] Generate a response when a customer asks, "What do you recommend?" taking into account the customer's past ordering history and current ingredient availability information.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Program processing flow

[0142] Step 1:

[0143] The device activates its camera and sensors to recognize users sitting at nearby tables.

[0144] Input: Video data acquired by the device through the camera and sensors.

[0145] Data processing: Facial recognition software is used to analyze the video data and identify the user's face.

[0146] Output: User's identity.

[0147] How it works: The device uses a camera to capture video of the area around the table, then processes the video with facial recognition software. If it recognizes a customer, it will begin a conversation with them, saying, "Hello, welcome. Would you like to see the menu?"

[0148] Step 2:

[0149] The terminal guides the user through the menu and asks questions about ordering.

[0150] Input: User response (e.g., "What do you recommend?").

[0151] Data processing: Using voice recognition software, the user's speech is converted into text data.

[0152] Output: The user's request (e.g., "recommendations").

[0153] How it works: The device collects what the user says through a microphone and analyzes it using speech recognition software. Based on the analyzed text, the device sends a message to the server asking, "What do you recommend?"

[0154] Step 3:

[0155] The server references ingredient inventory information and past request history to generate recommended dishes.

[0156] Input: User request details, raw material inventory information, past request history.

[0157] Data processing: Obtaining the necessary information from inventory management systems and databases and using generative AI models to generate appropriate recommendations.

[0158] Output: A list of recommended dishes (e.g. "Japanese-style hamburger steak and fried shrimp").

[0159] Specific operation: The server retrieves current stock information from the inventory management system and references the user's past order history from the database. Using this data, the generative AI model generates a message such as "Today's recommendations are Japanese-style hamburger steak and fried shrimp," and sends it to the terminal.

[0160] Step 4:

[0161] The device then presents the generated recommendations to the user.

[0162] Input: A list of recommended dishes received from the server.

[0163] Data processing: None (output as is).

[0164] Output: Information to the user.

[0165] Specific operation: The device will audibly convey to the user the message, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0166] Step 5:

[0167] The user places an order and the terminal reconfirms the order.

[0168] Input: User's order (e.g., "I'd like a Japanese-style hamburger steak, please").

[0169] Data processing: Analyzes using voice recognition software and sends the order details to the server.

[0170] Output: Confirmed order details and order information sent to the server.

[0171] Specific operation: The terminal analyzes the user's statement, "I'd like a Japanese-style hamburger, please," and reconfirms, "Is a Japanese-style hamburger okay?" If the user responds "Yes," the order details are sent to the server.

[0172] Step 6:

[0173] The server sends the order information to the cooking department and gives cooking instructions.

[0174] Input: Order information received from the terminal.

[0175] Data processing: Sends order information to the cooking department's system.

[0176] Output: Instructions to the cooking department.

[0177] Specific operation: The server sends the order information received from the terminal to the cooking department's system and issues an instruction to start cooking.

[0178] Step 7:

[0179] Once cooking is complete, the terminal serves the food to the user.

[0180] Input: Notification of completion of cooking from the cooking department.

[0181] Data processing: Send a cooking completion notification to the terminal.

[0182] Output: Sends food delivery instructions to the terminal and serves the food to the customer.

[0183] Specific operation: The kitchen department sends a cooking completion notification to the server, which then forwards the information to the terminal. The terminal then uses a serving robot to deliver the food to the specified table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak."

[0184] Step 8:

[0185] Respond to additional user requests.

[0186] Input: The customer's additional request (e.g., "Please give me some more water").

[0187] Data processing: Analyze using voice recognition software and send additional order information to the server.

[0188] Output: Additional request information and instructions to the server.

[0189] Specific operation: The terminal analyzes the user's additional request using voice recognition software and sends it to the server. The server receives the additional request and either instructs the kitchen department to handle it or uses an automated service system. The terminal then uses the automated service system to bring water to the user's table.

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Conventional food delivery systems have difficulty responding quickly to user orders and requests, potentially resulting in a poor customer experience. Furthermore, the interaction between users and the system is limited, making it difficult to smoothly implement menu suggestions and order confirmation processes. Therefore, there was a need for a system that could efficiently respond to diverse customer requests through an interactive interface.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes a means for recognizing a user using a camera and a sensor, a means for suggesting menu items to a customer using a chatbot, a means for sending order information to a database in real time using the customer's smartphone, and a means for notifying the customer of the order status, thereby enabling interactive interaction between the customer and the system and enabling orders and requests to be handled quickly and efficiently.

[0195] "Customer" means an individual or entity receiving services or goods.

[0196] "Recognition" is the process of identifying objects using devices such as cameras and sensors.

[0197] "Dialogue" refers to the exchange of information between a system and a user, including verbal communication.

[0198] A "menu" is a list of available dishes and services.

[0199] An "order" is an act by which a customer requests a company to provide a specific service or product.

[0200] A "kitchen" is a facility or place where food is prepared.

[0201] "Serving" is the act of delivering cooked food to customers.

[0202] "Request" refers to a request by a customer for the provision of additional services or products that the customer desires.

[0203] A "camera" is a device for taking still images and videos.

[0204] A "sensor" is a device that detects and measures changes in the environment.

[0205] A "chatbot" is artificial intelligence software that mimics text or voice interaction.

[0206] A "smartphone" is a mobile device with multiple functions, such as making calls, connecting to the Internet, and running applications.

[0207] "Real time" refers to the instantaneous processing or display of ongoing events.

[0208] A "database" is a structured system for efficiently storing, managing, and retrieving data.

[0209] "Notification" is the act of informing a user of specific information.

[0210] This invention is a system for realizing an interactive ordering application for food delivery services. By incorporating the following elements, this system can receive orders through dialogue with customers and automate everything from serving food to responding to additional requests.

[0211] First, when a customer launches the application using their smartphone, the device uses the smartphone's camera to recognize the customer. Image recognition technology, such as Google® Vision API, is used to identify the user. Once the device recognizes the user, a conversation begins, with a "Hello, welcome" message.

[0212] Next, the device uses its chatbot functionality to suggest menu items. This involves using chatbot APIs such as Google Dialogflow to have a natural conversation with the user. When a customer asks, "What do you recommend?", the server refers to ingredient inventory information and past order history to suggest today's recommended dishes. If the user wants to order a specific dish, the device confirms that information and sends it to the server.

[0213] When the server receives the order information, it sends it to a real-time database (e.g., Firebase Realtime Database) and issues instructions to the kitchen system. Once the order is cooked, the kitchen system notifies the server that the cooking is complete, and the server notifies the customer's smartphone of the delivery progress.

[0214] If the customer makes an additional request (e.g., "Can I have some more water, please?"), the device will again use the chatbot to accept the request and send the information to the server, which will then work with the kitchen system and automatic food distribution system to respond to the request.

[0215] Specific examples

[0216] 1. User Recognition: The user starts up their smartphone and opens the application. The camera recognizes the user's face and begins the conversation with a "Hello, welcome."

[0217] 2. Menu suggestions: When a user asks, "What do you recommend?", the chatbot will suggest, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0218] 3. Order receipt: The user enters "Japanese-style hamburger steak please," and the system reconfirms, "Are you sure you want a Japanese-style hamburger steak?" If the user answers "yes," the order is sent to the server.

[0219] 4. Kitchen integration: Order information is sent to the partner restaurant and cooking begins.

[0220] 5. Delivery Notifications: Your smartphone will send you push notifications to let you know the progress of your delivery.

[0221] 6. Handling additional requests: When a user requests "more water please," the chatbot will accept this and respond.

[0222] Prompt Sentence Examples

[0223] "Give me a scenario where a user opens your app and asks for food recommendations."

[0224] The user opens the smartphone app and asks, "What do you recommend?" The AI ​​chatbot responds, "Today's recommendation is Japanese-style hamburger steak and fried shrimp," based on ingredient availability information and past order history. The user then places an order, "I'd like a Japanese-style hamburger steak, please," and the AI ​​chatbot reconfirms, "Is a Japanese-style hamburger steak okay?"

[0225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0226] Step 1:

[0227] User Awareness

[0228] Input: The user turns on the smartphone and opens the application.

[0229] How it works: The device's camera and sensors are used to recognize the user's face. Specifically, image recognition technology (e.g., Google Vision API) is used to identify the user's face, and that information is sent to the server.

[0230] Output: Display a message ("Hello, welcome") that the user has been recognized.

[0231] Step 2:

[0232] Menu suggestions

[0233] Input: A user query: "What do you recommend?"

[0234] Operation: The device uses a chatbot function (e.g., Google Dialogflow) to communicate with the user. The server references ingredient inventory information and past order history to select today's recommended dishes.

[0235] Output: Display recommended menu items such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0236] Step 3:

[0237] Order receipt and confirmation

[0238] Input: A user orders a specific dish (e.g., "I'd like a Japanese-style hamburger steak, please").

[0239] Operation: The terminal checks the order details and asks again, "Is a Japanese-style hamburger okay?" After receiving the user's response, it sends all the order information to the server.

[0240] Output: A confirmation message of the order details and the final order details are confirmed.

[0241] Step 4:

[0242] Kitchen Cooperation

[0243] Input: Confirmed order information.

[0244] How it works: The server sends the order information to a real-time database (e.g., Firebase Realtime Database), which then passes that information to the kitchen systems of partner restaurants.

[0245] Output: The kitchen starts cooking based on the order and notifies the server after cooking is complete.

[0246] Step 5:

[0247] Delivery notification

[0248] Input: Notification from the kitchen that cooking is complete.

[0249] How it works: The server receives the cooking completion information and notifies the customer's smartphone of the delivery progress. Specifically, it notifies the progress in real time using the push notification function.

[0250] Output: Cooking completion notification and real-time delivery status updates.

[0251] Step 6:

[0252] Additional request response

[0253] Input: Any additional requests from the customer (e.g., "An extra water please.").

[0254] Operation: The terminal again uses the chatbot function to interact with the customer and accept any additional requests. The server then sends this information to the kitchen system and the automatic food distribution system, which then issues instructions on how to respond.

[0255] Output: The customer is notified that the add request has been accepted and the request is executed.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system is further combined with an emotion engine to recognize the customer's emotional state and respond appropriately accordingly. The system includes a means for recognizing customers, a means for suggesting menu items through dialogue with customers, a means for receiving and confirming customer orders, a means for transmitting order information to a kitchen, a means for serving the ordered food to customers, and a means for receiving and responding to additional requests from customers. The emotion engine also has a mechanism for analyzing the user's voice and facial expressions, recognizing their emotional state, and adjusting the content of the dialogue and the service accordingly.

[0258] Program processing

[0259] 1. User Awareness

[0260] The terminal uses cameras and sensors to recognize users seated at tables and confirm that customers have arrived.

[0261] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0262] 2. Emotional Recognition

[0263] The device uses an emotion engine to analyze the user's voice and facial expressions and recognize their emotional state.

[0264] If the user is excited, the device will speak to them in a calmer tone, asking questions such as "Is there anything special that happened today?" to calm the user down.

[0265] 3. Menu suggestions

[0266] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0267] When a user requests a recommendation, the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0268] If the emotion engine recognizes that the user is distressed, it will provide a more detailed explanation or additional suggestions.

[0269] 4. Order Receipt and Confirmation

[0270] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0271] The terminal will ask again, "Would you like a Japanese-style hamburger?" and if the user answers "Yes," the order information will be sent to the server.

[0272] 5. Kitchen collaboration

[0273] The server sends the order information to the kitchen system and issues a command to start cooking.

[0274] The kitchen system receives the order and begins cooking.

[0275] 6. Serving the food

[0276] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[0277] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0278] 7. Response to requests

[0279] The user requests "additional water please."

[0280] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0281] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0282] The terminal will bring the water to the user's table and fulfill their request.

[0283] Specific examples

[0284] Example of what happens when a user communicates a specific request

[0285] The device recognizes the user and says, "Hello, welcome."

[0286] A user asks, "What do you recommend?"

[0287] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0288] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0289] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0290] The terminal transmits the order information to the server.

[0291] The server sends the order information to the kitchen system.

[0292] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0293] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0294] Examples of using the Emotion Engine

[0295] The device recognizes the user and says, "Hello, welcome."

[0296] The emotion engine analyzes the user's facial expressions and recognizes excitement.

[0297] The device speaks in a slightly calmer tone, asking, "Is there anything special happening today?"

[0298] The emotion engine confirms that the user's emotions have calmed down and then proceeds with a suggestion, asking, "Have you decided what to order? Or would you like us to recommend some dishes?"

[0299] This system, which combines an emotion engine in this way, is able to provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[0300] The processing flow will be explained below.

[0301] Step 1:

[0302] The terminal uses cameras and sensors to recognize users sitting at tables, allowing the terminal to know that a customer has arrived.

[0303] Step 2:

[0304] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0305] Step 3:

[0306] The device uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if the user is excited, the device will adjust its response accordingly.

[0307] Step 4:

[0308] The device will ask, "Is there anything special happening today?" in a calmer tone, trying to calm the user's emotions.

[0309] Step 5:

[0310] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[0311] Step 6:

[0312] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0313] Step 7:

[0314] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0315] Step 8:

[0316] If the emotion engine recognizes that the user appears distressed, the device will make an additional suggestion, saying, "If you'd like, I can provide a more detailed explanation."

[0317] Step 9:

[0318] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0319] Step 10:

[0320] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[0321] Step 11:

[0322] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[0323] Step 12:

[0324] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[0325] Step 13:

[0326] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0327] Step 14:

[0328] The user requests "additional water please."

[0329] Step 15:

[0330] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0331] Step 16:

[0332] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0333] Step 17:

[0334] The terminal will bring the water to the user's table and fulfill their request.

[0335] Through this series of steps, the system enables smooth dialogue with customers, ordering, serving food, and responding to requests, reducing the burden on staff while improving the quality of service provided to customers. In addition, the use of an emotion engine enables optimal responses according to the user's emotional state, contributing to improved customer satisfaction.

[0336] Example 2

[0337] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] In traditional restaurant operations, a large number of human resources are required to handle customer orders, serve food, and handle additional requests, resulting in poor operational efficiency and reduced customer satisfaction. It is also difficult for employees to accurately grasp customers' emotional states, often resulting in inconsistent service quality. Furthermore, some systems lack AI-based dialogue and emotion recognition capabilities, limiting customer service. To solve these problems, a multifunctional and efficient system is needed.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0340] In this invention, the server includes means for recognizing a user, means for interacting with a customer and proposing a menu, means for receiving and confirming a customer order, means for transmitting the order information to the cooking department, means for serving the ordered dishes to the customer, means for receiving and responding to additional requests from the customer, means for using an emotion engine that analyzes the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services based on their emotional state, thereby making it possible to improve the efficiency of customer service operations and customer satisfaction.

[0341] A "means for recognizing a user" is a technology, device, or method for detecting the arrival of a customer at a designated location and confirming the customer's presence.

[0342] A "means for interacting with customers and suggesting menu items" is a technology, device, or method for presenting menu items and assisting customers in making selections through conversation with the customer.

[0343] "Means for receiving and confirming customer orders" means the technology, device, or method for receiving and reconfirming customer menu item selections.

[0344] "Means for transmitting order information to the cooking department" refers to technology, devices, or methods for transmitting the order details received from the customer to the cook or cooking system.

[0345] "Means for delivering ordered food to customers" refers to technology, equipment, or methods for automatically delivering food to customers after cooking is complete.

[0346] The "means for accepting and responding to additional requests from customers" refers to a technology, device, or method for accepting additional requests from customers and responding accordingly.

[0347] "Means using an emotion engine that analyzes a user's voice and facial expressions to recognize their emotional state" refers to a technology, device, or method that analyzes a user's speech and facial expressions and identifies the user's emotions from the results.

[0348] The "means for adjusting dialogue content or services based on emotional state" refers to a technology, device, or method for dynamically changing the dialogue content or service content provided based on the results of recognizing the user's emotional state.

[0349] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. Furthermore, by combining this system with an emotion engine, it is possible to recognize the emotional state of the customer and respond appropriately accordingly. This system comprises the following means:

[0350] 1. User recognition methods:

[0351] The terminal uses a camera (e.g., a generic name "camera device") and a sensor (e.g., "sensor device") to recognize a user sitting at a table. Image recognition software (e.g., "image analysis software") is used to recognize the user. This confirms that a customer has arrived, and the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?". Speech recognition and generation use a speech recognition API and a speech generation API (e.g., "speech processing API").

[0352] 2. Emotion recognition means:

[0353] The device uses an emotion engine (e.g., "emotion analysis software") to analyze the user's voice and facial expressions and recognize their emotional state. If the user is excited, the device can speak in a calmer tone, asking, "Is there anything special happening today?" This adjusts the conversation with the customer to proceed smoothly.

[0354] 3. Menu suggestion methods:

[0355] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" If the user requests a recommendation, the server references ingredient inventory information and the user's past order history. Based on this information, a chat generation AI (for example, a "dialogue generation AI") is used to suggest, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." If the emotion engine recognizes that the user seems troubled, the device will provide a detailed explanation.

[0356] 4. Order receipt and confirmation methods:

[0357] If a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information will be sent to the server in JSON format via the HTTP API.

[0358] 5. Kitchen collaboration methods:

[0359] The server sends the order information to the cooking department's system (e.g., a "cooking management system") and instructs it to start cooking. The cooking management system receives the order and starts cooking. The order information is displayed on a display in the cooking department.

[0360] 6. Food Serving Method:

[0361] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. The terminal then carries the food to the customer, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." It is also possible to use an automatic robot (for example, an "autonomous mobile robot") to serve the food.

[0362] 7. How to respond to requests:

[0363] If a user requests "an extra glass of water, please," the device will use voice recognition to understand the request and respond with "I understand, I'll bring it right away." The additional order information is sent to the server, which again instructs the cooking system to respond. If necessary, an automatic food distribution system can also be used. The device will respond quickly to additional requests, increasing customer satisfaction.

[0364] Specific examples

[0365] If the user has a specific request, the following might happen:

[0366] 1. The device recognizes the user and says, "Hello, welcome."

[0367] 2. A user asks, "What do you recommend?"

[0368] 3. The device responds, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0369] 4. The user places an order, "I'd like a Japanese-style hamburger, please."

[0370] 5. The terminal will ask again, "Would you like a Japanese-style hamburger?" and the user will answer "Yes."

[0371] 6. The terminal sends the order information to the server.

[0372] 7. The server sends the order information to the cooking management system.

[0373] 8. The cooking management system notifies the terminal that the food is ready, and the server issues serving instructions to the terminal.

[0374] 9. The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[0375] Example prompt sentence:

[0376] "Hello, welcome. Would you like to see the menu?"

[0377] "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0378] "Would you like a Japanese-style hamburger?"

[0379] In this way, this system, which combines an emotion engine, can provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[0380] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0381] Step 1:

[0382] Recognize the user

[0383] The terminal uses a camera and sensor device to recognize the user sitting at the table. It receives camera images and sensor information as input and uses image analysis software to confirm the user's presence. As output, it generates data indicating that the user has been recognized. Specifically, the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?"

[0384] Step 2:

[0385] Emotion recognition

[0386] The device uses emotion analysis software to analyze the user's voice and facial expressions and recognize their emotional state. The user's voice and image data are taken as input and processed by the emotion analysis software. The output is the user's emotional state (e.g., excited, calm, anxious). Specifically, if the user is excited, the device will ask, "Did anything special happen today?" in a slightly calmer tone.

[0387] Step 3:

[0388] Menu suggestions

[0389] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" The device takes the user's response as input. When the user requests a recommendation, the server executes a database query to look up ingredient inventory information and past order history. As output, the conversation generation AI suggests, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." Specifically, if the emotion analysis software recognizes that the user appears to be struggling, the device will provide a detailed explanation.

[0390] Step 4:

[0391] Order receipt and confirmation

[0392] When the user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" As input, the voice recognition API converts the user's utterance into text data and generates a confirmation question. As output, the user's final confirmation is obtained and order information is generated in JSON format. The specific operation is to send the order information to the server.

[0393] Step 5:

[0394] Kitchen Cooperation

[0395] The server sends the order information to the cooking management system and instructs it to start cooking. As input, the order information received by the server is analyzed and passed to the cooking management system. As output, the order details are displayed on the cooking management system and cooking begins. As a specific operation, the order details are displayed on a display in the cooking department.

[0396] Step 6:

[0397] Serving food

[0398] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. As input, it receives the cooking completion notification and issues a distribution instruction to the terminal. As output, it generates data indicating that the food will be served. Specifically, the terminal brings the food while saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[0399] Step 7:

[0400] Responding to requests

[0401] If a user requests "an extra glass of water, please," the device uses voice recognition to understand the request and responds with "I understand, I'll bring it right away." As input, the device acquires and analyzes the voice data of the additional request. As output, additional order information is generated and sent to the server. Specifically, the server again instructs the cooking management system to respond, and the automatic food distribution system is used as necessary. The device then brings the water to the user's table, fulfilling the request.

[0402] (Application example 2)

[0403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] In order to increase customer satisfaction and provide efficient customer service in virtual stores, conventional systems have difficulty responding to customers' emotional states. This is because they are unable to adequately respond to customers' anxieties and questions in an online shopping environment. Furthermore, conventional systems lack the means to quickly and appropriately respond to customers' additional requests. Furthermore, conventional systems make it difficult to provide individually customized experiences, raising concerns about a decline in customer satisfaction.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0406] In this invention, the server includes means for recognizing customers, means for interacting with customers and suggesting menu items, means for receiving and confirming customer orders, means for transmitting order information to cooking facilities, means for providing customers with ordered items, means for receiving and responding to customer requests for additional items, means for analyzing the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services according to their emotional state, thereby enabling the provision of customized services according to the customer's emotional state.

[0407] "Customer recognition means" refers to technology that uses devices such as cameras and sensors to identify users and detect their presence.

[0408] "Means of interacting with customers and suggesting menu items" refers to technologies such as voice dialogue systems and chatbots that interact with users to provide appropriate options.

[0409] "Means for receiving and confirming customer orders" refers to the interface and processing system for receiving and confirming user selections for goods and services.

[0410] The "means for transmitting order information to the cooking facility" refers to a communication technology or protocol for transmitting the user's order details to the cooking facility.

[0411] The "means of providing customers with ordered products" refers to a logistics system and actuator technology for reliably delivering ordered products to users.

[0412] The "means for accepting and responding to additional requests from customers" refers to an interface and processing system for quickly responding to additional requests and changes from users.

[0413] "Means for analyzing the user's voice and facial expressions and recognizing their emotional state" refers to image and voice recognition technology that analyzes the user's voice and facial expression data to determine their emotional state.

[0414] "Means for adjusting dialogue content and services according to emotional state" refers to technology for flexibly changing the content of the services and dialogue provided based on the recognized emotional state.

[0415] In this invention, the following hardware and software are used to realize a virtual store system that recognizes customers and provides customized dialogue and services according to their emotional state.

[0416] Hardware and Software Configuration

[0417] Hardware

[0418] Smartphones: Equipped with a camera and microphone, they can recognize users and accept voice input.

[0419] Head-mounted display (HMD): A device for viewing and interacting with a virtual environment.

[0420] software

[0421] Image and audio analysis library: Uses OpenCV, Google ML Kit, etc. to recognize users' faces and analyze their audio.

[0422] Emotion Recognition Engine: Uses the Emotion API to recognize the user's emotional state by analyzing their voice and facial expressions.

[0423] Generative chat AI: Generates dialogue using GPT models and other natural language generation models.

[0424] Web server: Uses Node.js and handles all communication and data management.

[0425] System Operation

[0426] User Awareness

[0427] First, the device recognizes the user using its camera and microphone. It uses OpenCV to analyze the video data from the camera and perform facial recognition. It also captures audio through the microphone and converts the user's speech into text using a speech analysis function such as Google ML Kit. This allows the device to confirm that the user is a new visitor and begin a conversation by asking, "Hello, welcome. What are you looking for today?"

[0428] Emotion recognition

[0429] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. For example, if the user is nervous, the device will speak in a calmer tone, asking, "Did anything special happen today?" In this way, the device adjusts the content of the conversation according to the user's emotional state.

[0430] Menu suggestions

[0431] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and a 4K smart TV" is generated.

[0432] Order receipt and confirmation

[0433] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server. The server then sends the order information to the food preparation facility or logistics system, and the product is prepared.

[0434] Responding to requests

[0435] Similarly, if the user makes an additional request such as "I would like gift wrapping," the terminal accepts the request and the server makes corresponding arrangements based on that request.

[0436] Examples of concrete examples and prompts

[0437] When a user enters a virtual store

[0438] User: "Can you recommend some products?"

[0439] Assistant: "Welcome! Today's recommendations are the latest Bluetooth earphones and a 4K smart TV. Which one are you interested in?"

[0440] Example of confirmation when a customer decides to purchase

[0441] Assistant: "Are you buying Bluetooth earphones? Understood. Would you like gift wrapping?"

[0442] User: "Yes, please."

[0443] Example prompts using the Emotion Engine

[0444] When the user is nervous

[0445] Assistant: "Is there anything special happening today? Let's talk for a bit and get settled. What product are you looking for?"

[0446] In this way, by using pre-set prompts, it becomes possible to provide customized services in virtual stores that correspond to the user's emotions and needs.

[0447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0448] Step 1: User Awareness

[0449] The device uses a camera and microphone to recognize the user's face and voice. Specifically, it processes the video data from the camera using OpenCV to perform facial recognition. It also analyzes the audio data collected by the microphone using Google ML Kit and converts the user's speech into text. Based on this data, the device begins a conversation by asking, "Hello, welcome. What are you looking for today?"

[0450] Input: Camera video data and audio data

[0451] Output: User's face recognition results and voice-to-text data

[0452] Step 2: Emotion Recognition

[0453] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. Specifically, audio and video data is input into the Emotion API, and the resulting emotional state is received as the analysis result. If the user is nervous, the device will speak in a calm tone, asking, "Is there anything special that happened today?"

[0454] Input: Audio and video data

[0455] Output: Emotion recognition results (e.g., nervousness, excitement, joy, etc.)

[0456] Step 3: Menu proposal

[0457] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and 4K smart TVs" is generated.

[0458] Input: User requests, inventory information, past order history

[0459] Output: Customized product suggestions

[0460] Step 4: Order receipt and confirmation

[0461] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server, which then sends the order information to the food preparation facility or logistics system.

[0462] Input: User's order details

[0463] Output: Confirmed order information, data sent to the server

[0464] Step 5: Response to requests

[0465] If the user makes an additional request, such as "I would like gift wrapping," the terminal accepts the request and the server arranges for the response based on that request. Specifically, the content of the additional request is converted into text and sent to the server. The server then carries out the necessary procedures and notifies the user of the final confirmation.

[0466] Input: Content of additional request

[0467] Output: Notification and confirmation message based on request

[0468] In this way, each step is executed sequentially to provide a customized service to the user.

[0469] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0471] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0472] [Second embodiment]

[0473] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0474] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0476] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0480] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0481] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0482] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0483] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0484] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0485] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing customers, a means for interacting with customers to suggest menu items, a means for receiving and confirming customer orders, a means for transmitting order information to the kitchen, a means for serving ordered dishes to customers, and a means for receiving and responding to additional requests from customers. The specific program processing is explained below in natural language.

[0486] Program processing

[0487] 1. User Awareness

[0488] The device uses a camera and sensors to recognize users sitting at the table.

[0489] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0490] 2. Menu suggestions

[0491] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0492] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0493] 3. Order Receipt and Confirmation

[0494] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0495] The terminal will reconfirm "Would you like a Japanese-style hamburger?" and if the user answers "Yes", it will send the order information to the server.

[0496] 4. Kitchen collaboration

[0497] The server sends the order information to the kitchen system and issues an instruction to start cooking.

[0498] 5. Serving the food

[0499] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[0500] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0501] 6. Response to requests

[0502] The user requests "additional water please."

[0503] The terminal responds, "Understood. We'll bring it to you right away," and sends additional order information to the server.

[0504] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0505] The terminal delivers the water to the user's table.

[0506] Specific examples

[0507] Example of what happens when a user communicates a specific request

[0508] The device recognizes the user and says, "Hello, welcome."

[0509] A user asks, "What do you recommend?"

[0510] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0511] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0512] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0513] The terminal transmits the order information to the server.

[0514] The server sends the order information to the kitchen system.

[0515] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0516] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0517] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[0518] The processing flow will be explained below.

[0519] Step 1:

[0520] The terminal uses a camera and sensors to recognize the user sitting at a table, allowing the terminal to know that a customer has arrived at the restaurant.

[0521] Step 2:

[0522] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0523] Step 3:

[0524] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[0525] Step 4:

[0526] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0527] Step 5:

[0528] When the user responds, "Please tell me your recommendation," the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0529] Step 6:

[0530] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0531] Step 7:

[0532] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[0533] Step 8:

[0534] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[0535] Step 9:

[0536] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[0537] Step 10:

[0538] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0539] Step 11:

[0540] The user requests "additional water please."

[0541] Step 12:

[0542] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0543] Step 13:

[0544] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0545] Step 14:

[0546] The terminal will bring the water to the user's table and fulfill their request.

[0547] Example 1

[0548] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0549] In conventional restaurants, the process of receiving an order from a customer, transmitting it to the kitchen, and serving the food required a great deal of time and effort. It was also difficult to respond quickly to additional customer requests, which could lead to a decline in the quality of service. This led to issues such as lower customer satisfaction and a deterioration in store operational efficiency.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0551] In this invention, the server includes a means for recognizing a user, a means for interacting with the user to provide guidance, a means for receiving and confirming a user's request, a means for transmitting the request information to the cooking department, a means for providing the user with the requested food, and a means for receiving and responding to additional requests from the user. This automates the entire process from when the user enters the store to when the user completes their order, enabling fast and efficient service. Furthermore, by referencing past request history and ingredient inventory information, guidance suggestions based on the user's preferences can be made, and by using a dialogue generation AI, orders and requests can be accepted through natural dialogue. This increases user satisfaction and improves the efficiency of store operations.

[0552] "User" refers to a customer who receives services using the system.

[0553] "Means for recognizing" refers to means for identifying a user using a photographing device or a detecting device.

[0554] "Means for suggesting guidance through dialogue" refers to means for communicating with users via voice or text and suggesting appropriate menus or services.

[0555] "Means for receiving and confirming requests" refers to the means for receiving and confirming orders or requests from users.

[0556] "Means for transmitting request information to the cooking department" refers to means for transmitting received orders and requests to the cooking department.

[0557] "Means for providing requested food" refers to the means for delivering cooked food to the customer.

[0558] "Means for accepting and responding to additional requests" refers to means for accepting additional requests from users and providing services in response to those requests.

[0559] "Raw material inventory information" refers to data showing the inventory status of food ingredients, beverages, etc.

[0560] "Past request history" refers to a record of orders or requests made by a user in the past.

[0561] "Dialogue generation artificial intelligence" refers to AI technology that allows for natural dialogue with users.

[0562] "Photography device" refers to equipment for capturing images or videos, such as cameras and video equipment.

[0563] "Sensing equipment" refers to a sensor, device, or other equipment used to detect a particular condition.

[0564] "Data protocol" refers to the rules and procedures for exchanging information.

[0565] A "generative artificial intelligence model" refers to an AI model that automatically generates appropriate guidance and responses based on user input.

[0566] An "automated serving system" refers to a system that automatically delivers food, drinks, etc.

[0567] This invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing a customer, a means for interacting with the customer to suggest directions, a means for receiving and confirming the customer's request, a means for transmitting the request information to a cooking department, a means for providing the customer with the completed food, and a means for receiving and responding to the customer's additional requests.

[0568] First, to recognize the user, the device activates a camera (e.g., a general-purpose camera) and a detector (e.g., a multi-use detector) to scan for users sitting at nearby tables. Specifically, the device uses facial recognition software (e.g., general-purpose recognition software) to identify the user and initiates a dialogue with them, saying, "Hello, welcome. Would you like to see the menu?" This dialogue uses voice recognition technology (e.g., general-purpose voice recognition technology).

[0569] Next, if the user asks "What do you recommend?" without selecting a menu item, the device sends that information to the server. The server references raw material inventory information (e.g., a general-purpose inventory management system) and past request history (e.g., a general-purpose database) and generates appropriate guidance using a generative AI model (e.g., a general-purpose AI model). For example, it might generate a suggestion such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp" and send it to the device, which then relays that information to the user.

[0570] When a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal asks, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information is sent to the server. The server then sends the received order information to the cooking department's system (e.g., a general-purpose cooking management system) and issues a command to start cooking.

[0571] When cooking is complete, the kitchen department sends a completion notification to the server, which then sends that information to the terminal. The terminal then delivers the food to the designated table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." A serving robot (e.g., a general-purpose serving robot) is used for this.

[0572] Furthermore, if the customer requests "an additional water, please," the terminal responds, "Understood. We will bring it to you immediately," and sends the additional order information to the server. The server again instructs the kitchen department to respond, or handles the request using an automated service system (e.g., a general-purpose automated service system). The terminal then uses the automated service system to bring the water to the customer's table.

[0573] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[0574] Examples:

[0575] An example of processing when a user conveys a specific request is shown below.

[0576] The device recognizes the user and says, "Hello, welcome."

[0577] A user asks, "What do you recommend?"

[0578] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0579] A customer orders, "I'd like a Japanese-style hamburger, please."

[0580] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0581] The terminal transmits the order information to the server.

[0582] The server sends the order information to the cooking department's system.

[0583] The cooking department system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0584] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0585] Example prompt sentence:

[0586] Generate a response when a customer asks, "What do you recommend?" taking into account the customer's past ordering history and current ingredient availability information.

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Program processing flow

[0589] Step 1:

[0590] The device activates its camera and sensors to recognize users sitting at nearby tables.

[0591] Input: Video data acquired by the device through the camera and sensors.

[0592] Data processing: Facial recognition software is used to analyze the video data and identify the user's face.

[0593] Output: User's identity.

[0594] How it works: The device uses a camera to capture video of the area around the table, then processes the video with facial recognition software. If it recognizes a customer, it will begin a conversation with them, saying, "Hello, welcome. Would you like to see the menu?"

[0595] Step 2:

[0596] The terminal guides the user through the menu and asks questions about ordering.

[0597] Input: User response (e.g., "What do you recommend?").

[0598] Data processing: Using voice recognition software, the user's speech is converted into text data.

[0599] Output: The user's request (e.g., "recommendations").

[0600] How it works: The device collects what the user says through a microphone and analyzes it using speech recognition software. Based on the analyzed text, the device sends a message to the server asking, "What do you recommend?"

[0601] Step 3:

[0602] The server references ingredient inventory information and past request history to generate recommended dishes.

[0603] Input: User request details, raw material inventory information, past request history.

[0604] Data processing: Obtaining the necessary information from inventory management systems and databases and using generative AI models to generate appropriate recommendations.

[0605] Output: A list of recommended dishes (e.g. "Japanese-style hamburger steak and fried shrimp").

[0606] Specific operation: The server retrieves current stock information from the inventory management system and references the user's past order history from the database. Using this data, the generative AI model generates a message such as "Today's recommendations are Japanese-style hamburger steak and fried shrimp," and sends it to the terminal.

[0607] Step 4:

[0608] The device then presents the generated recommendations to the user.

[0609] Input: A list of recommended dishes received from the server.

[0610] Data processing: None (output as is).

[0611] Output: Information to the user.

[0612] Specific operation: The device will audibly convey to the user the message, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0613] Step 5:

[0614] The user places an order and the terminal reconfirms the order.

[0615] Input: User's order (e.g., "I'd like a Japanese-style hamburger steak, please").

[0616] Data processing: Analyzes using voice recognition software and sends the order details to the server.

[0617] Output: Confirmed order details and order information sent to the server.

[0618] Specific operation: The terminal analyzes the user's statement, "I'd like a Japanese-style hamburger, please," and reconfirms, "Is a Japanese-style hamburger okay?" If the user responds "Yes," the order details are sent to the server.

[0619] Step 6:

[0620] The server sends the order information to the cooking department and gives cooking instructions.

[0621] Input: Order information received from the terminal.

[0622] Data processing: Sends order information to the cooking department's system.

[0623] Output: Instructions to the cooking department.

[0624] Specific operation: The server sends the order information received from the terminal to the cooking department's system and issues an instruction to start cooking.

[0625] Step 7:

[0626] Once cooking is complete, the terminal serves the food to the user.

[0627] Input: Notification of completion of cooking from the cooking department.

[0628] Data processing: Send a cooking completion notification to the terminal.

[0629] Output: Sends food delivery instructions to the terminal and serves the food to the customer.

[0630] Specific operation: The kitchen department sends a cooking completion notification to the server, which then forwards the information to the terminal. The terminal then uses a serving robot to deliver the food to the specified table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak."

[0631] Step 8:

[0632] Respond to additional user requests.

[0633] Input: The customer's additional request (e.g., "Please give me some more water").

[0634] Data processing: Analyze using voice recognition software and send additional order information to the server.

[0635] Output: Additional request information and instructions to the server.

[0636] Specific operation: The terminal analyzes the user's additional request using voice recognition software and sends it to the server. The server receives the additional request and either instructs the kitchen department to handle it or uses an automated service system. The terminal then uses the automated service system to bring water to the user's table.

[0637] (Application example 1)

[0638] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0639] Conventional food delivery systems have difficulty responding quickly to user orders and requests, potentially resulting in a poor customer experience. Furthermore, the interaction between users and the system is limited, making it difficult to smoothly implement menu suggestions and order confirmation processes. Therefore, there was a need for a system that could efficiently respond to diverse customer requests through an interactive interface.

[0640] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0641] In this invention, the server includes a means for recognizing a user using a camera and a sensor, a means for suggesting menu items to a customer using a chatbot, a means for sending order information to a database in real time using the customer's smartphone, and a means for notifying the customer of the order status, thereby enabling interactive interaction between the customer and the system and enabling orders and requests to be handled quickly and efficiently.

[0642] "Customer" means an individual or entity receiving services or goods.

[0643] "Recognition" is the process of identifying objects using devices such as cameras and sensors.

[0644] "Dialogue" refers to the exchange of information between a system and a user, including verbal communication.

[0645] A "menu" is a list of available dishes and services.

[0646] An "order" is an act by which a customer requests a company to provide a specific service or product.

[0647] A "kitchen" is a facility or place where food is prepared.

[0648] "Serving" is the act of delivering cooked food to customers.

[0649] "Request" refers to a request by a customer for the provision of additional services or products that the customer desires.

[0650] A "camera" is a device for taking still images and videos.

[0651] A "sensor" is a device that detects and measures changes in the environment.

[0652] A "chatbot" is artificial intelligence software that mimics text or voice interaction.

[0653] A "smartphone" is a mobile device with multiple functions, such as making calls, connecting to the Internet, and running applications.

[0654] "Real time" refers to the instantaneous processing or display of ongoing events.

[0655] A "database" is a structured system for efficiently storing, managing, and retrieving data.

[0656] "Notification" is the act of informing a user of specific information.

[0657] This invention is a system for realizing an interactive ordering application for food delivery services. By incorporating the following elements, this system can receive orders through dialogue with customers and automate everything from serving food to responding to additional requests.

[0658] First, when a customer launches the application using their smartphone, the device uses the smartphone's camera to recognize the customer. Image recognition technology, such as Google Vision API, is used to identify the user. Once the device recognizes the user, a conversation begins with "Hello, welcome."

[0659] Next, the device uses its chatbot functionality to suggest menu items. This involves using chatbot APIs such as Google Dialogflow to have a natural conversation with the user. When a customer asks, "What do you recommend?", the server refers to ingredient inventory information and past order history to suggest today's recommended dishes. If the user wants to order a specific dish, the device confirms that information and sends it to the server.

[0660] When the server receives the order information, it sends it to a real-time database (e.g., Firebase Realtime Database) and issues instructions to the kitchen system. Once the order is cooked, the kitchen system notifies the server that the cooking is complete, and the server notifies the customer's smartphone of the delivery progress.

[0661] If the customer makes an additional request (e.g., "Can I have some more water, please?"), the device will again use the chatbot to accept the request and send the information to the server, which will then work with the kitchen system and automatic food distribution system to respond to the request.

[0662] Specific examples

[0663] 1. User Recognition: The user starts up their smartphone and opens the application. The camera recognizes the user's face and begins the conversation with a "Hello, welcome."

[0664] 2. Menu suggestions: When a user asks, "What do you recommend?", the chatbot will suggest, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0665] 3. Order receipt: The user enters "Japanese-style hamburger steak please," and the system reconfirms, "Are you sure you want a Japanese-style hamburger steak?" If the user answers "yes," the order is sent to the server.

[0666] 4. Kitchen integration: Order information is sent to the partner restaurant and cooking begins.

[0667] 5. Delivery Notifications: Your smartphone will send you push notifications to let you know the progress of your delivery.

[0668] 6. Handling additional requests: When a user requests "more water please," the chatbot will accept this and respond.

[0669] Prompt Sentence Examples

[0670] "Give me a scenario where a user opens your app and asks for food recommendations."

[0671] The user opens the smartphone app and asks, "What do you recommend?" The AI ​​chatbot responds, "Today's recommendation is Japanese-style hamburger steak and fried shrimp," based on ingredient availability information and past order history. The user then places an order, "I'd like a Japanese-style hamburger steak, please," and the AI ​​chatbot reconfirms, "Is a Japanese-style hamburger steak okay?"

[0672] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0673] Step 1:

[0674] User Awareness

[0675] Input: The user turns on the smartphone and opens the application.

[0676] How it works: The device's camera and sensors are used to recognize the user's face. Specifically, image recognition technology (e.g., Google Vision API) is used to identify the user's face, and that information is sent to the server.

[0677] Output: Display a message ("Hello, welcome") that the user has been recognized.

[0678] Step 2:

[0679] Menu suggestions

[0680] Input: A user query: "What do you recommend?"

[0681] Operation: The device uses a chatbot function (e.g., Google Dialogflow) to communicate with the user. The server references ingredient inventory information and past order history to select today's recommended dishes.

[0682] Output: Display recommended menu items such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0683] Step 3:

[0684] Order receipt and confirmation

[0685] Input: A user orders a specific dish (e.g., "I'd like a Japanese-style hamburger steak, please").

[0686] Operation: The terminal checks the order details and asks again, "Is a Japanese-style hamburger okay?" After receiving the user's response, it sends all the order information to the server.

[0687] Output: A confirmation message of the order details and the final order details are confirmed.

[0688] Step 4:

[0689] Kitchen Cooperation

[0690] Input: Confirmed order information.

[0691] How it works: The server sends the order information to a real-time database (e.g., Firebase Realtime Database), which then passes that information to the kitchen systems of partner restaurants.

[0692] Output: The kitchen starts cooking based on the order and notifies the server after cooking is complete.

[0693] Step 5:

[0694] Delivery notification

[0695] Input: Notification from the kitchen that cooking is complete.

[0696] How it works: The server receives the cooking completion information and notifies the customer's smartphone of the delivery progress. Specifically, it notifies the progress in real time using the push notification function.

[0697] Output: Cooking completion notification and real-time delivery status updates.

[0698] Step 6:

[0699] Additional request response

[0700] Input: Any additional requests from the customer (e.g., "An extra water please.").

[0701] Operation: The terminal again uses the chatbot function to interact with the customer and accept any additional requests. The server then sends this information to the kitchen system and the automatic food distribution system, which then issues instructions on how to respond.

[0702] Output: The customer is notified that the add request has been accepted and the request is executed.

[0703] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0704] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system is further combined with an emotion engine to recognize the customer's emotional state and respond appropriately accordingly. The system includes a means for recognizing customers, a means for suggesting menu items through dialogue with customers, a means for receiving and confirming customer orders, a means for transmitting order information to a kitchen, a means for serving the ordered food to customers, and a means for receiving and responding to additional requests from customers. The emotion engine also has a mechanism for analyzing the user's voice and facial expressions, recognizing their emotional state, and adjusting the content of the dialogue and the service accordingly.

[0705] Program processing

[0706] 1. User Awareness

[0707] The terminal uses cameras and sensors to recognize users seated at tables and confirm that customers have arrived.

[0708] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0709] 2. Emotional Recognition

[0710] The device uses an emotion engine to analyze the user's voice and facial expressions and recognize their emotional state.

[0711] If the user is excited, the device will speak to them in a calmer tone, asking questions such as "Is there anything special that happened today?" to calm the user down.

[0712] 3. Menu suggestions

[0713] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0714] When a user requests a recommendation, the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0715] If the emotion engine recognizes that the user is distressed, it will provide a more detailed explanation or additional suggestions.

[0716] 4. Order Receipt and Confirmation

[0717] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0718] The terminal will ask again, "Would you like a Japanese-style hamburger?" and if the user answers "Yes," the order information will be sent to the server.

[0719] 5. Kitchen collaboration

[0720] The server sends the order information to the kitchen system and issues a command to start cooking.

[0721] The kitchen system receives the order and begins cooking.

[0722] 6. Serving the food

[0723] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[0724] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0725] 7. Response to requests

[0726] The user requests "additional water please."

[0727] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0728] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0729] The terminal will bring the water to the user's table and fulfill their request.

[0730] Specific examples

[0731] Example of what happens when a user communicates a specific request

[0732] The device recognizes the user and says, "Hello, welcome."

[0733] A user asks, "What do you recommend?"

[0734] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0735] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0736] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0737] The terminal transmits the order information to the server.

[0738] The server sends the order information to the kitchen system.

[0739] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0740] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0741] Examples of using the Emotion Engine

[0742] The device recognizes the user and says, "Hello, welcome."

[0743] The emotion engine analyzes the user's facial expressions and recognizes excitement.

[0744] The device speaks in a slightly calmer tone, asking, "Is there anything special happening today?"

[0745] The emotion engine confirms that the user's emotions have calmed down and then proceeds with a suggestion, asking, "Have you decided what to order? Or would you like us to recommend some dishes?"

[0746] This system, which combines an emotion engine in this way, is able to provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[0747] The processing flow will be explained below.

[0748] Step 1:

[0749] The terminal uses cameras and sensors to recognize users sitting at tables, allowing the terminal to know that a customer has arrived.

[0750] Step 2:

[0751] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0752] Step 3:

[0753] The device uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if the user is excited, the device will adjust its response accordingly.

[0754] Step 4:

[0755] The device will ask, "Is there anything special happening today?" in a calmer tone, trying to calm the user's emotions.

[0756] Step 5:

[0757] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[0758] Step 6:

[0759] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0760] Step 7:

[0761] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0762] Step 8:

[0763] If the emotion engine recognizes that the user appears distressed, the device will make an additional suggestion, saying, "If you'd like, I can provide a more detailed explanation."

[0764] Step 9:

[0765] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0766] Step 10:

[0767] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[0768] Step 11:

[0769] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[0770] Step 12:

[0771] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[0772] Step 13:

[0773] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0774] Step 14:

[0775] The user requests "additional water please."

[0776] Step 15:

[0777] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0778] Step 16:

[0779] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0780] Step 17:

[0781] The terminal will bring the water to the user's table and fulfill their request.

[0782] Through this series of steps, the system enables smooth dialogue with customers, ordering, serving food, and responding to requests, reducing the burden on staff while improving the quality of service provided to customers. In addition, the use of an emotion engine enables optimal responses according to the user's emotional state, contributing to improved customer satisfaction.

[0783] Example 2

[0784] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0785] In traditional restaurant operations, a large number of human resources are required to handle customer orders, serve food, and handle additional requests, resulting in poor operational efficiency and reduced customer satisfaction. It is also difficult for employees to accurately grasp customers' emotional states, often resulting in inconsistent service quality. Furthermore, some systems lack AI-based dialogue and emotion recognition capabilities, limiting customer service. To solve these problems, a multifunctional and efficient system is needed.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0787] In this invention, the server includes means for recognizing a user, means for interacting with a customer and proposing a menu, means for receiving and confirming a customer order, means for transmitting the order information to the cooking department, means for serving the ordered dishes to the customer, means for receiving and responding to additional requests from the customer, means for using an emotion engine that analyzes the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services based on their emotional state, thereby making it possible to improve the efficiency of customer service operations and customer satisfaction.

[0788] A "means for recognizing a user" is a technology, device, or method for detecting the arrival of a customer at a designated location and confirming the customer's presence.

[0789] A "means for interacting with customers and suggesting menu items" is a technology, device, or method for presenting menu items and assisting customers in making selections through conversation with the customer.

[0790] "Means for receiving and confirming customer orders" means the technology, device, or method for receiving and reconfirming customer menu item selections.

[0791] "Means for transmitting order information to the cooking department" refers to technology, devices, or methods for transmitting the order details received from the customer to the cook or cooking system.

[0792] "Means for delivering ordered food to customers" refers to technology, equipment, or methods for automatically delivering food to customers after cooking is complete.

[0793] The "means for accepting and responding to additional requests from customers" refers to a technology, device, or method for accepting additional requests from customers and responding accordingly.

[0794] "Means using an emotion engine that analyzes a user's voice and facial expressions to recognize their emotional state" refers to a technology, device, or method that analyzes a user's speech and facial expressions and identifies the user's emotions from the results.

[0795] The "means for adjusting dialogue content or services based on emotional state" refers to a technology, device, or method for dynamically changing the dialogue content or service content provided based on the results of recognizing the user's emotional state.

[0796] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. Furthermore, by combining this system with an emotion engine, it is possible to recognize the emotional state of the customer and respond appropriately accordingly. This system comprises the following means:

[0797] 1. User recognition methods:

[0798] The terminal uses a camera (e.g., a generic name "camera device") and a sensor (e.g., "sensor device") to recognize a user sitting at a table. Image recognition software (e.g., "image analysis software") is used to recognize the user. This confirms that a customer has arrived, and the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?". Speech recognition and generation use a speech recognition API and a speech generation API (e.g., "speech processing API").

[0799] 2. Emotion recognition means:

[0800] The device uses an emotion engine (e.g., "emotion analysis software") to analyze the user's voice and facial expressions and recognize their emotional state. If the user is excited, the device can speak in a calmer tone, asking, "Is there anything special happening today?" This adjusts the conversation with the customer to proceed smoothly.

[0801] 3. Menu suggestion methods:

[0802] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" If the user requests a recommendation, the server references ingredient inventory information and the user's past order history. Based on this information, a chat generation AI (for example, a "dialogue generation AI") is used to suggest, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." If the emotion engine recognizes that the user seems troubled, the device will provide a detailed explanation.

[0803] 4. Order receipt and confirmation methods:

[0804] If a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information will be sent to the server in JSON format via the HTTP API.

[0805] 5. Kitchen collaboration methods:

[0806] The server sends the order information to the cooking department's system (e.g., a "cooking management system") and instructs it to start cooking. The cooking management system receives the order and starts cooking. The order information is displayed on a display in the cooking department.

[0807] 6. Food Serving Method:

[0808] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. The terminal then carries the food to the customer, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." It is also possible to use an automatic robot (for example, an "autonomous mobile robot") to serve the food.

[0809] 7. How to respond to requests:

[0810] If a user requests "an extra glass of water, please," the device will use voice recognition to understand the request and respond with "I understand, I'll bring it right away." The additional order information is sent to the server, which again instructs the cooking system to respond. If necessary, an automatic food distribution system can also be used. The device will respond quickly to additional requests, increasing customer satisfaction.

[0811] Specific examples

[0812] If the user has a specific request, the following might happen:

[0813] 1. The device recognizes the user and says, "Hello, welcome."

[0814] 2. A user asks, "What do you recommend?"

[0815] 3. The device responds, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0816] 4. The user places an order, "I'd like a Japanese-style hamburger, please."

[0817] 5. The terminal will ask again, "Would you like a Japanese-style hamburger?" and the user will answer "Yes."

[0818] 6. The terminal sends the order information to the server.

[0819] 7. The server sends the order information to the cooking management system.

[0820] 8. The cooking management system notifies the terminal that the food is ready, and the server issues serving instructions to the terminal.

[0821] 9. The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[0822] Example prompt sentence:

[0823] "Hello, welcome. Would you like to see the menu?"

[0824] "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0825] "Would you like a Japanese-style hamburger?"

[0826] In this way, this system, which combines an emotion engine, can provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[0827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0828] Step 1:

[0829] Recognize the user

[0830] The terminal uses a camera and sensor device to recognize the user sitting at the table. It receives camera images and sensor information as input and uses image analysis software to confirm the user's presence. As output, it generates data indicating that the user has been recognized. Specifically, the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?"

[0831] Step 2:

[0832] Emotion recognition

[0833] The device uses emotion analysis software to analyze the user's voice and facial expressions and recognize their emotional state. The user's voice and image data are taken as input and processed by the emotion analysis software. The output is the user's emotional state (e.g., excited, calm, anxious). Specifically, if the user is excited, the device will ask, "Did anything special happen today?" in a slightly calmer tone.

[0834] Step 3:

[0835] Menu suggestions

[0836] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" The device takes the user's response as input. When the user requests a recommendation, the server executes a database query to look up ingredient inventory information and past order history. As output, the conversation generation AI suggests, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." Specifically, if the emotion analysis software recognizes that the user appears to be struggling, the device will provide a detailed explanation.

[0837] Step 4:

[0838] Order receipt and confirmation

[0839] When the user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" As input, the voice recognition API converts the user's utterance into text data and generates a confirmation question. As output, the user's final confirmation is obtained and order information is generated in JSON format. The specific operation is to send the order information to the server.

[0840] Step 5:

[0841] Kitchen Cooperation

[0842] The server sends the order information to the cooking management system and instructs it to start cooking. As input, the order information received by the server is analyzed and passed to the cooking management system. As output, the order details are displayed on the cooking management system and cooking begins. As a specific operation, the order details are displayed on a display in the cooking department.

[0843] Step 6:

[0844] Serving food

[0845] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. As input, it receives the cooking completion notification and issues a distribution instruction to the terminal. As output, it generates data indicating that the food will be served. Specifically, the terminal brings the food while saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[0846] Step 7:

[0847] Responding to requests

[0848] If a user requests "an extra glass of water, please," the device uses voice recognition to understand the request and responds with "I understand, I'll bring it right away." As input, the device acquires and analyzes the voice data of the additional request. As output, additional order information is generated and sent to the server. Specifically, the server again instructs the cooking management system to respond, and the automatic food distribution system is used as necessary. The device then brings the water to the user's table, fulfilling the request.

[0849] (Application example 2)

[0850] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] In order to increase customer satisfaction and provide efficient customer service in virtual stores, conventional systems have difficulty responding to customers' emotional states. This is because they are unable to adequately respond to customers' anxieties and questions in an online shopping environment. Furthermore, conventional systems lack the means to quickly and appropriately respond to customers' additional requests. Furthermore, conventional systems make it difficult to provide individually customized experiences, raising concerns about a decline in customer satisfaction.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0853] In this invention, the server includes means for recognizing customers, means for interacting with customers and suggesting menu items, means for receiving and confirming customer orders, means for transmitting order information to cooking facilities, means for providing customers with ordered items, means for receiving and responding to customer requests for additional items, means for analyzing the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services according to their emotional state, thereby enabling the provision of customized services according to the customer's emotional state.

[0854] "Customer recognition means" refers to technology that uses devices such as cameras and sensors to identify users and detect their presence.

[0855] "Means of interacting with customers and suggesting menu items" refers to technologies such as voice dialogue systems and chatbots that interact with users to provide appropriate options.

[0856] "Means for receiving and confirming customer orders" refers to the interface and processing system for receiving and confirming user selections for goods and services.

[0857] The "means for transmitting order information to the cooking facility" refers to a communication technology or protocol for transmitting the user's order details to the cooking facility.

[0858] The "means of providing customers with ordered products" refers to a logistics system and actuator technology for reliably delivering ordered products to users.

[0859] The "means for accepting and responding to additional requests from customers" refers to an interface and processing system for quickly responding to additional requests and changes from users.

[0860] "Means for analyzing the user's voice and facial expressions and recognizing their emotional state" refers to image and voice recognition technology that analyzes the user's voice and facial expression data to determine their emotional state.

[0861] "Means for adjusting dialogue content and services according to emotional state" refers to technology for flexibly changing the content of the services and dialogue provided based on the recognized emotional state.

[0862] In this invention, the following hardware and software are used to realize a virtual store system that recognizes customers and provides customized dialogue and services according to their emotional state.

[0863] Hardware and Software Configuration

[0864] Hardware

[0865] Smartphones: Equipped with a camera and microphone, they can recognize users and accept voice input.

[0866] Head-mounted display (HMD): A device for viewing and interacting with a virtual environment.

[0867] software

[0868] Image and audio analysis library: Uses OpenCV, Google ML Kit, etc. to recognize users' faces and analyze their audio.

[0869] Emotion Recognition Engine: Uses the Emotion API to recognize the user's emotional state by analyzing their voice and facial expressions.

[0870] Generative chat AI: Generates dialogue using GPT models and other natural language generation models.

[0871] Web server: Uses Node.js and handles all communication and data management.

[0872] System Operation

[0873] User Awareness

[0874] First, the device recognizes the user using its camera and microphone. It uses OpenCV to analyze the video data from the camera and perform facial recognition. It also captures audio through the microphone and converts the user's speech into text using a speech analysis function such as Google ML Kit. This allows the device to confirm that the user is a new visitor and begin a conversation by asking, "Hello, welcome. What are you looking for today?"

[0875] Emotion recognition

[0876] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. For example, if the user is nervous, the device will speak in a calmer tone, asking, "Did anything special happen today?" In this way, the device adjusts the content of the conversation according to the user's emotional state.

[0877] Menu suggestions

[0878] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and a 4K smart TV" is generated.

[0879] Order receipt and confirmation

[0880] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server. The server then sends the order information to the food preparation facility or logistics system, and the product is prepared.

[0881] Responding to requests

[0882] Similarly, if the user makes an additional request such as "I would like gift wrapping," the terminal accepts the request and the server makes corresponding arrangements based on that request.

[0883] Examples of concrete examples and prompts

[0884] When a user enters a virtual store

[0885] User: "Can you recommend some products?"

[0886] Assistant: "Welcome! Today's recommendations are the latest Bluetooth earphones and a 4K smart TV. Which one are you interested in?"

[0887] Example of confirmation when a customer decides to purchase

[0888] Assistant: "Are you buying Bluetooth earphones? Understood. Would you like gift wrapping?"

[0889] User: "Yes, please."

[0890] Example prompts using the Emotion Engine

[0891] When the user is nervous

[0892] Assistant: "Is there anything special happening today? Let's talk for a bit and get settled. What product are you looking for?"

[0893] In this way, by using pre-set prompts, it becomes possible to provide customized services in virtual stores that correspond to the user's emotions and needs.

[0894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0895] Step 1: User Awareness

[0896] The device uses a camera and microphone to recognize the user's face and voice. Specifically, it processes the video data from the camera using OpenCV to perform facial recognition. It also analyzes the audio data collected by the microphone using Google ML Kit and converts the user's speech into text. Based on this data, the device begins a conversation by asking, "Hello, welcome. What are you looking for today?"

[0897] Input: Camera video data and audio data

[0898] Output: User's face recognition results and voice-to-text data

[0899] Step 2: Emotion Recognition

[0900] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. Specifically, audio and video data is input into the Emotion API, and the resulting emotional state is received as the analysis result. If the user is nervous, the device will speak in a calm tone, asking, "Is there anything special that happened today?"

[0901] Input: Audio and video data

[0902] Output: Emotion recognition results (e.g., nervousness, excitement, joy, etc.)

[0903] Step 3: Menu proposal

[0904] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and 4K smart TVs" is generated.

[0905] Input: User requests, inventory information, past order history

[0906] Output: Customized product suggestions

[0907] Step 4: Order receipt and confirmation

[0908] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server, which then sends the order information to the food preparation facility or logistics system.

[0909] Input: User's order details

[0910] Output: Confirmed order information, data sent to the server

[0911] Step 5: Response to requests

[0912] If the user makes an additional request, such as "I would like gift wrapping," the terminal accepts the request and the server arranges for the response based on that request. Specifically, the content of the additional request is converted into text and sent to the server. The server then carries out the necessary procedures and notifies the user of the final confirmation.

[0913] Input: Content of additional request

[0914] Output: Notification and confirmation message based on request

[0915] In this way, each step is executed sequentially to provide a customized service to the user.

[0916] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0918] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0919] [Third embodiment]

[0920] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0921] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0922] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0923] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0924] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0925] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0926] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0927] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0932] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing customers, a means for interacting with customers to suggest menu items, a means for receiving and confirming customer orders, a means for transmitting order information to the kitchen, a means for serving ordered dishes to customers, and a means for receiving and responding to additional requests from customers. The specific program processing is explained below in natural language.

[0933] Program processing

[0934] 1. User Awareness

[0935] The device uses a camera and sensors to recognize users sitting at the table.

[0936] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0937] 2. Menu suggestions

[0938] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0939] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0940] 3. Order Receipt and Confirmation

[0941] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0942] The terminal will reconfirm "Would you like a Japanese-style hamburger?" and if the user answers "Yes", it will send the order information to the server.

[0943] 4. Kitchen collaboration

[0944] The server sends the order information to the kitchen system and issues an instruction to start cooking.

[0945] 5. Serving the food

[0946] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[0947] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0948] 6. Response to requests

[0949] The user requests "additional water please."

[0950] The terminal responds, "Understood. We'll bring it to you right away," and sends additional order information to the server.

[0951] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0952] The terminal delivers the water to the user's table.

[0953] Specific examples

[0954] Example of what happens when a user communicates a specific request

[0955] The device recognizes the user and says, "Hello, welcome."

[0956] A user asks, "What do you recommend?"

[0957] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[0958] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0959] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[0960] The terminal transmits the order information to the server.

[0961] The server sends the order information to the kitchen system.

[0962] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[0963] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[0964] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[0965] The processing flow will be explained below.

[0966] Step 1:

[0967] The terminal uses a camera and sensors to recognize the user sitting at a table, allowing the terminal to know that a customer has arrived at the restaurant.

[0968] Step 2:

[0969] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[0970] Step 3:

[0971] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[0972] Step 4:

[0973] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[0974] Step 5:

[0975] When the user responds, "Please tell me your recommendation," the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[0976] Step 6:

[0977] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[0978] Step 7:

[0979] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[0980] Step 8:

[0981] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[0982] Step 9:

[0983] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[0984] Step 10:

[0985] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[0986] Step 11:

[0987] The user requests "additional water please."

[0988] Step 12:

[0989] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[0990] Step 13:

[0991] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[0992] Step 14:

[0993] The terminal will bring the water to the user's table and fulfill their request.

[0994] Example 1

[0995] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0996] In conventional restaurants, the process of receiving an order from a customer, transmitting it to the kitchen, and serving the food required a great deal of time and effort. It was also difficult to respond quickly to additional customer requests, which could lead to a decline in the quality of service. This led to issues such as lower customer satisfaction and a deterioration in store operational efficiency.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0998] In this invention, the server includes a means for recognizing a user, a means for interacting with the user to provide guidance, a means for receiving and confirming a user's request, a means for transmitting the request information to the cooking department, a means for providing the user with the requested food, and a means for receiving and responding to additional requests from the user. This automates the entire process from when the user enters the store to when the user completes their order, enabling fast and efficient service. Furthermore, by referencing past request history and ingredient inventory information, guidance suggestions based on the user's preferences can be made, and by using a dialogue generation AI, orders and requests can be accepted through natural dialogue. This increases user satisfaction and improves the efficiency of store operations.

[0999] "User" refers to a customer who receives services using the system.

[1000] "Means for recognizing" refers to means for identifying a user using a photographing device or a detecting device.

[1001] "Means for suggesting guidance through dialogue" refers to means for communicating with users via voice or text and suggesting appropriate menus or services.

[1002] "Means for receiving and confirming requests" refers to the means for receiving and confirming orders or requests from users.

[1003] "Means for transmitting request information to the cooking department" refers to means for transmitting received orders and requests to the cooking department.

[1004] "Means for providing requested food" refers to the means for delivering cooked food to the customer.

[1005] "Means for accepting and responding to additional requests" refers to means for accepting additional requests from users and providing services in response to those requests.

[1006] "Raw material inventory information" refers to data showing the inventory status of food ingredients, beverages, etc.

[1007] "Past request history" refers to a record of orders or requests made by a user in the past.

[1008] "Dialogue generation artificial intelligence" refers to AI technology that allows for natural dialogue with users.

[1009] "Photography device" refers to equipment for capturing images or videos, such as cameras and video equipment.

[1010] "Sensing equipment" refers to a sensor, device, or other equipment used to detect a particular condition.

[1011] "Data protocol" refers to the rules and procedures for exchanging information.

[1012] A "generative artificial intelligence model" refers to an AI model that automatically generates appropriate guidance and responses based on user input.

[1013] An "automated serving system" refers to a system that automatically delivers food, drinks, etc.

[1014] This invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing a customer, a means for interacting with the customer to suggest directions, a means for receiving and confirming the customer's request, a means for transmitting the request information to a cooking department, a means for providing the customer with the completed food, and a means for receiving and responding to the customer's additional requests.

[1015] First, to recognize the user, the device activates a camera (e.g., a general-purpose camera) and a detector (e.g., a multi-use detector) to scan for users sitting at nearby tables. Specifically, the device uses facial recognition software (e.g., general-purpose recognition software) to identify the user and initiates a dialogue with them, saying, "Hello, welcome. Would you like to see the menu?" This dialogue uses voice recognition technology (e.g., general-purpose voice recognition technology).

[1016] Next, if the user asks "What do you recommend?" without selecting a menu item, the device sends that information to the server. The server references raw material inventory information (e.g., a general-purpose inventory management system) and past request history (e.g., a general-purpose database) and generates appropriate guidance using a generative AI model (e.g., a general-purpose AI model). For example, it might generate a suggestion such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp" and send it to the device, which then relays that information to the user.

[1017] When a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal asks, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information is sent to the server. The server then sends the received order information to the cooking department's system (e.g., a general-purpose cooking management system) and issues a command to start cooking.

[1018] When cooking is complete, the kitchen department sends a completion notification to the server, which then sends that information to the terminal. The terminal then delivers the food to the designated table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." A serving robot (e.g., a general-purpose serving robot) is used for this.

[1019] Furthermore, if the customer requests "an additional water, please," the terminal responds, "Understood. We will bring it to you immediately," and sends the additional order information to the server. The server again instructs the kitchen department to respond, or handles the request using an automated service system (e.g., a general-purpose automated service system). The terminal then uses the automated service system to bring the water to the customer's table.

[1020] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[1021] Examples:

[1022] An example of processing when a user conveys a specific request is shown below.

[1023] The device recognizes the user and says, "Hello, welcome."

[1024] A user asks, "What do you recommend?"

[1025] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1026] A customer orders, "I'd like a Japanese-style hamburger, please."

[1027] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[1028] The terminal transmits the order information to the server.

[1029] The server sends the order information to the cooking department's system.

[1030] The cooking department system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[1031] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[1032] Example prompt sentence:

[1033] Generate a response when a customer asks, "What do you recommend?" taking into account the customer's past ordering history and current ingredient availability information.

[1034] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1035] Program processing flow

[1036] Step 1:

[1037] The device activates its camera and sensors to recognize users sitting at nearby tables.

[1038] Input: Video data acquired by the device through the camera and sensors.

[1039] Data processing: Facial recognition software is used to analyze the video data and identify the user's face.

[1040] Output: User's identity.

[1041] How it works: The device uses a camera to capture video of the area around the table, then processes the video with facial recognition software. If it recognizes a customer, it will begin a conversation with them, saying, "Hello, welcome. Would you like to see the menu?"

[1042] Step 2:

[1043] The terminal guides the user through the menu and asks questions about ordering.

[1044] Input: User response (e.g., "What do you recommend?").

[1045] Data processing: Using voice recognition software, the user's speech is converted into text data.

[1046] Output: The user's request (e.g., "recommendations").

[1047] How it works: The device collects what the user says through a microphone and analyzes it using speech recognition software. Based on the analyzed text, the device sends a message to the server asking, "What do you recommend?"

[1048] Step 3:

[1049] The server references ingredient inventory information and past request history to generate recommended dishes.

[1050] Input: User request details, raw material inventory information, past request history.

[1051] Data processing: Obtaining the necessary information from inventory management systems and databases and using generative AI models to generate appropriate recommendations.

[1052] Output: A list of recommended dishes (e.g. "Japanese-style hamburger steak and fried shrimp").

[1053] Specific operation: The server retrieves current stock information from the inventory management system and references the user's past order history from the database. Using this data, the generative AI model generates a message such as "Today's recommendations are Japanese-style hamburger steak and fried shrimp," and sends it to the terminal.

[1054] Step 4:

[1055] The device then presents the generated recommendations to the user.

[1056] Input: A list of recommended dishes received from the server.

[1057] Data processing: None (output as is).

[1058] Output: Information to the user.

[1059] Specific operation: The device will audibly convey to the user the message, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1060] Step 5:

[1061] The user places an order and the terminal reconfirms the order.

[1062] Input: User's order (e.g., "I'd like a Japanese-style hamburger steak, please").

[1063] Data processing: Analyzes using voice recognition software and sends the order details to the server.

[1064] Output: Confirmed order details and order information sent to the server.

[1065] Specific operation: The terminal analyzes the user's statement, "I'd like a Japanese-style hamburger, please," and reconfirms, "Is a Japanese-style hamburger okay?" If the user responds "Yes," the order details are sent to the server.

[1066] Step 6:

[1067] The server sends the order information to the cooking department and gives cooking instructions.

[1068] Input: Order information received from the terminal.

[1069] Data processing: Sends order information to the cooking department's system.

[1070] Output: Instructions to the cooking department.

[1071] Specific operation: The server sends the order information received from the terminal to the cooking department's system and issues an instruction to start cooking.

[1072] Step 7:

[1073] Once cooking is complete, the terminal serves the food to the user.

[1074] Input: Notification of completion of cooking from the cooking department.

[1075] Data processing: Send a cooking completion notification to the terminal.

[1076] Output: Sends food delivery instructions to the terminal and serves the food to the customer.

[1077] Specific operation: The kitchen department sends a cooking completion notification to the server, which then forwards the information to the terminal. The terminal then uses a serving robot to deliver the food to the specified table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak."

[1078] Step 8:

[1079] Respond to additional user requests.

[1080] Input: The customer's additional request (e.g., "Please give me some more water").

[1081] Data processing: Analyze using voice recognition software and send additional order information to the server.

[1082] Output: Additional request information and instructions to the server.

[1083] Specific operation: The terminal analyzes the user's additional request using voice recognition software and sends it to the server. The server receives the additional request and either instructs the kitchen department to handle it or uses an automated service system. The terminal then uses the automated service system to bring water to the user's table.

[1084] (Application example 1)

[1085] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1086] Conventional food delivery systems have difficulty responding quickly to user orders and requests, potentially resulting in a poor customer experience. Furthermore, the interaction between users and the system is limited, making it difficult to smoothly implement menu suggestions and order confirmation processes. Therefore, there was a need for a system that could efficiently respond to diverse customer requests through an interactive interface.

[1087] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1088] In this invention, the server includes a means for recognizing a user using a camera and a sensor, a means for suggesting menu items to a customer using a chatbot, a means for sending order information to a database in real time using the customer's smartphone, and a means for notifying the customer of the order status, thereby enabling interactive interaction between the customer and the system and enabling orders and requests to be handled quickly and efficiently.

[1089] "Customer" means an individual or entity receiving services or goods.

[1090] "Recognition" is the process of identifying objects using devices such as cameras and sensors.

[1091] "Dialogue" refers to the exchange of information between a system and a user, including verbal communication.

[1092] A "menu" is a list of available dishes and services.

[1093] An "order" is an act by which a customer requests a company to provide a specific service or product.

[1094] A "kitchen" is a facility or place where food is prepared.

[1095] "Serving" is the act of delivering cooked food to customers.

[1096] "Request" refers to a request by a customer for the provision of additional services or products that the customer desires.

[1097] A "camera" is a device for taking still images and videos.

[1098] A "sensor" is a device that detects and measures changes in the environment.

[1099] A "chatbot" is artificial intelligence software that mimics text or voice interaction.

[1100] A "smartphone" is a mobile device with multiple functions, such as making calls, connecting to the Internet, and running applications.

[1101] "Real time" refers to the instantaneous processing or display of ongoing events.

[1102] A "database" is a structured system for efficiently storing, managing, and retrieving data.

[1103] "Notification" is the act of informing a user of specific information.

[1104] This invention is a system for realizing an interactive ordering application for food delivery services. By incorporating the following elements, this system can receive orders through dialogue with customers and automate everything from serving food to responding to additional requests.

[1105] First, when a customer launches the application using their smartphone, the device uses the smartphone's camera to recognize the customer. Image recognition technology, such as Google Vision API, is used to identify the user. Once the device recognizes the user, a conversation begins with "Hello, welcome."

[1106] Next, the device uses its chatbot functionality to suggest menu items. This involves using chatbot APIs such as Google Dialogflow to have a natural conversation with the user. When a customer asks, "What do you recommend?", the server refers to ingredient inventory information and past order history to suggest today's recommended dishes. If the user wants to order a specific dish, the device confirms that information and sends it to the server.

[1107] When the server receives the order information, it sends it to a real-time database (e.g., Firebase Realtime Database) and issues instructions to the kitchen system. Once the order is cooked, the kitchen system notifies the server that the cooking is complete, and the server notifies the customer's smartphone of the delivery progress.

[1108] If the customer makes an additional request (e.g., "Can I have some more water, please?"), the device will again use the chatbot to accept the request and send the information to the server, which will then work with the kitchen system and automatic food distribution system to respond to the request.

[1109] Specific examples

[1110] 1. User Recognition: The user starts up their smartphone and opens the application. The camera recognizes the user's face and begins the conversation with a "Hello, welcome."

[1111] 2. Menu suggestions: When a user asks, "What do you recommend?", the chatbot will suggest, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1112] 3. Order receipt: The user enters "Japanese-style hamburger steak please," and the system reconfirms, "Are you sure you want a Japanese-style hamburger steak?" If the user answers "yes," the order is sent to the server.

[1113] 4. Kitchen integration: Order information is sent to the partner restaurant and cooking begins.

[1114] 5. Delivery Notifications: Your smartphone will send you push notifications to let you know the progress of your delivery.

[1115] 6. Handling additional requests: When a user requests "more water please," the chatbot will accept this and respond.

[1116] Prompt Sentence Examples

[1117] "Give me a scenario where a user opens your app and asks for food recommendations."

[1118] The user opens the smartphone app and asks, "What do you recommend?" The AI ​​chatbot responds, "Today's recommendation is Japanese-style hamburger steak and fried shrimp," based on ingredient availability information and past order history. The user then places an order, "I'd like a Japanese-style hamburger steak, please," and the AI ​​chatbot reconfirms, "Is a Japanese-style hamburger steak okay?"

[1119] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1120] Step 1:

[1121] User Awareness

[1122] Input: The user turns on the smartphone and opens the application.

[1123] How it works: The device's camera and sensors are used to recognize the user's face. Specifically, image recognition technology (e.g., Google Vision API) is used to identify the user's face, and that information is sent to the server.

[1124] Output: Display a message ("Hello, welcome") that the user has been recognized.

[1125] Step 2:

[1126] Menu suggestions

[1127] Input: A user query: "What do you recommend?"

[1128] Operation: The device uses a chatbot function (e.g., Google Dialogflow) to communicate with the user. The server references ingredient inventory information and past order history to select today's recommended dishes.

[1129] Output: Display recommended menu items such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1130] Step 3:

[1131] Order receipt and confirmation

[1132] Input: A user orders a specific dish (e.g., "I'd like a Japanese-style hamburger steak, please").

[1133] Operation: The terminal checks the order details and asks again, "Is a Japanese-style hamburger okay?" After receiving the user's response, it sends all the order information to the server.

[1134] Output: A confirmation message of the order details and the final order details are confirmed.

[1135] Step 4:

[1136] Kitchen Cooperation

[1137] Input: Confirmed order information.

[1138] How it works: The server sends the order information to a real-time database (e.g., Firebase Realtime Database), which then passes that information to the kitchen systems of partner restaurants.

[1139] Output: The kitchen starts cooking based on the order and notifies the server after cooking is complete.

[1140] Step 5:

[1141] Delivery notification

[1142] Input: Notification from the kitchen that cooking is complete.

[1143] How it works: The server receives the cooking completion information and notifies the customer's smartphone of the delivery progress. Specifically, it notifies the progress in real time using the push notification function.

[1144] Output: Cooking completion notification and real-time delivery status updates.

[1145] Step 6:

[1146] Additional request response

[1147] Input: Any additional requests from the customer (e.g., "An extra water please.").

[1148] Operation: The terminal again uses the chatbot function to interact with the customer and accept any additional requests. The server then sends this information to the kitchen system and the automatic food distribution system, which then issues instructions on how to respond.

[1149] Output: The customer is notified that the add request has been accepted and the request is executed.

[1150] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1151] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system is further combined with an emotion engine to recognize the customer's emotional state and respond appropriately accordingly. The system includes a means for recognizing customers, a means for suggesting menu items through dialogue with customers, a means for receiving and confirming customer orders, a means for transmitting order information to a kitchen, a means for serving the ordered food to customers, and a means for receiving and responding to additional requests from customers. The emotion engine also has a mechanism for analyzing the user's voice and facial expressions, recognizing their emotional state, and adjusting the content of the dialogue and the service accordingly.

[1152] Program processing

[1153] 1. User Awareness

[1154] The terminal uses cameras and sensors to recognize users seated at tables and confirm that customers have arrived.

[1155] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1156] 2. Emotional Recognition

[1157] The device uses an emotion engine to analyze the user's voice and facial expressions and recognize their emotional state.

[1158] If the user is excited, the device will speak to them in a calmer tone, asking questions such as "Is there anything special that happened today?" to calm the user down.

[1159] 3. Menu suggestions

[1160] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1161] When a user requests a recommendation, the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1162] If the emotion engine recognizes that the user is distressed, it will provide a more detailed explanation or additional suggestions.

[1163] 4. Order Receipt and Confirmation

[1164] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1165] The terminal will ask again, "Would you like a Japanese-style hamburger?" and if the user answers "Yes," the order information will be sent to the server.

[1166] 5. Kitchen collaboration

[1167] The server sends the order information to the kitchen system and issues a command to start cooking.

[1168] The kitchen system receives the order and begins cooking.

[1169] 6. Serving the food

[1170] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[1171] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1172] 7. Response to requests

[1173] The user requests "additional water please."

[1174] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[1175] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1176] The terminal will bring the water to the user's table and fulfill their request.

[1177] Specific examples

[1178] Example of what happens when a user communicates a specific request

[1179] The device recognizes the user and says, "Hello, welcome."

[1180] A user asks, "What do you recommend?"

[1181] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1182] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1183] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[1184] The terminal transmits the order information to the server.

[1185] The server sends the order information to the kitchen system.

[1186] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[1187] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[1188] Examples of using the Emotion Engine

[1189] The device recognizes the user and says, "Hello, welcome."

[1190] The emotion engine analyzes the user's facial expressions and recognizes excitement.

[1191] The device speaks in a slightly calmer tone, asking, "Is there anything special happening today?"

[1192] The emotion engine confirms that the user's emotions have calmed down and then proceeds with a suggestion, asking, "Have you decided what to order? Or would you like us to recommend some dishes?"

[1193] This system, which combines an emotion engine in this way, is able to provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[1194] The processing flow will be explained below.

[1195] Step 1:

[1196] The terminal uses cameras and sensors to recognize users sitting at tables, allowing the terminal to know that a customer has arrived.

[1197] Step 2:

[1198] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1199] Step 3:

[1200] The device uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if the user is excited, the device will adjust its response accordingly.

[1201] Step 4:

[1202] The device will ask, "Is there anything special happening today?" in a calmer tone, trying to calm the user's emotions.

[1203] Step 5:

[1204] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[1205] Step 6:

[1206] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1207] Step 7:

[1208] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1209] Step 8:

[1210] If the emotion engine recognizes that the user appears distressed, the device will make an additional suggestion, saying, "If you'd like, I can provide a more detailed explanation."

[1211] Step 9:

[1212] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1213] Step 10:

[1214] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[1215] Step 11:

[1216] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[1217] Step 12:

[1218] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[1219] Step 13:

[1220] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1221] Step 14:

[1222] The user requests "additional water please."

[1223] Step 15:

[1224] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[1225] Step 16:

[1226] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1227] Step 17:

[1228] The terminal will bring the water to the user's table and fulfill their request.

[1229] Through this series of steps, the system enables smooth dialogue with customers, ordering, serving food, and responding to requests, reducing the burden on staff while improving the quality of service provided to customers. In addition, the use of an emotion engine enables optimal responses according to the user's emotional state, contributing to improved customer satisfaction.

[1230] Example 2

[1231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1232] In traditional restaurant operations, a large number of human resources are required to handle customer orders, serve food, and handle additional requests, resulting in poor operational efficiency and reduced customer satisfaction. It is also difficult for employees to accurately grasp customers' emotional states, often resulting in inconsistent service quality. Furthermore, some systems lack AI-based dialogue and emotion recognition capabilities, limiting customer service. To solve these problems, a multifunctional and efficient system is needed.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1234] In this invention, the server includes means for recognizing a user, means for interacting with a customer and proposing a menu, means for receiving and confirming a customer order, means for transmitting the order information to the cooking department, means for serving the ordered dishes to the customer, means for receiving and responding to additional requests from the customer, means for using an emotion engine that analyzes the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services based on their emotional state, thereby making it possible to improve the efficiency of customer service operations and customer satisfaction.

[1235] A "means for recognizing a user" is a technology, device, or method for detecting the arrival of a customer at a designated location and confirming the customer's presence.

[1236] A "means for interacting with customers and suggesting menu items" is a technology, device, or method for presenting menu items and assisting customers in making selections through conversation with the customer.

[1237] "Means for receiving and confirming customer orders" means the technology, device, or method for receiving and reconfirming customer menu item selections.

[1238] "Means for transmitting order information to the cooking department" refers to technology, devices, or methods for transmitting the order details received from the customer to the cook or cooking system.

[1239] "Means for delivering ordered food to customers" refers to technology, equipment, or methods for automatically delivering food to customers after cooking is complete.

[1240] The "means for accepting and responding to additional requests from customers" refers to a technology, device, or method for accepting additional requests from customers and responding accordingly.

[1241] "Means using an emotion engine that analyzes a user's voice and facial expressions to recognize their emotional state" refers to a technology, device, or method that analyzes a user's speech and facial expressions and identifies the user's emotions from the results.

[1242] The "means for adjusting dialogue content or services based on emotional state" refers to a technology, device, or method for dynamically changing the dialogue content or service content provided based on the results of recognizing the user's emotional state.

[1243] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. Furthermore, by combining this system with an emotion engine, it is possible to recognize the emotional state of the customer and respond appropriately accordingly. This system comprises the following means:

[1244] 1. User recognition methods:

[1245] The terminal uses a camera (e.g., a generic name "camera device") and a sensor (e.g., "sensor device") to recognize a user sitting at a table. Image recognition software (e.g., "image analysis software") is used to recognize the user. This confirms that a customer has arrived, and the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?". Speech recognition and generation use a speech recognition API and a speech generation API (e.g., "speech processing API").

[1246] 2. Emotion recognition means:

[1247] The device uses an emotion engine (e.g., "emotion analysis software") to analyze the user's voice and facial expressions and recognize their emotional state. If the user is excited, the device can speak in a calmer tone, asking, "Is there anything special happening today?" This adjusts the conversation with the customer to proceed smoothly.

[1248] 3. Menu suggestion methods:

[1249] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" If the user requests a recommendation, the server references ingredient inventory information and the user's past order history. Based on this information, a chat generation AI (for example, a "dialogue generation AI") is used to suggest, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." If the emotion engine recognizes that the user seems troubled, the device will provide a detailed explanation.

[1250] 4. Order receipt and confirmation methods:

[1251] If a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information will be sent to the server in JSON format via the HTTP API.

[1252] 5. Kitchen collaboration methods:

[1253] The server sends the order information to the cooking department's system (e.g., a "cooking management system") and instructs it to start cooking. The cooking management system receives the order and starts cooking. The order information is displayed on a display in the cooking department.

[1254] 6. Food Serving Method:

[1255] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. The terminal then carries the food to the customer, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." It is also possible to use an automatic robot (for example, an "autonomous mobile robot") to serve the food.

[1256] 7. How to respond to requests:

[1257] If a user requests "an extra glass of water, please," the device will use voice recognition to understand the request and respond with "I understand, I'll bring it right away." The additional order information is sent to the server, which again instructs the cooking system to respond. If necessary, an automatic food distribution system can also be used. The device will respond quickly to additional requests, increasing customer satisfaction.

[1258] Specific examples

[1259] If the user has a specific request, the following might happen:

[1260] 1. The device recognizes the user and says, "Hello, welcome."

[1261] 2. A user asks, "What do you recommend?"

[1262] 3. The device responds, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1263] 4. The user places an order, "I'd like a Japanese-style hamburger, please."

[1264] 5. The terminal will ask again, "Would you like a Japanese-style hamburger?" and the user will answer "Yes."

[1265] 6. The terminal sends the order information to the server.

[1266] 7. The server sends the order information to the cooking management system.

[1267] 8. The cooking management system notifies the terminal that the food is ready, and the server issues serving instructions to the terminal.

[1268] 9. The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[1269] Example prompt sentence:

[1270] "Hello, welcome. Would you like to see the menu?"

[1271] "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1272] "Would you like a Japanese-style hamburger?"

[1273] In this way, this system, which combines an emotion engine, can provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[1274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1275] Step 1:

[1276] Recognize the user

[1277] The terminal uses a camera and sensor device to recognize the user sitting at the table. It receives camera images and sensor information as input and uses image analysis software to confirm the user's presence. As output, it generates data indicating that the user has been recognized. Specifically, the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?"

[1278] Step 2:

[1279] Emotion recognition

[1280] The device uses emotion analysis software to analyze the user's voice and facial expressions and recognize their emotional state. The user's voice and image data are taken as input and processed by the emotion analysis software. The output is the user's emotional state (e.g., excited, calm, anxious). Specifically, if the user is excited, the device will ask, "Did anything special happen today?" in a slightly calmer tone.

[1281] Step 3:

[1282] Menu suggestions

[1283] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" The device takes the user's response as input. When the user requests a recommendation, the server executes a database query to look up ingredient inventory information and past order history. As output, the conversation generation AI suggests, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." Specifically, if the emotion analysis software recognizes that the user appears to be struggling, the device will provide a detailed explanation.

[1284] Step 4:

[1285] Order receipt and confirmation

[1286] When the user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" As input, the voice recognition API converts the user's utterance into text data and generates a confirmation question. As output, the user's final confirmation is obtained and order information is generated in JSON format. The specific operation is to send the order information to the server.

[1287] Step 5:

[1288] Kitchen Cooperation

[1289] The server sends the order information to the cooking management system and instructs it to start cooking. As input, the order information received by the server is analyzed and passed to the cooking management system. As output, the order details are displayed on the cooking management system and cooking begins. As a specific operation, the order details are displayed on a display in the cooking department.

[1290] Step 6:

[1291] Serving food

[1292] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. As input, it receives the cooking completion notification and issues a distribution instruction to the terminal. As output, it generates data indicating that the food will be served. Specifically, the terminal brings the food while saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[1293] Step 7:

[1294] Responding to requests

[1295] If a user requests "an extra glass of water, please," the device uses voice recognition to understand the request and responds with "I understand, I'll bring it right away." As input, the device acquires and analyzes the voice data of the additional request. As output, additional order information is generated and sent to the server. Specifically, the server again instructs the cooking management system to respond, and the automatic food distribution system is used as necessary. The device then brings the water to the user's table, fulfilling the request.

[1296] (Application example 2)

[1297] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1298] In order to increase customer satisfaction and provide efficient customer service in virtual stores, conventional systems have difficulty responding to customers' emotional states. This is because they are unable to adequately respond to customers' anxieties and questions in an online shopping environment. Furthermore, conventional systems lack the means to quickly and appropriately respond to customers' additional requests. Furthermore, conventional systems make it difficult to provide individually customized experiences, raising concerns about a decline in customer satisfaction.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1300] In this invention, the server includes means for recognizing customers, means for interacting with customers and suggesting menu items, means for receiving and confirming customer orders, means for transmitting order information to cooking facilities, means for providing customers with ordered items, means for receiving and responding to customer requests for additional items, means for analyzing the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services according to their emotional state, thereby enabling the provision of customized services according to the customer's emotional state.

[1301] "Customer recognition means" refers to technology that uses devices such as cameras and sensors to identify users and detect their presence.

[1302] "Means of interacting with customers and suggesting menu items" refers to technologies such as voice dialogue systems and chatbots that interact with users to provide appropriate options.

[1303] "Means for receiving and confirming customer orders" refers to the interface and processing system for receiving and confirming user selections for goods and services.

[1304] The "means for transmitting order information to the cooking facility" refers to a communication technology or protocol for transmitting the user's order details to the cooking facility.

[1305] The "means of providing customers with ordered products" refers to a logistics system and actuator technology for reliably delivering ordered products to users.

[1306] The "means for accepting and responding to additional requests from customers" refers to an interface and processing system for quickly responding to additional requests and changes from users.

[1307] "Means for analyzing the user's voice and facial expressions and recognizing their emotional state" refers to image and voice recognition technology that analyzes the user's voice and facial expression data to determine their emotional state.

[1308] "Means for adjusting dialogue content and services according to emotional state" refers to technology for flexibly changing the content of the services and dialogue provided based on the recognized emotional state.

[1309] In this invention, the following hardware and software are used to realize a virtual store system that recognizes customers and provides customized dialogue and services according to their emotional state.

[1310] Hardware and Software Configuration

[1311] Hardware

[1312] Smartphones: Equipped with a camera and microphone, they can recognize users and accept voice input.

[1313] Head-mounted display (HMD): A device for viewing and interacting with a virtual environment.

[1314] software

[1315] Image and audio analysis library: Uses OpenCV, Google ML Kit, etc. to recognize users' faces and analyze their audio.

[1316] Emotion Recognition Engine: Uses the Emotion API to recognize the user's emotional state by analyzing their voice and facial expressions.

[1317] Generative chat AI: Generates dialogue using GPT models and other natural language generation models.

[1318] Web server: Uses Node.js and handles all communication and data management.

[1319] System Operation

[1320] User Awareness

[1321] First, the device recognizes the user using its camera and microphone. It uses OpenCV to analyze the video data from the camera and perform facial recognition. It also captures audio through the microphone and converts the user's speech into text using a speech analysis function such as Google ML Kit. This allows the device to confirm that the user is a new visitor and begin a conversation by asking, "Hello, welcome. What are you looking for today?"

[1322] Emotion recognition

[1323] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. For example, if the user is nervous, the device will speak in a calmer tone, asking, "Did anything special happen today?" In this way, the device adjusts the content of the conversation according to the user's emotional state.

[1324] Menu suggestions

[1325] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and a 4K smart TV" is generated.

[1326] Order receipt and confirmation

[1327] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server. The server then sends the order information to the food preparation facility or logistics system, and the product is prepared.

[1328] Responding to requests

[1329] Similarly, if the user makes an additional request such as "I would like gift wrapping," the terminal accepts the request and the server makes corresponding arrangements based on that request.

[1330] Examples of concrete examples and prompts

[1331] When a user enters a virtual store

[1332] User: "Can you recommend some products?"

[1333] Assistant: "Welcome! Today's recommendations are the latest Bluetooth earphones and a 4K smart TV. Which one are you interested in?"

[1334] Example of confirmation when a customer decides to purchase

[1335] Assistant: "Are you buying Bluetooth earphones? Understood. Would you like gift wrapping?"

[1336] User: "Yes, please."

[1337] Example prompts using the Emotion Engine

[1338] When the user is nervous

[1339] Assistant: "Is there anything special happening today? Let's talk for a bit and get settled. What product are you looking for?"

[1340] In this way, by using pre-set prompts, it becomes possible to provide customized services in virtual stores that correspond to the user's emotions and needs.

[1341] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1342] Step 1: User Awareness

[1343] The device uses a camera and microphone to recognize the user's face and voice. Specifically, it processes the video data from the camera using OpenCV to perform facial recognition. It also analyzes the audio data collected by the microphone using Google ML Kit and converts the user's speech into text. Based on this data, the device begins a conversation by asking, "Hello, welcome. What are you looking for today?"

[1344] Input: Camera video data and audio data

[1345] Output: User's face recognition results and voice-to-text data

[1346] Step 2: Emotion Recognition

[1347] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. Specifically, audio and video data is input into the Emotion API, and the resulting emotional state is received as the analysis result. If the user is nervous, the device will speak in a calm tone, asking, "Is there anything special that happened today?"

[1348] Input: Audio and video data

[1349] Output: Emotion recognition results (e.g., nervousness, excitement, joy, etc.)

[1350] Step 3: Menu proposal

[1351] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and 4K smart TVs" is generated.

[1352] Input: User requests, inventory information, past order history

[1353] Output: Customized product suggestions

[1354] Step 4: Order receipt and confirmation

[1355] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server, which then sends the order information to the food preparation facility or logistics system.

[1356] Input: User's order details

[1357] Output: Confirmed order information, data sent to the server

[1358] Step 5: Response to requests

[1359] If the user makes an additional request, such as "I would like gift wrapping," the terminal accepts the request and the server arranges for the response based on that request. Specifically, the content of the additional request is converted into text and sent to the server. The server then carries out the necessary procedures and notifies the user of the final confirmation.

[1360] Input: Content of additional request

[1361] Output: Notification and confirmation message based on request

[1362] In this way, each step is executed sequentially to provide a customized service to the user.

[1363] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1364] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1365] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1366] [Fourth embodiment]

[1367] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1368] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1369] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1370] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1371] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1372] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1373] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1374] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1375] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1376] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1377] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1378] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1379] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1380] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing customers, a means for interacting with customers to suggest menu items, a means for receiving and confirming customer orders, a means for transmitting order information to the kitchen, a means for serving ordered dishes to customers, and a means for receiving and responding to additional requests from customers. The specific program processing is explained below in natural language.

[1381] Program processing

[1382] 1. User Awareness

[1383] The device uses a camera and sensors to recognize users sitting at the table.

[1384] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1385] 2. Menu suggestions

[1386] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1387] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1388] 3. Order Receipt and Confirmation

[1389] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1390] The terminal will reconfirm "Would you like a Japanese-style hamburger?" and if the user answers "Yes", it will send the order information to the server.

[1391] 4. Kitchen collaboration

[1392] The server sends the order information to the kitchen system and issues an instruction to start cooking.

[1393] 5. Serving the food

[1394] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[1395] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1396] 6. Response to requests

[1397] The user requests "additional water please."

[1398] The terminal responds, "Understood. We'll bring it to you right away," and sends additional order information to the server.

[1399] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1400] The terminal delivers the water to the user's table.

[1401] Specific examples

[1402] Example of what happens when a user communicates a specific request

[1403] The device recognizes the user and says, "Hello, welcome."

[1404] A user asks, "What do you recommend?"

[1405] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1406] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1407] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[1408] The terminal transmits the order information to the server.

[1409] The server sends the order information to the kitchen system.

[1410] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[1411] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[1412] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[1413] The processing flow will be explained below.

[1414] Step 1:

[1415] The terminal uses a camera and sensors to recognize the user sitting at a table, allowing the terminal to know that a customer has arrived at the restaurant.

[1416] Step 2:

[1417] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1418] Step 3:

[1419] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[1420] Step 4:

[1421] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1422] Step 5:

[1423] When the user responds, "Please tell me your recommendation," the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1424] Step 6:

[1425] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1426] Step 7:

[1427] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[1428] Step 8:

[1429] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[1430] Step 9:

[1431] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[1432] Step 10:

[1433] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1434] Step 11:

[1435] The user requests "additional water please."

[1436] Step 12:

[1437] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[1438] Step 13:

[1439] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1440] Step 14:

[1441] The terminal will bring the water to the user's table and fulfill their request.

[1442] Example 1

[1443] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1444] In conventional restaurants, the process of receiving an order from a customer, transmitting it to the kitchen, and serving the food required a great deal of time and effort. It was also difficult to respond quickly to additional customer requests, which could lead to a decline in the quality of service. This led to issues such as lower customer satisfaction and a deterioration in store operational efficiency.

[1445] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1446] In this invention, the server includes a means for recognizing a user, a means for interacting with the user to provide guidance, a means for receiving and confirming a user's request, a means for transmitting the request information to the cooking department, a means for providing the user with the requested food, and a means for receiving and responding to additional requests from the user. This automates the entire process from when the user enters the store to when the user completes their order, enabling fast and efficient service. Furthermore, by referencing past request history and ingredient inventory information, guidance suggestions based on the user's preferences can be made, and by using a dialogue generation AI, orders and requests can be accepted through natural dialogue. This increases user satisfaction and improves the efficiency of store operations.

[1447] "User" refers to a customer who receives services using the system.

[1448] "Means for recognizing" refers to means for identifying a user using a photographing device or a detecting device.

[1449] "Means for suggesting guidance through dialogue" refers to means for communicating with users via voice or text and suggesting appropriate menus or services.

[1450] "Means for receiving and confirming requests" refers to the means for receiving and confirming orders or requests from users.

[1451] "Means for transmitting request information to the cooking department" refers to means for transmitting received orders and requests to the cooking department.

[1452] "Means for providing requested food" refers to the means for delivering cooked food to the customer.

[1453] "Means for accepting and responding to additional requests" refers to means for accepting additional requests from users and providing services in response to those requests.

[1454] "Raw material inventory information" refers to data showing the inventory status of food ingredients, beverages, etc.

[1455] "Past request history" refers to a record of orders or requests made by a user in the past.

[1456] "Dialogue generation artificial intelligence" refers to AI technology that allows for natural dialogue with users.

[1457] "Photography device" refers to equipment for capturing images or videos, such as cameras and video equipment.

[1458] "Sensing equipment" refers to a sensor, device, or other equipment used to detect a particular condition.

[1459] "Data protocol" refers to the rules and procedures for exchanging information.

[1460] A "generative artificial intelligence model" refers to an AI model that automatically generates appropriate guidance and responses based on user input.

[1461] An "automated serving system" refers to a system that automatically delivers food, drinks, etc.

[1462] This invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system includes a means for recognizing a customer, a means for interacting with the customer to suggest directions, a means for receiving and confirming the customer's request, a means for transmitting the request information to a cooking department, a means for providing the customer with the completed food, and a means for receiving and responding to the customer's additional requests.

[1463] First, to recognize the user, the device activates a camera (e.g., a general-purpose camera) and a detector (e.g., a multi-use detector) to scan for users sitting at nearby tables. Specifically, the device uses facial recognition software (e.g., general-purpose recognition software) to identify the user and initiates a dialogue with them, saying, "Hello, welcome. Would you like to see the menu?" This dialogue uses voice recognition technology (e.g., general-purpose voice recognition technology).

[1464] Next, if the user asks "What do you recommend?" without selecting a menu item, the device sends that information to the server. The server references raw material inventory information (e.g., a general-purpose inventory management system) and past request history (e.g., a general-purpose database) and generates appropriate guidance using a generative AI model (e.g., a general-purpose AI model). For example, it might generate a suggestion such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp" and send it to the device, which then relays that information to the user.

[1465] When a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal asks, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information is sent to the server. The server then sends the received order information to the cooking department's system (e.g., a general-purpose cooking management system) and issues a command to start cooking.

[1466] When cooking is complete, the kitchen department sends a completion notification to the server, which then sends that information to the terminal. The terminal then delivers the food to the designated table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." A serving robot (e.g., a general-purpose serving robot) is used for this.

[1467] Furthermore, if the customer requests "an additional water, please," the terminal responds, "Understood. We will bring it to you immediately," and sends the additional order information to the server. The server again instructs the kitchen department to respond, or handles the request using an automated service system (e.g., a general-purpose automated service system). The terminal then uses the automated service system to bring the water to the customer's table.

[1468] In this way, the system enables smooth interaction with customers, ordering, serving food, and responding to requests, reducing the burden on staff while providing high-quality service to customers.

[1469] Examples:

[1470] An example of processing when a user conveys a specific request is shown below.

[1471] The device recognizes the user and says, "Hello, welcome."

[1472] A user asks, "What do you recommend?"

[1473] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1474] A customer orders, "I'd like a Japanese-style hamburger, please."

[1475] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[1476] The terminal transmits the order information to the server.

[1477] The server sends the order information to the cooking department's system.

[1478] The cooking department system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[1479] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[1480] Example prompt sentence:

[1481] Generate a response when a customer asks, "What do you recommend?" taking into account the customer's past ordering history and current ingredient availability information.

[1482] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1483] Program processing flow

[1484] Step 1:

[1485] The device activates its camera and sensors to recognize users sitting at nearby tables.

[1486] Input: Video data acquired by the device through the camera and sensors.

[1487] Data processing: Facial recognition software is used to analyze the video data and identify the user's face.

[1488] Output: User's identity.

[1489] How it works: The device uses a camera to capture video of the area around the table, then processes the video with facial recognition software. If it recognizes a customer, it will begin a conversation with them, saying, "Hello, welcome. Would you like to see the menu?"

[1490] Step 2:

[1491] The terminal guides the user through the menu and asks questions about ordering.

[1492] Input: User response (e.g., "What do you recommend?").

[1493] Data processing: Using voice recognition software, the user's speech is converted into text data.

[1494] Output: The user's request (e.g., "recommendations").

[1495] How it works: The device collects what the user says through a microphone and analyzes it using speech recognition software. Based on the analyzed text, the device sends a message to the server asking, "What do you recommend?"

[1496] Step 3:

[1497] The server references ingredient inventory information and past request history to generate recommended dishes.

[1498] Input: User request details, raw material inventory information, past request history.

[1499] Data processing: Obtaining the necessary information from inventory management systems and databases and using generative AI models to generate appropriate recommendations.

[1500] Output: A list of recommended dishes (e.g. "Japanese-style hamburger steak and fried shrimp").

[1501] Specific operation: The server retrieves current stock information from the inventory management system and references the user's past order history from the database. Using this data, the generative AI model generates a message such as "Today's recommendations are Japanese-style hamburger steak and fried shrimp," and sends it to the terminal.

[1502] Step 4:

[1503] The device then presents the generated recommendations to the user.

[1504] Input: A list of recommended dishes received from the server.

[1505] Data processing: None (output as is).

[1506] Output: Information to the user.

[1507] Specific operation: The device will audibly convey to the user the message, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1508] Step 5:

[1509] The user places an order and the terminal reconfirms the order.

[1510] Input: User's order (e.g., "I'd like a Japanese-style hamburger steak, please").

[1511] Data processing: Analyzes using voice recognition software and sends the order details to the server.

[1512] Output: Confirmed order details and order information sent to the server.

[1513] Specific operation: The terminal analyzes the user's statement, "I'd like a Japanese-style hamburger, please," and reconfirms, "Is a Japanese-style hamburger okay?" If the user responds "Yes," the order details are sent to the server.

[1514] Step 6:

[1515] The server sends the order information to the cooking department and gives cooking instructions.

[1516] Input: Order information received from the terminal.

[1517] Data processing: Sends order information to the cooking department's system.

[1518] Output: Instructions to the cooking department.

[1519] Specific operation: The server sends the order information received from the terminal to the cooking department's system and issues an instruction to start cooking.

[1520] Step 7:

[1521] Once cooking is complete, the terminal serves the food to the user.

[1522] Input: Notification of completion of cooking from the cooking department.

[1523] Data processing: Send a cooking completion notification to the terminal.

[1524] Output: Sends food delivery instructions to the terminal and serves the food to the customer.

[1525] Specific operation: The kitchen department sends a cooking completion notification to the server, which then forwards the information to the terminal. The terminal then uses a serving robot to deliver the food to the specified table, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak."

[1526] Step 8:

[1527] Respond to additional user requests.

[1528] Input: The customer's additional request (e.g., "Please give me some more water").

[1529] Data processing: Analyze using voice recognition software and send additional order information to the server.

[1530] Output: Additional request information and instructions to the server.

[1531] Specific operation: The terminal analyzes the user's additional request using voice recognition software and sends it to the server. The server receives the additional request and either instructs the kitchen department to handle it or uses an automated service system. The terminal then uses the automated service system to bring water to the user's table.

[1532] (Application example 1)

[1533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1534] Conventional food delivery systems have difficulty responding quickly to user orders and requests, potentially resulting in a poor customer experience. Furthermore, the interaction between users and the system is limited, making it difficult to smoothly implement menu suggestions and order confirmation processes. Therefore, there was a need for a system that could efficiently respond to diverse customer requests through an interactive interface.

[1535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1536] In this invention, the server includes a means for recognizing a user using a camera and a sensor, a means for suggesting menu items to a customer using a chatbot, a means for sending order information to a database in real time using the customer's smartphone, and a means for notifying the customer of the order status, thereby enabling interactive interaction between the customer and the system and enabling orders and requests to be handled quickly and efficiently.

[1537] "Customer" means an individual or entity receiving services or goods.

[1538] "Recognition" is the process of identifying objects using devices such as cameras and sensors.

[1539] "Dialogue" refers to the exchange of information between a system and a user, including verbal communication.

[1540] A "menu" is a list of available dishes and services.

[1541] An "order" is an act by which a customer requests a company to provide a specific service or product.

[1542] A "kitchen" is a facility or place where food is prepared.

[1543] "Serving" is the act of delivering cooked food to customers.

[1544] "Request" refers to a request by a customer for the provision of additional services or products that the customer desires.

[1545] A "camera" is a device for taking still images and videos.

[1546] A "sensor" is a device that detects and measures changes in the environment.

[1547] A "chatbot" is artificial intelligence software that mimics text or voice interaction.

[1548] A "smartphone" is a mobile device with multiple functions, such as making calls, connecting to the Internet, and running applications.

[1549] "Real time" refers to the instantaneous processing or display of ongoing events.

[1550] A "database" is a structured system for efficiently storing, managing, and retrieving data.

[1551] "Notification" is the act of informing a user of specific information.

[1552] This invention is a system for realizing an interactive ordering application for food delivery services. By incorporating the following elements, this system can receive orders through dialogue with customers and automate everything from serving food to responding to additional requests.

[1553] First, when a customer launches the application using their smartphone, the device uses the smartphone's camera to recognize the customer. Image recognition technology, such as Google Vision API, is used to identify the user. Once the device recognizes the user, a conversation begins with "Hello, welcome."

[1554] Next, the device uses its chatbot functionality to suggest menu items. This involves using chatbot APIs such as Google Dialogflow to have a natural conversation with the user. When a customer asks, "What do you recommend?", the server refers to ingredient inventory information and past order history to suggest today's recommended dishes. If the user wants to order a specific dish, the device confirms that information and sends it to the server.

[1555] When the server receives the order information, it sends it to a real-time database (e.g., Firebase Realtime Database) and issues instructions to the kitchen system. Once the order is cooked, the kitchen system notifies the server that the cooking is complete, and the server notifies the customer's smartphone of the delivery progress.

[1556] If the customer makes an additional request (e.g., "Can I have some more water, please?"), the device will again use the chatbot to accept the request and send the information to the server, which will then work with the kitchen system and automatic food distribution system to respond to the request.

[1557] Specific examples

[1558] 1. User Recognition: The user starts up their smartphone and opens the application. The camera recognizes the user's face and begins the conversation with a "Hello, welcome."

[1559] 2. Menu suggestions: When a user asks, "What do you recommend?", the chatbot will suggest, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1560] 3. Order receipt: The user enters "Japanese-style hamburger steak please," and the system reconfirms, "Are you sure you want a Japanese-style hamburger steak?" If the user answers "yes," the order is sent to the server.

[1561] 4. Kitchen integration: Order information is sent to the partner restaurant and cooking begins.

[1562] 5. Delivery Notifications: Your smartphone will send you push notifications to let you know the progress of your delivery.

[1563] 6. Handling additional requests: When a user requests "more water please," the chatbot will accept this and respond.

[1564] Prompt Sentence Examples

[1565] "Give me a scenario where a user opens your app and asks for food recommendations."

[1566] The user opens the smartphone app and asks, "What do you recommend?" The AI ​​chatbot responds, "Today's recommendation is Japanese-style hamburger steak and fried shrimp," based on ingredient availability information and past order history. The user then places an order, "I'd like a Japanese-style hamburger steak, please," and the AI ​​chatbot reconfirms, "Is a Japanese-style hamburger steak okay?"

[1567] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1568] Step 1:

[1569] User Awareness

[1570] Input: The user turns on the smartphone and opens the application.

[1571] How it works: The device's camera and sensors are used to recognize the user's face. Specifically, image recognition technology (e.g., Google Vision API) is used to identify the user's face, and that information is sent to the server.

[1572] Output: Display a message ("Hello, welcome") that the user has been recognized.

[1573] Step 2:

[1574] Menu suggestions

[1575] Input: A user query: "What do you recommend?"

[1576] Operation: The device uses a chatbot function (e.g., Google Dialogflow) to communicate with the user. The server references ingredient inventory information and past order history to select today's recommended dishes.

[1577] Output: Display recommended menu items such as "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1578] Step 3:

[1579] Order receipt and confirmation

[1580] Input: A user orders a specific dish (e.g., "I'd like a Japanese-style hamburger steak, please").

[1581] Operation: The terminal checks the order details and asks again, "Is a Japanese-style hamburger okay?" After receiving the user's response, it sends all the order information to the server.

[1582] Output: A confirmation message of the order details and the final order details are confirmed.

[1583] Step 4:

[1584] Kitchen Cooperation

[1585] Input: Confirmed order information.

[1586] How it works: The server sends the order information to a real-time database (e.g., Firebase Realtime Database), which then passes that information to the kitchen systems of partner restaurants.

[1587] Output: The kitchen starts cooking based on the order and notifies the server after cooking is complete.

[1588] Step 5:

[1589] Delivery notification

[1590] Input: Notification from the kitchen that cooking is complete.

[1591] How it works: The server receives the cooking completion information and notifies the customer's smartphone of the delivery progress. Specifically, it notifies the progress in real time using the push notification function.

[1592] Output: Cooking completion notification and real-time delivery status updates.

[1593] Step 6:

[1594] Additional request response

[1595] Input: Any additional requests from the customer (e.g., "An extra water please.").

[1596] Operation: The terminal again uses the chatbot function to interact with the customer and accept any additional requests. The server then sends this information to the kitchen system and the automatic food distribution system, which then issues instructions on how to respond.

[1597] Output: The customer is notified that the add request has been accepted and the request is executed.

[1598] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1599] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. The system is further combined with an emotion engine to recognize the customer's emotional state and respond appropriately accordingly. The system includes a means for recognizing customers, a means for suggesting menu items through dialogue with customers, a means for receiving and confirming customer orders, a means for transmitting order information to a kitchen, a means for serving the ordered food to customers, and a means for receiving and responding to additional requests from customers. The emotion engine also has a mechanism for analyzing the user's voice and facial expressions, recognizing their emotional state, and adjusting the content of the dialogue and the service accordingly.

[1600] Program processing

[1601] 1. User Awareness

[1602] The terminal uses cameras and sensors to recognize users seated at tables and confirm that customers have arrived.

[1603] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1604] 2. Emotional Recognition

[1605] The device uses an emotion engine to analyze the user's voice and facial expressions and recognize their emotional state.

[1606] If the user is excited, the device will speak to them in a calmer tone, asking questions such as "Is there anything special that happened today?" to calm the user down.

[1607] 3. Menu suggestions

[1608] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1609] When a user requests a recommendation, the server refers to the ingredients' inventory information and the user's past order history, and suggests, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1610] If the emotion engine recognizes that the user is distressed, it will provide a more detailed explanation or additional suggestions.

[1611] 4. Order Receipt and Confirmation

[1612] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1613] The terminal will ask again, "Would you like a Japanese-style hamburger?" and if the user answers "Yes," the order information will be sent to the server.

[1614] 5. Kitchen collaboration

[1615] The server sends the order information to the kitchen system and issues a command to start cooking.

[1616] The kitchen system receives the order and begins cooking.

[1617] 6. Serving the food

[1618] The kitchen system notifies the server that cooking is complete, and the server sends that information to the terminal and issues serving instructions.

[1619] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1620] 7. Response to requests

[1621] The user requests "additional water please."

[1622] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[1623] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1624] The terminal will bring the water to the user's table and fulfill their request.

[1625] Specific examples

[1626] Example of what happens when a user communicates a specific request

[1627] The device recognizes the user and says, "Hello, welcome."

[1628] A user asks, "What do you recommend?"

[1629] The terminal replies, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1630] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1631] The terminal reconfirms, "Are you sure you want a Japanese-style hamburger steak?" and the user answers, "Yes."

[1632] The terminal transmits the order information to the server.

[1633] The server sends the order information to the kitchen system.

[1634] The kitchen system notifies the user that cooking is complete, and the server issues serving instructions to the terminal.

[1635] The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger."

[1636] Examples of using the Emotion Engine

[1637] The device recognizes the user and says, "Hello, welcome."

[1638] The emotion engine analyzes the user's facial expressions and recognizes excitement.

[1639] The device speaks in a slightly calmer tone, asking, "Is there anything special happening today?"

[1640] The emotion engine confirms that the user's emotions have calmed down and then proceeds with a suggestion, asking, "Have you decided what to order? Or would you like us to recommend some dishes?"

[1641] This system, which combines an emotion engine in this way, is able to provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[1642] The processing flow will be explained below.

[1643] Step 1:

[1644] The terminal uses cameras and sensors to recognize users sitting at tables, allowing the terminal to know that a customer has arrived.

[1645] Step 2:

[1646] The terminal begins the conversation by saying, "Hello, welcome. Would you like to see the menu?"

[1647] Step 3:

[1648] The device uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if the user is excited, the device will adjust its response accordingly.

[1649] Step 4:

[1650] The device will ask, "Is there anything special happening today?" in a calmer tone, trying to calm the user's emotions.

[1651] Step 5:

[1652] If the user responds "Yes, I'd like to see the menu," the terminal will provide a digitally displayed or printed menu.

[1653] Step 6:

[1654] The device asks, "Are you ready to order? Or would you like us to suggest some dishes?"

[1655] Step 7:

[1656] If the user answers, "Please tell me your recommendation," the server will refer to the ingredient inventory information and the user's past order history, and suggest, via chat generation AI, "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1657] Step 8:

[1658] If the emotion engine recognizes that the user appears distressed, the device will make an additional suggestion, saying, "If you'd like, I can provide a more detailed explanation."

[1659] Step 9:

[1660] The user places an order saying, "I'd like a Japanese-style hamburger steak, please."

[1661] Step 10:

[1662] The terminal will ask again, "Would you like a Japanese-style hamburger?" If the user answers "Yes," the order information will be sent to the server.

[1663] Step 11:

[1664] The server sends the order information to the kitchen system and issues a command to start cooking. The kitchen system receives the order and starts cooking.

[1665] Step 12:

[1666] The kitchen system notifies the server that the food is ready, and the server sends that information to the terminal and issues serving instructions.

[1667] Step 13:

[1668] The terminal will say, "Sorry to keep you waiting, here's a Japanese-style hamburger steak," and bring the food to the designated table.

[1669] Step 14:

[1670] The user requests "additional water please."

[1671] Step 15:

[1672] The terminal responds, "Understood, we'll bring it to you right away," and sends the additional order information to the server.

[1673] Step 16:

[1674] The server will again instruct the kitchen system to respond or use the automatic food distribution system.

[1675] Step 17:

[1676] The terminal will bring the water to the user's table and fulfill their request.

[1677] Through this series of steps, the system enables smooth dialogue with customers, ordering, serving food, and responding to requests, reducing the burden on staff while improving the quality of service provided to customers. In addition, the use of an emotion engine enables optimal responses according to the user's emotional state, contributing to improved customer satisfaction.

[1678] Example 2

[1679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1680] In traditional restaurant operations, a large number of human resources are required to handle customer orders, serve food, and handle additional requests, resulting in poor operational efficiency and reduced customer satisfaction. It is also difficult for employees to accurately grasp customers' emotional states, often resulting in inconsistent service quality. Furthermore, some systems lack AI-based dialogue and emotion recognition capabilities, limiting customer service. To solve these problems, a multifunctional and efficient system is needed.

[1681] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1682] In this invention, the server includes means for recognizing a user, means for interacting with a customer and proposing a menu, means for receiving and confirming a customer order, means for transmitting the order information to the cooking department, means for serving the ordered dishes to the customer, means for receiving and responding to additional requests from the customer, means for using an emotion engine that analyzes the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services based on their emotional state, thereby making it possible to improve the efficiency of customer service operations and customer satisfaction.

[1683] A "means for recognizing a user" is a technology, device, or method for detecting the arrival of a customer at a designated location and confirming the customer's presence.

[1684] A "means for interacting with customers and suggesting menu items" is a technology, device, or method for presenting menu items and assisting customers in making selections through conversation with the customer.

[1685] "Means for receiving and confirming customer orders" means the technology, device, or method for receiving and reconfirming customer menu item selections.

[1686] "Means for transmitting order information to the cooking department" refers to technology, devices, or methods for transmitting the order details received from the customer to the cook or cooking system.

[1687] "Means for delivering ordered food to customers" refers to technology, equipment, or methods for automatically delivering food to customers after cooking is complete.

[1688] The "means for accepting and responding to additional requests from customers" refers to a technology, device, or method for accepting additional requests from customers and responding accordingly.

[1689] "Means using an emotion engine that analyzes a user's voice and facial expressions to recognize their emotional state" refers to a technology, device, or method that analyzes a user's speech and facial expressions and identifies the user's emotions from the results.

[1690] The "means for adjusting dialogue content or services based on emotional state" refers to a technology, device, or method for dynamically changing the dialogue content or service content provided based on the results of recognizing the user's emotional state.

[1691] The present invention relates to a system that receives orders through dialogue with customers and automates everything from serving food to responding to additional requests. Furthermore, by combining this system with an emotion engine, it is possible to recognize the emotional state of the customer and respond appropriately accordingly. This system comprises the following means:

[1692] 1. User recognition methods:

[1693] The terminal uses a camera (e.g., a generic name "camera device") and a sensor (e.g., "sensor device") to recognize a user sitting at a table. Image recognition software (e.g., "image analysis software") is used to recognize the user. This confirms that a customer has arrived, and the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?". Speech recognition and generation use a speech recognition API and a speech generation API (e.g., "speech processing API").

[1694] 2. Emotion recognition means:

[1695] The device uses an emotion engine (e.g., "emotion analysis software") to analyze the user's voice and facial expressions and recognize their emotional state. If the user is excited, the device can speak in a calmer tone, asking, "Is there anything special happening today?" This adjusts the conversation with the customer to proceed smoothly.

[1696] 3. Menu suggestion methods:

[1697] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" If the user requests a recommendation, the server references ingredient inventory information and the user's past order history. Based on this information, a chat generation AI (for example, a "dialogue generation AI") is used to suggest, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." If the emotion engine recognizes that the user seems troubled, the device will provide a detailed explanation.

[1698] 4. Order receipt and confirmation methods:

[1699] If a user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" If the user answers "Yes," the order information will be sent to the server in JSON format via the HTTP API.

[1700] 5. Kitchen collaboration methods:

[1701] The server sends the order information to the cooking department's system (e.g., a "cooking management system") and instructs it to start cooking. The cooking management system receives the order and starts cooking. The order information is displayed on a display in the cooking department.

[1702] 6. Food Serving Method:

[1703] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. The terminal then carries the food to the customer, saying, "Sorry to keep you waiting. Here's your Japanese-style hamburger steak." It is also possible to use an automatic robot (for example, an "autonomous mobile robot") to serve the food.

[1704] 7. How to respond to requests:

[1705] If a user requests "an extra glass of water, please," the device will use voice recognition to understand the request and respond with "I understand, I'll bring it right away." The additional order information is sent to the server, which again instructs the cooking system to respond. If necessary, an automatic food distribution system can also be used. The device will respond quickly to additional requests, increasing customer satisfaction.

[1706] Specific examples

[1707] If the user has a specific request, the following might happen:

[1708] 1. The device recognizes the user and says, "Hello, welcome."

[1709] 2. A user asks, "What do you recommend?"

[1710] 3. The device responds, "Today's recommendations are Japanese-style hamburger steak and fried shrimp."

[1711] 4. The user places an order, "I'd like a Japanese-style hamburger, please."

[1712] 5. The terminal will ask again, "Would you like a Japanese-style hamburger?" and the user will answer "Yes."

[1713] 6. The terminal sends the order information to the server.

[1714] 7. The server sends the order information to the cooking management system.

[1715] 8. The cooking management system notifies the terminal that the food is ready, and the server issues serving instructions to the terminal.

[1716] 9. The terminal brings the food and serves it, saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[1717] Example prompt sentence:

[1718] "Hello, welcome. Would you like to see the menu?"

[1719] "Today's recommendation is Japanese-style hamburger steak and fried shrimp."

[1720] "Would you like a Japanese-style hamburger?"

[1721] In this way, this system, which combines an emotion engine, can provide services that correspond to the emotional state of the customer, thereby improving customer satisfaction and reducing the workload of staff.

[1722] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1723] Step 1:

[1724] Recognize the user

[1725] The terminal uses a camera and sensor device to recognize the user sitting at the table. It receives camera images and sensor information as input and uses image analysis software to confirm the user's presence. As output, it generates data indicating that the user has been recognized. Specifically, the terminal begins a dialogue by saying, "Hello, welcome. Would you like to see the menu?"

[1726] Step 2:

[1727] Emotion recognition

[1728] The device uses emotion analysis software to analyze the user's voice and facial expressions and recognize their emotional state. The user's voice and image data are taken as input and processed by the emotion analysis software. The output is the user's emotional state (e.g., excited, calm, anxious). Specifically, if the user is excited, the device will ask, "Did anything special happen today?" in a slightly calmer tone.

[1729] Step 3:

[1730] Menu suggestions

[1731] The device asks the user, "Have you decided what to order? Or would you like us to recommend a dish?" The device takes the user's response as input. When the user requests a recommendation, the server executes a database query to look up ingredient inventory information and past order history. As output, the conversation generation AI suggests, "Today's recommendation is Japanese-style hamburger steak and fried shrimp." Specifically, if the emotion analysis software recognizes that the user appears to be struggling, the device will provide a detailed explanation.

[1732] Step 4:

[1733] Order receipt and confirmation

[1734] When the user places an order by saying, "I'd like a Japanese-style hamburger, please," the terminal will reconfirm by asking, "Is a Japanese-style hamburger okay?" As input, the voice recognition API converts the user's utterance into text data and generates a confirmation question. As output, the user's final confirmation is obtained and order information is generated in JSON format. The specific operation is to send the order information to the server.

[1735] Step 5:

[1736] Kitchen Cooperation

[1737] The server sends the order information to the cooking management system and instructs it to start cooking. As input, the order information received by the server is analyzed and passed to the cooking management system. As output, the order details are displayed on the cooking management system and cooking begins. As a specific operation, the order details are displayed on a display in the cooking department.

[1738] Step 6:

[1739] Serving food

[1740] When the cooking management system notifies the server that cooking is complete, the server sends that information to the terminal, which then instructs the terminal to serve the food to the customer. As input, it receives the cooking completion notification and issues a distribution instruction to the terminal. As output, it generates data indicating that the food will be served. Specifically, the terminal brings the food while saying, "Sorry to keep you waiting, here's a Japanese-style hamburger steak."

[1741] Step 7:

[1742] Responding to requests

[1743] If a user requests "an extra glass of water, please," the device uses voice recognition to understand the request and responds with "I understand, I'll bring it right away." As input, the device acquires and analyzes the voice data of the additional request. As output, additional order information is generated and sent to the server. Specifically, the server again instructs the cooking management system to respond, and the automatic food distribution system is used as necessary. The device then brings the water to the user's table, fulfilling the request.

[1744] (Application example 2)

[1745] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1746] In order to increase customer satisfaction and provide efficient customer service in virtual stores, conventional systems have difficulty responding to customers' emotional states. This is because they are unable to adequately respond to customers' anxieties and questions in an online shopping environment. Furthermore, conventional systems lack the means to quickly and appropriately respond to customers' additional requests. Furthermore, conventional systems make it difficult to provide individually customized experiences, raising concerns about a decline in customer satisfaction.

[1747] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1748] In this invention, the server includes means for recognizing customers, means for interacting with customers and suggesting menu items, means for receiving and confirming customer orders, means for transmitting order information to cooking facilities, means for providing customers with ordered items, means for receiving and responding to customer requests for additional items, means for analyzing the user's voice and facial expressions to recognize their emotional state, and means for adjusting the content of the interaction and services according to their emotional state, thereby enabling the provision of customized services according to the customer's emotional state.

[1749] "Customer recognition means" refers to technology that uses devices such as cameras and sensors to identify users and detect their presence.

[1750] "Means of interacting with customers and suggesting menu items" refers to technologies such as voice dialogue systems and chatbots that interact with users to provide appropriate options.

[1751] "Means for receiving and confirming customer orders" refers to the interface and processing system for receiving and confirming user selections for goods and services.

[1752] The "means for transmitting order information to the cooking facility" refers to a communication technology or protocol for transmitting the user's order details to the cooking facility.

[1753] The "means of providing customers with ordered products" refers to a logistics system and actuator technology for reliably delivering ordered products to users.

[1754] The "means for accepting and responding to additional requests from customers" refers to an interface and processing system for quickly responding to additional requests and changes from users.

[1755] "Means for analyzing the user's voice and facial expressions and recognizing their emotional state" refers to image and voice recognition technology that analyzes the user's voice and facial expression data to determine their emotional state.

[1756] "Means for adjusting dialogue content and services according to emotional state" refers to technology for flexibly changing the content of the services and dialogue provided based on the recognized emotional state.

[1757] In this invention, the following hardware and software are used to realize a virtual store system that recognizes customers and provides customized dialogue and services according to their emotional state.

[1758] Hardware and Software Configuration

[1759] Hardware

[1760] Smartphones: Equipped with a camera and microphone, they can recognize users and accept voice input.

[1761] Head-mounted display (HMD): A device for viewing and interacting with a virtual environment.

[1762] software

[1763] Image and audio analysis library: Uses OpenCV, Google ML Kit, etc. to recognize users' faces and analyze their audio.

[1764] Emotion Recognition Engine: Uses the Emotion API to recognize the user's emotional state by analyzing their voice and facial expressions.

[1765] Generative chat AI: Generates dialogue using GPT models and other natural language generation models.

[1766] Web server: Uses Node.js and handles all communication and data management.

[1767] System Operation

[1768] User Awareness

[1769] First, the device recognizes the user using its camera and microphone. It uses OpenCV to analyze the video data from the camera and perform facial recognition. It also captures audio through the microphone and converts the user's speech into text using a speech analysis function such as Google ML Kit. This allows the device to confirm that the user is a new visitor and begin a conversation by asking, "Hello, welcome. What are you looking for today?"

[1770] Emotion recognition

[1771] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. For example, if the user is nervous, the device will speak in a calmer tone, asking, "Did anything special happen today?" In this way, the device adjusts the content of the conversation according to the user's emotional state.

[1772] Menu suggestions

[1773] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and a 4K smart TV" is generated.

[1774] Order receipt and confirmation

[1775] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server. The server then sends the order information to the food preparation facility or logistics system, and the product is prepared.

[1776] Responding to requests

[1777] Similarly, if the user makes an additional request such as "I would like gift wrapping," the terminal accepts the request and the server makes corresponding arrangements based on that request.

[1778] Examples of concrete examples and prompts

[1779] When a user enters a virtual store

[1780] User: "Can you recommend some products?"

[1781] Assistant: "Welcome! Today's recommendations are the latest Bluetooth earphones and a 4K smart TV. Which one are you interested in?"

[1782] Example of confirmation when a customer decides to purchase

[1783] Assistant: "Are you buying Bluetooth earphones? Understood. Would you like gift wrapping?"

[1784] User: "Yes, please."

[1785] Example prompts using the Emotion Engine

[1786] When the user is nervous

[1787] Assistant: "Is there anything special happening today? Let's talk for a bit and get settled. What product are you looking for?"

[1788] In this way, by using pre-set prompts, it becomes possible to provide customized services in virtual stores that correspond to the user's emotions and needs.

[1789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1790] Step 1: User Awareness

[1791] The device uses a camera and microphone to recognize the user's face and voice. Specifically, it processes the video data from the camera using OpenCV to perform facial recognition. It also analyzes the audio data collected by the microphone using Google ML Kit and converts the user's speech into text. Based on this data, the device begins a conversation by asking, "Hello, welcome. What are you looking for today?"

[1792] Input: Camera video data and audio data

[1793] Output: User's face recognition results and voice-to-text data

[1794] Step 2: Emotion Recognition

[1795] The device uses the Emotion API to analyze the user's voice and facial expressions and recognize their emotional state in real time. Specifically, audio and video data is input into the Emotion API, and the resulting emotional state is received as the analysis result. If the user is nervous, the device will speak in a calm tone, asking, "Is there anything special that happened today?"

[1796] Input: Audio and video data

[1797] Output: Emotion recognition results (e.g., nervousness, excitement, joy, etc.)

[1798] Step 3: Menu proposal

[1799] If a user is searching for a product, the device will suggest, "Here are today's recommended products." The server references inventory information and past order history, and uses chat generation AI to suggest individually customized products to the user. Dialogue content such as, "Today's recommendations are the latest Bluetooth earphones and 4K smart TVs" is generated.

[1800] Input: User requests, inventory information, past order history

[1801] Output: Customized product suggestions

[1802] Step 4: Order receipt and confirmation

[1803] When a user places an order by saying, "I'd like to purchase Bluetooth earphones," the device will ask, "Is this OK?" After this confirmation, the order information is sent to the server, which then sends the order information to the food preparation facility or logistics system.

[1804] Input: User's order details

[1805] Output: Confirmed order information, data sent to the server

[1806] Step 5: Response to requests

[1807] If the user makes an additional request, such as "I would like gift wrapping," the terminal accepts the request and the server arranges for the response based on that request. Specifically, the content of the additional request is converted into text and sent to the server. The server then carries out the necessary procedures and notifies the user of the final confirmation.

[1808] Input: Content of additional request

[1809] Output: Notification and confirmation message based on request

[1810] In this way, each step is executed sequentially to provide a customized service to the user.

[1811] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1813] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1814] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1815] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1816] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1817] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1818] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1819] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1820] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1821] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1822] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1823] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1824] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1825] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1826] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1827] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1828] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1829] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1830] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1831] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1832] The following is further disclosed regarding the above embodiment.

[1833] (Claim 1)

[1834] A means of customer recognition;

[1835] A means of interacting with customers and proposing menus,

[1836] a means of receiving and confirming customer orders;

[1837] a means for transmitting order information to a kitchen;

[1838] a means for delivering the ordered food to the customer;

[1839] A means of receiving and responding to additional customer requests;

[1840] A system including:

[1841] (Claim 2)

[1842] The system according to claim 1, further comprising means for suggesting a menu by referring to ingredient inventory information and past order history.

[1843] (Claim 3)

[1844] The system of claim 1, further comprising a means for interacting with customers using chat generation AI and accepting orders and requests.

[1845] "Example 1"

[1846] (Claim 1)

[1847] a means for recognizing the user;

[1848] A means of interacting with users and suggesting directions;

[1849] a means for receiving and verifying user requests;

[1850] means for transmitting request information to a cooking department;

[1851] a means for providing the requested food to the patron;

[1852] a means for accepting and responding to user addition requests;

[1853] A system including:

[1854] (Claim 2)

[1855] The system of claim 1, further comprising means for providing guidance by referencing raw material inventory information and past request history.

[1856] (Claim 3)

[1857] 2. The system of claim 1, further comprising means for interacting with a user using a dialogue generation artificial intelligence and accepting requests and demands.

[1858] (Claim 4)

[1859] 10. The system of claim 1, further comprising means for using a photographing device and a sensing device to recognize a user.

[1860] (Claim 5)

[1861] 10. The system of claim 1, further comprising means for using a data protocol for communication with the culinary department.

[1862] (Claim 6)

[1863] 3. The system of claim 2, comprising means for using a generative artificial intelligence model to generate guidance suggestions.

[1864] (Claim 7)

[1865] 10. The system of claim 1, further comprising means for responding to additional requests using an automated provisioning system.

[1866] "Application Example 1"

[1867] (Claim 1)

[1868] A means of customer recognition;

[1869] A means of interacting with customers and proposing menus,

[1870] a means of receiving and confirming customer orders;

[1871] a means for transmitting order information to a kitchen;

[1872] a means for delivering the ordered food to the customer;

[1873] A means of receiving and responding to additional customer requests;

[1874] A means of recognizing users using cameras and sensors;

[1875] A way to use chatbots to suggest menu items to customers, and

[1876] a means of transmitting order information to a database in real time using the customer's smartphone;

[1877] The system includes a means of notifying customers of the status of their orders.

[1878] (Claim 2)

[1879] The system according to claim 1, further comprising means for suggesting a menu by referring to ingredient inventory information and past order history.

[1880] (Claim 3)

[1881] The system of claim 1, further comprising a means for interacting with customers using chat generation AI and accepting orders and requests.

[1882] "Example 2: Combining Emotion Engines"

[1883] (Claim 1)

[1884] a means for recognizing a user;

[1885] A means of interacting with customers and proposing menus,

[1886] a means of receiving and confirming customer orders;

[1887] a means for transmitting order information to a food preparation department;

[1888] a means for delivering the ordered food to the customer;

[1889] A means of receiving and responding to additional customer requests;

[1890] a means for using an emotion engine that analyzes the user's voice and facial expressions to recognize the user's emotional state;

[1891] means for adjusting dialogue content and services based on emotional state;

[1892] A system including:

[1893] (Claim 2)

[1894] The system according to claim 1, further comprising means for suggesting a menu by referring to ingredient inventory information and past order history.

[1895] (Claim 3)

[1896] 10. The system of claim 1, further comprising means for interacting with customers using a conversation generation artificial intelligence to accept orders and requests.

[1897] "Application example 2 when combining emotion engines"

[1898] (Claim 1)

[1899] A means of customer recognition;

[1900] A means of interacting with customers and proposing menus,

[1901] a means of receiving and confirming customer orders;

[1902] means for transmitting order information to a food preparation facility;

[1903] a means for providing completed orders to customers;

[1904] A means of receiving and responding to additional customer requests;

[1905] means for analyzing the user's voice and facial expressions to recognize the user's emotional state;

[1906] A means for adjusting dialogue content and services according to emotional state;

[1907] A system including:

[1908] (Claim 2)

[1909] The system according to claim 1, further comprising means for suggesting a menu by referring to ingredient inventory information and past order history.

[1910] (Claim 3)

[1911] The system of claim 1, further comprising a means for interacting with customers using chat generation AI and accepting orders and requests. [Explanation of symbols]

[1912] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of customer recognition; A means of interacting with customers and proposing menus, a means of receiving and confirming customer orders; a means for transmitting order information to a kitchen; a means for delivering the ordered food to the customer; A means of receiving and responding to additional customer requests; A system including:

2. The system according to claim 1, further comprising means for suggesting a menu by referring to ingredient inventory information and past order history.

3. The system according to claim 1, further comprising a means for interacting with customers using a chat generation AI and accepting orders and requests.

Images