System

Abstract

A system is provided.SOLUTION: A system comprising: a terminal means operable by a user; a means for identifying a face of the user and analyzing attribute information by using face recognition software; a server means for acquiring a past order history of the user based on the analyzed attribute information; a means for generating a recommended menu based on the acquired order history and the attribute information; a means for displaying the generated recommended menu on the terminal means; and a means for transmitting and storing order information of the user to the server means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, restaurants have a large number of menu options, which means that first-time customers and those who are unsure what to choose lack appropriate advice and recommendations.In addition, technology that provides personalized menus that reflect users' preferences and past ordering history is not yet fully developed, which results in a standardized user experience that cannot meet individual needs. [Means for solving the problem]

[0005] To solve this problem, the present invention provides a system including a terminal means operable by a user, means for identifying the user's face using facial recognition software and analyzing attribute information, server means for acquiring the user's past order history based on the analyzed attribute information, server means for generating recommended menu items based on the acquired order history and attribute information, means for displaying the generated recommended menu items on the terminal means, and means for transmitting and saving the user's order information to the server means. This allows users to smoothly select a menu item that suits them and enables restaurants to efficiently provide personalized services.

[0006] The "terminal means" is an electronic device that can be operated by a user, and is a device for performing face recognition and menu display.

[0007] "Facial recognition software" is software that captures a user's face and estimates attribute information such as age and gender through image analysis.

[0008] "Attribute information" is data that indicates specific characteristics of a user, such as age and gender.

[0009] "Server means" refers to devices and software that store and manage user attribute information and past order history, and generate recommended menus based on that information.

[0010] "Past order history" is data that records products previously ordered by a user and related information.

[0011] The "recommended menu" is a list of recommended menus generated based on the user's attribute information and past order history.

[0012] The term "algorithm" refers to the calculation procedures and rules used by the server means to recommend the most suitable menu item based on the user's attribute information and order history.

[0013] A "unique ID" is an identifier assigned to each user to uniquely identify them. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. The following describes in detail an embodiment of the present invention.

[0036] Facial Recognition and Attribute Analysis

[0037] 1. Tablet startup and face recognition

[0038] The user picks up the tablet and launches the ordering application.

[0039] The device uses a built-in camera to capture the user's face.

[0040] The device runs facial recognition software that analyzes the captured image to generate demographic information about the user, such as their age and gender.

[0041] 2. Processing of Attribute Information by the Server

[0042] The terminal transmits the acquired attribute information to the server.

[0043] The server checks whether the user is already registered based on the received attribute information.

[0044] In the case of an existing user, the server retrieves the user's past order history from the database.

[0045] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[0046] Generate menu recommendations

[0047] 3. View your past order history

[0048] For existing users, the server analyzes the user's preferences based on past order history.

[0049] For new users, the server generates a general recommendation menu based on their attribute information.

[0050] 4. Generate recommended menu

[0051] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[0052] The server sends the generated recommended menu list to the terminal.

[0053] Displaying recommended menu items and ordering

[0054] 5. Display of recommended menu items

[0055] The terminal displays the recommended menu list received from the server on the ordering application.

[0056] The recommended menu is highlighted to make it easier for users to make a selection.

[0057] 6. Placing orders and collecting feedback

[0058] The user selects the desired menu item and presses the order confirmation button.

[0059] The terminal transmits the selected order information to the server.

[0060] The server stores the received order information in a database and uses it to reflect future recommendations.

[0061] Specific examples

[0062] For example, when a 30-year-old male user opens an ordering application on a tablet, the device captures his face using the camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server then stores the order information in a database and uses it as data for future recommendations.

[0063] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The user picks up the tablet and launches the ordering application.

[0067] Step 2:

[0068] The device uses its built-in camera to capture the user's face.

[0069] Step 3:

[0070] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[0071] Step 4:

[0072] The terminal transmits the acquired attribute information to the server.

[0073] Step 5:

[0074] Based on the attribute information received by the server, the server searches the database to check whether the user is already registered.

[0075] Step 6:

[0076] If the server is an existing user, it retrieves the user's past order history from the database.

[0077] Step 7:

[0078] When the server receives a new user, it registers face information and attribute information, generates a unique ID, and stores it in the database.

[0079] Step 8:

[0080] The server analyzes the user's preferences based on the past order history and attribute information acquired.

[0081] Step 9:

[0082] The server runs an algorithm to generate recommended menu items based on the user's attribute information and past order data.

[0083] Step 10:

[0084] The server sends the generated recommended menu list to the terminal.

[0085] Step 11:

[0086] The terminal displays the recommended menu list received from the server on the ordering application.

[0087] Step 12:

[0088] The user selects a product from the displayed menu and presses the order confirmation button.

[0089] Step 13:

[0090] The terminal transmits the selected order information to the server.

[0091] Step 14:

[0092] The server stores the received order information in a database.

[0093] Step 15:

[0094] The server uses the stored order information as data to reflect in future recommendations.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Existing systems have the problem that when providing personalized recommendations to users, the users themselves must perform many manual operations, making it difficult to automatically provide menus that meet the user's preferences. Another issue is that they cannot efficiently utilize user attribute information or past order history, making it difficult to provide highly accurate recommendations.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes an input device that can be operated by a user, a terminal that uses facial recognition software to identify the user's face and analyze their attribute information, a server that acquires the user's past order history based on the analyzed attribute information, a server that generates recommended menu items based on the acquired order history and attribute information, a display that displays the generated recommended menu items on the terminal, and a recording that transmits the user's order information to the server and stores it. This allows the user to receive menu recommendations that suit their preferences with minimal manual operation. Furthermore, highly accurate recommendations can be realized by efficiently utilizing attribute information and past order history.

[0100] The "user-operable input device means" is a device that allows a user to input information into the system by operating it himself.

[0101] "Facial recognition software" is a program that analyzes a user's facial image to identify the individual and extract attribute information.

[0102] "Terminal means" refers to a device that executes face recognition software, captures the user's face, and generates attribute information.

[0103] "Server means" refers to a central computer that stores and manages user attribute information and order history, and performs advanced calculation processing.

[0104] A "display means" is a device for visually presenting information to a user, or a partial function thereof.

[0105] "Storage means" refers to a device or function for storing data or information generated within the system.

[0106] "Information analysis means" refers to a program or function for analyzing collected data and extracting useful attribute information.

[0107] An "algorithm" is a set of calculations or procedures for solving a particular problem.

[0108] The present invention is directed to a system including a server and an input device operable by a user, which utilizes facial recognition and order history to recommend personalized menus. Hereinafter, embodiments of the present invention will be described in detail.

[0109] Facial Recognition and Attribute Analysis

[0110] 1. Tablet startup and face recognition

[0111] The user picks up the tablet and launches the ordering application. After turning on the tablet and tapping the application icon to launch the application, the message "Starting face recognition" is displayed.

[0112] The device captures the user's face using the built-in camera. When the camera is activated, it automatically recognizes the position of the user's face and captures it. This captured image is then analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition).

[0113] The device runs facial recognition software, which analyzes the captured image to generate demographic information about the user, such as their age and gender. Image analysis extracts the user's facial features and runs a model to estimate their age and gender. This process generates demographic information (e.g., "30-year-old male").

[0114] Server-based attribute processing

[0115] 2. Processing of Attribute Information by the Server

[0116] The device sends the acquired attribute information to the server. The generated attribute information is sent to the server in JSON format using the HTTP POST method.

[0117] The server queries a database (e.g., MySQL (registered trademark) or PostgreSQL) based on the received attribute information to check whether the user is already registered, and confirms the user's existence based on the face ID and attribute information.

[0118] If the user is an existing user, the server retrieves the user's past order history from the database. It uses an SQL query to retrieve the past order history associated with the user ID and makes it available within the application.

[0119] For a new user, the server registers the face information along with the attribute information, generates a new unique ID, and saves it in the database. The server registers the user's attribute information and the captured face image in the database and generates a new unique user ID.

[0120] Generate menu recommendations

[0121] 3. View your past order history

[0122] The server analyzes the user's preferences based on the user's past order history. It uses machine learning models (e.g., K-Means clustering and logistic regression) to analyze the user's past order history and identify the user's preferences.

[0123] 4. Generate recommended menu

[0124] The server runs a recommendation algorithm (e.g., collaborative filtering or content-based filtering) based on the user's attribute information and past order history. It combines the facial recognition results, attribute information, and past order history to generate a personalized menu using a pre-defined algorithm.

[0125] The server sends the generated recommended menu list to the device. It generates the recommended menu list in JSON format and returns it as an HTTP response.

[0126] Displaying recommended menu items and ordering

[0127] 5. Display of recommended menu items

[0128] The terminal displays the recommended menu list received from the server on the ordering application. The terminal parses the received data and displays it appropriately in the user interface. Recommended menu items are highlighted with eye-catching colors and fonts.

[0129] 6. Placing orders and collecting feedback

[0130] The user selects the desired menu item and presses the order confirmation button. By touching the screen, the user selects the menu item and taps the confirmation button. The terminal sends the selected order information to the server.

[0131] The terminal sends the selected order information to the server. Information such as the selected menu ID is sent in JSON format to the server.

[0132] The server stores the received order information in a database and uses it for future recommendations. The server also stores the transmitted data in a database and records it as an order history.

[0133] Specific examples

[0134] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[0135] Example prompts for generative AI models

[0136] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[0137] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] System program processing flow

[0140] Step 1: Boot up the tablet and start the ordering application

[0141] The user picks up the tablet and launches the ordering application. Specifically, they turn on the tablet and tap the application icon. The message "Starting face recognition" appears on the screen. The input is the user's touch operation, and the output is the initial screen of the application.

[0142] Step 2: Obtaining user attribute information through facial recognition

[0143] The device captures the user's face using the built-in camera and analyzes the image using facial recognition software (e.g., OpenCV, Amazon Rekognition). The camera activates, identifies the user's face, and acquires image data. The input is the captured facial image, and the output is attribute information such as age and gender. Specific operations include extracting facial feature points and estimating age and gender.

[0144] Step 3: Send attribute information to the server and confirm with the user

[0145] The device sends the acquired attribute information in JSON format to the server. The server queries the database and checks whether the user is already registered based on the received attribute information. The input is the JSON data of the attribute information, and the output is the result of determining whether the user is new or existing. Specifically, the database search and face ID matching are performed.

[0146] Step 4: Get the order history of an existing user

[0147] If the user is an existing user, the server retrieves the user's past order history from the database. The input is the user ID, and the output is the user's past order history. The specific operation is to retrieve the order history from the database using an SQL query.

[0148] Step 5: Register a new user

[0149] When a user is new, the server registers their face information along with their attribute information in the database and generates a new unique user ID. The input is attribute information and face image data, and the output is the new user ID. Specifically, the new user data is inserted into the database and a unique ID is generated.

[0150] Step 6: Menu recommendations based on order history and attribute information

[0151] The server generates recommended menus based on the user's past order history and attribute information. It runs a recommendation algorithm to create a personalized menu list. The input is attribute information and order history, and the output is a list of recommended menus. Specific operations include collaborative filtering and content-based filtering.

[0152] Step 7: Displaying recommended menu items

[0153] The server generates a recommended menu list and sends it to the terminal, which then displays it on the ordering application. The input is the recommended menu list, and the output is the menu displayed on the ordering application. Specifically, the received data is parsed and displayed on the user interface.

[0154] Step 8: User confirms order

[0155] The user selects the menu item they want and presses the order confirmation button. The input is the user's selection and touch operation, and the output is the order data. The specific operation involves selecting a menu item and pressing the confirmation button.

[0156] Step 9: Send and store order information on the server

[0157] The terminal sends the selected order information in JSON format to the server. The server saves the order information in a database. The input is the order information, and the output is updating the database. Specifically, the order data is sent to the server and saved in the database.

[0158] Specific examples

[0159] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[0160] Example prompts for generative AI models

[0161] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[0162] (Application example 1)

[0163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] Conventional personalized menu and product recommendation systems mainly use terminals or PCs, which poses challenges in improving user convenience and experience. Furthermore, there is a lack of means for providing recommendation information in real time within the store, requiring users to actively operate their terminals. Therefore, there is a need for a more intuitive and convenient way to provide personalized information to users.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0166] In this invention, the server includes a terminal means operable by a user, a means for identifying a user's face and analyzing attribute information using face recognition software, a server means for acquiring the user's past order history based on the analyzed attribute information, a server means for generating recommended menu items based on the acquired order history and attribute information, a means for displaying the generated recommended menu items on the terminal means, a means for transmitting and storing the user's order information to the server means, and an information display means for performing face recognition and displaying recommended menu items using smart glasses. This enables users to intuitively and conveniently receive personalized menu and product recommendations in a store.

[0167] "User-operable terminal means" refers to an electronic terminal device that can be operated by the user himself / herself to input and display information.

[0168] "Means of identifying a user's face and analyzing attribute information using facial recognition software" refers to a combination of software and hardware that uses a camera and facial recognition algorithm to capture a user's face, and then extracts and analyzes attribute information such as age and gender from the video data.

[0169] The "server means for acquiring the user's past order history based on the analyzed attribute information" is a server that has the function of searching a database based on the user's attribute information and acquiring the user's past order history.

[0170] The "server means for generating recommended menus based on the acquired order history and attribute information" refers to a server that has an algorithm for analyzing the acquired user order history and attribute information and automatically generating recommended menus based thereon.

[0171] The "means for displaying the generated recommended menu on the terminal means" refers to software and hardware that has the function of displaying the recommended menu generated by the server on the user's terminal.

[0172] The "means for transmitting user order information to server means and storing it" refers to communication means and storage means for transmitting the menu selected by the user to the server and storing that information in a database.

[0173] "Information display means for performing facial recognition and displaying recommended menus using smart glasses" refers to a device and system that performs facial recognition through smart glasses worn by the user and displays personalized recommended menus based on the results on the glasses' display.

[0174] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. A specific embodiment of the present invention will be described.

[0175] Facial Recognition and Attribute Analysis

[0176] 1. Tablet startup and face recognition

[0177] When a user wears the smart glasses and enters a store, the smart glasses automatically activate their camera and capture the user's face.

[0178] Facial recognition software built into the smart glasses analyzes this image and generates demographic information about the user, such as their age and gender.

[0179] 2. Processing of Attribute Information by the Server

[0180] The smart glasses transmit the acquired attribute information to the server.

[0181] The server checks whether the user is already registered based on the received attribute information.

[0182] In the case of an existing user, the server retrieves the user's past order history from the database.

[0183] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[0184] Generate menu recommendations

[0185] 3. View your past order history

[0186] For existing users, the server analyzes the user's preferences based on past order history.

[0187] For new users, the server generates a general recommendation menu based on their attribute information.

[0188] 4. Generate recommended menu

[0189] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[0190] The server sends the generated recommended menu list to the smart glasses.

[0191] Displaying recommended menu items and ordering

[0192] 5. Display of recommended menu items

[0193] The smart glasses display the recommended menu list received from the server on the glasses' display.

[0194] The recommended menu is highlighted to make it easier for users to make a selection.

[0195] 6. Placing orders and collecting feedback

[0196] The user operates the smart glasses to select the desired menu item and confirm the order.

[0197] The smart glasses transmit the selected order information to the server.

[0198] The server stores the received order information in a database and uses it to reflect future recommendations.

[0199] To implement this system, the following hardware and software are used:

[0200] Hardware

[0201] Smart glasses (e.g., Google® Glass®, etc.)

[0202] Built-in camera (integrated into smart glasses)

[0203] software

[0204] OpenCV library (face recognition)

[0205] REST API (sending and receiving face images and attribute information)

[0206] Smart Glasses SDK

[0207] Specific examples

[0208] For example, when a 30-year-old male user wears smart glasses and enters a store, the camera in the smart glasses captures his face and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. As a result, it is determined that he has previously preferred casual sneakers, and the latest sneaker collection is generated as a recommended menu. The recommended menu is displayed on the smart glasses, and the user selects new sneakers from the menu and confirms the purchase. The server saves the order information in a database and uses it as data for future recommendations.

[0209] Prompt Sentence Examples

[0210] "You are an AI that recommends products in a store. When a user enters the store, it recognizes their face and suggests products based on their past purchase history. For example, if the user has liked sneakers in the past, it will recommend a new sneaker collection."

[0211] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0212] Step 1: Capture the user's face and generate attribute information

[0213] When a user wearing smart glasses enters a store, the glasses automatically activate the camera. The camera captures the user's face, and facial recognition software analyzes the facial image to generate attribute information such as age and gender. The input is the camera's video data, and the output is the user's attribute information.

[0214] Step 2: Sending attribute information to the server

[0215] The smart glasses send the attribute information generated by the facial recognition software to the server. The input is the attribute information, and the output is the attribute information sent to the server. The smart glasses' communication module is used.

[0216] Step 3: Get past order history

[0217] The server checks whether the user is already registered based on the attribute information it receives. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers the user's face information along with the attribute information, generates a unique ID, and saves it in the database. The input is attribute information, and the output is the user's past order history or new user registration information.

[0218] Step 4: Generate a recommendation menu

[0219] The server runs an algorithm that generates menu recommendations based on the user's attribute information and past order history. The input is attribute information and order history, and the output is a list of recommended menu items. A generative AI model is used to analyze past data and make predictions.

[0220] Step 5: Send recommendations to your smart glasses

[0221] The server sends the generated recommended menu list to the smart glasses. The input is the recommended menu list, and the output is the recommended menu list sent to the smart glasses. A communication module is used.

[0222] Step 6: Displaying recommended menu items

[0223] The smart glasses display the recommended menu list received from the server on the glasses' display. The input is the recommended menu list, and the output is the menu displayed on the display. The smart glasses' display control module is used.

[0224] Step 7: Confirm your order

[0225] The user operates the smart glasses to select the desired menu and confirm the order. The input is the user's selection information, and the output is the selected order information. The interface of the smart glasses is used.

[0226] Step 8: Send and store order information on the server

[0227] The smart glasses send the selected order information to the server, which stores the information in a database. The input is the order information, and the output is the order information stored on the server. A communication module and a database management system are used.

[0228] At each step, specific data processing and calculations are performed, enabling users to receive intuitive, convenient, and personalized menu and product recommendations in-store, improving the user experience.

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. The following describes in detail embodiments of the present invention.

[0231] Facial Recognition and Attribute Analysis

[0232] 1. Tablet startup and face recognition

[0233] The user picks up the tablet and launches the ordering application.

[0234] The device uses a built-in camera to capture the user's face.

[0235] The device runs facial recognition software and analyzes the user's age, gender and other demographic information from the captured image.

[0236] 2. Emotion Recognition by Emotion Engine

[0237] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state (e.g., joy, surprise, sadness, anger, etc.).

[0238] 3. Processing of attribute information and emotion information by the server

[0239] The terminal transmits the acquired attribute information and emotion information to the server.

[0240] Based on the received information, the server searches the database to check whether the user is already registered.

[0241] In the case of an existing user, the server retrieves the user's past order history from the database.

[0242] For new users, the server registers their facial information along with their attribute and emotion information, generates a unique ID, and stores it in the database.

[0243] Generate menu recommendations

[0244] 4. View your past order history

[0245] For existing users, the server analyzes the user's preferences based on past order history.

[0246] For new users, the server generates a general recommendation menu based on attribute information and emotional information.

[0247] 5. Generate recommended menu

[0248] The server analyzes the user's attribute information, emotional information, and past order data and runs an algorithm to generate menu recommendations.

[0249] If the user is excited, the system will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy menus, making recommendations based on the user's emotional state.

[0250] The server sends the generated recommended menu list to the terminal.

[0251] Displaying recommended menu items and ordering

[0252] 6. Display recommended menu items

[0253] The terminal displays the recommended menu list received from the server on the ordering application.

[0254] The recommended menu is highlighted to make it easier for users to make a selection.

[0255] 7. Confirming orders and collecting feedback

[0256] The user selects a product from the displayed menu and presses the order confirmation button.

[0257] The terminal transmits the selected order information to the server.

[0258] The server stores the received order information in a database and uses it to reflect future recommendations.

[0259] Specific examples

[0260] For example, when a 30-year-old male user opens an ordering application on a tablet, the device uses a camera to capture his face and recognize his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu is displayed on the device, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[0261] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[0262] The processing flow will be explained below.

[0263] Step 1:

[0264] The user picks up the tablet and launches the ordering application.

[0265] Step 2:

[0266] The device uses its built-in camera to capture the user's face.

[0267] Step 3:

[0268] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[0269] Step 4:

[0270] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[0271] Step 5:

[0272] The terminal transmits the acquired attribute information and emotion information to the server.

[0273] Step 6:

[0274] Based on the attribute information and emotion information received by the server, the server searches the database to check whether the user is already registered.

[0275] Step 7:

[0276] If the server is an existing user, it retrieves the user's past order history from the database.

[0277] Step 8:

[0278] When the server receives a new user, it registers face information, attribute information, and emotion information, generates a unique ID, and stores it in the database.

[0279] Step 9:

[0280] The server analyzes the user's preferences based on the past order history, attribute information, and emotional information acquired by the server.

[0281] Step 10:

[0282] The server runs an algorithm to generate menu recommendations based on the user's attribute information, emotional information, and past order data.

[0283] Step 11:

[0284] The system adjusts the recommended menu according to the user's emotional state; for example, if the user is excited, it will recommend hearty dishes or sweets, and if the user is relaxed, it will recommend light meals or healthy options.

[0285] Step 12:

[0286] The server sends the generated recommended menu list to the terminal.

[0287] Step 13:

[0288] The terminal displays the recommended menu list received from the server on the ordering application.

[0289] Step 14:

[0290] The user selects a product from the displayed menu and presses the order confirmation button.

[0291] Step 15:

[0292] The terminal transmits the selected order information to the server.

[0293] Step 16:

[0294] The server stores the received order information in a database.

[0295] Step 17:

[0296] The server uses the stored order information as data to reflect in future recommendations.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] To provide personalized services at restaurants and other establishments, it is important to take into account not only the user's attribute information and past ordering history, but also their emotional state at the time. However, conventional systems have not been able to properly analyze the user's emotional state and reflect it in recommendations. This has made it difficult to provide even more personalized services.

[0300] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes terminal means operable by the user, means for identifying the user's face and analyzing attribute information using face recognition software, means for analyzing the user's emotional state using the analyzed attribute information and an emotion engine, means for acquiring the user's past order history based on the analyzed attribute information and emotion information, means for generating recommended menus based on the acquired order history, attribute information, and emotion information, means for displaying the generated recommended menus on the terminal means, and means for transmitting the user's order information to the server means and storing it. This makes it possible to analyze the user's emotional state and provide personalized recommended menus based on the analyzed emotional state.

[0301] "User-operable terminal means" refers to an electronic device that can be physically operated by a user, and that can execute a specific application to display information or input data.

[0302] "Facial recognition software" refers to a program or algorithm that analyzes facial images captured by a camera or other imaging device to detect, identify, and analyze certain facial features.

[0303] "Attribute information" refers to personal characteristic data such as a user's age and gender, and includes data obtained by facial recognition or other analytical means.

[0304] An "emotion engine" refers to an algorithm or program that determines a user's current emotional state from their facial expressions, voice, etc.

[0305] "Server means" refers to a computer system or network device for storing, processing, and transmitting data, and includes programs and services for managing and executing a series of operations.

[0306] "Analyzed attribute information and emotion information" refers to the user's personal characteristic data and emotional state data obtained and analyzed by the facial recognition software and emotion engine.

[0307] "Past order history" refers to data that records the details of orders previously placed by a user, and is used to analyze the user's preferences.

[0308] A "recommended menu" refers to a list of dishes and drinks suggested to the user based on the acquired data, and is generated by a specific algorithm.

[0309] "Order information" refers to information about the menu items selected by the user, and includes data for specifically recording the contents of the order.

[0310] The present invention is a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. This system utilizes facial recognition and emotion analysis of the user to recommend personalized menus. Specific embodiments for implementing the present invention are described below.

[0311] First, a terminal that can be operated by the user is used. Examples of such terminals include tablets and smartphones. This terminal has an ordering application installed and is activated by the user's operation.

[0312] The device uses the built-in camera to capture the user's face. This process uses a built-in camera module (e.g., a standard webcam). After the facial image is captured, facial recognition software (e.g., a common facial recognition API) is used to identify the user's face and analyze attribute information such as age and gender. This analysis is performed using algorithms built into the facial recognition software.

[0313] Next, the device inputs the facial expression data obtained from the facial image into an emotion engine. The emotion engine (e.g., emotion analysis API) analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). After obtaining the analysis results, the device transmits this attribute information and emotion information to the server.

[0314] The server checks whether the user is already registered in the database based on the received attribute and emotion information. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers new face, attribute, and emotion information, and generates and stores a unique ID. This registration process uses database manipulation software (e.g., a standard SQL database).

[0315] The server analyzes user preferences based on past data and newly acquired information. For existing users, it generates personalized menu recommendations based on past order history. For new users, it generates general menu recommendations based on attribute information and emotional information. This menu generation uses machine learning algorithms (e.g., a typical recommender system).

[0316] The generated recommended menu is sent from the server to the device and displayed on the ordering application. The user selects products based on this and confirms the order. The order information is sent from the device to the server, which stores it in a database and reflects it in the next recommendation.

[0317] As a concrete example, consider the case where a 30-year-old male user opens an ordering application on a tablet. The device uses a camera to capture his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state of "happiness" from his facial expression. The server confirms that the user's facial information has already been registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta. The server generates recommended menu items such as specialty pizza and carbonara pasta, and recommends more of certain menu items considering his state of happiness. The recommended menu items are displayed on the device, and the user makes a selection and confirms the order. The server stores this order information in a database and uses it for future recommendations.

[0318] The above is a specific embodiment of the present invention. This system allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide more personalized services that take emotional states into consideration.

[0319] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0320] Step 1:

[0321] The user turns on the tablet and launches the dedicated ordering application.

[0322] Input: User actions

[0323] Specific action: Tap the ordering application icon on the tablet's home screen.

[0324] Output: The ordering application is launched.

[0325] Step 2:

[0326] The device activates its built-in camera to capture the user's face.

[0327] Input: A face image captured from the camera

[0328] Specific operation: The tablet's front camera takes a picture of the user's face, and facial image data is generated.

[0329] Output: Face image data

[0330] Step 3:

[0331] The device uses facial recognition software to analyze the user's age, gender, and other attribute information from the captured facial image.

[0332] Input: Facial image data

[0333] Specific operation: Calls a facial recognition API and runs an image processing algorithm to estimate age and gender.

[0334] Output: Attribute information (age, gender, etc.)

[0335] Step 4:

[0336] The terminal inputs the acquired facial image data and attribute information into an emotion engine to analyze the user's emotional state.

[0337] Input: Facial image data, attribute information

[0338] Specific operation: Calls the emotion analysis API and quantifies the emotional state (e.g., joy, surprise, sadness, anger, etc.) from the facial image.

[0339] Output: Emotional information

[0340] Step 5:

[0341] The terminal transmits the analyzed attribute information and emotion information to the server.

[0342] Input: attribute information, emotion information

[0343] Specific operation: Data is sent to the server using the HTTPS protocol.

[0344] Output: Data sent to the server

[0345] Step 6:

[0346] Based on the information received by the server, it checks whether the user is registered in the database.

[0347] Input: attribute information, emotion information

[0348] Specific behavior: Runs an SQL database query to check the user's enrollment status based on facial information.

[0349] Output: Information on whether the user is an existing user or a new user

[0350] Step 7:

[0351] If the user is an existing user, the server retrieves the user's past order history from the database, and if the user is a new user, the server registers new face information, attribute information, and emotion information, and generates and saves a unique ID.

[0352] Input: User registration status, attribute information, emotional information

[0353] Specific operation: For existing users, retrieve past order data based on the user ID, and for new users, create a new record and insert the data.

[0354] Output: If an existing user, the retrieved past order history. If a new user, the registration completion ID.

[0355] Step 8:

[0356] Based on the data acquired by the server and newly analyzed information, the server analyzes the user's preferences and generates recommended menus.

[0357] Input: past order history, attribute information, emotional information

[0358] What it does: It uses machine learning algorithms to analyze and generate a menu tailored to the user's preferences.

[0359] Output: Recommended menu list

[0360] Step 9:

[0361] The server sends the generated recommended menu list to the terminal.

[0362] Input: Recommended menu list

[0363] Specific operation: Sends data to the terminal using the HTTPS protocol.

[0364] Output: Recommended menu list sent to the device

[0365] Step 10:

[0366] The terminal displays the recommended menu list received from the server on the ordering application.

[0367] Input: Recommended menu list

[0368] What it does: Displays the list on the ordering application screen so that the user can visually confirm it.

[0369] Output: Screen showing recommended menu

[0370] Step 11:

[0371] The user selects a product from the displayed menu and presses the order confirmation button.

[0372] Input: Select from the recommended menu

[0373] Specific actions: Tap to select a menu item and tap the order confirmation button.

[0374] Output: Selected menu and order confirmation information

[0375] Step 12:

[0376] The terminal transmits the selected order information to the server.

[0377] Input: Selected menu and order confirmation information

[0378] Specific operation: Order data is sent to the server using the HTTPS protocol.

[0379] Output: Order information sent to the server

[0380] Step 13:

[0381] The server stores the received order information in a database and uses it to make future recommendations.

[0382] Input: Order Information

[0383] Specific operation: Inserts order information into the database and saves the new data.

[0384] Output: Order information stored in the database

[0385] This concludes the overall processing flow of the system, which provides a personalized menu based on the user's emotional state and preferences.

[0386] (Application example 2)

[0387] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional personalized menu recommendation systems generally make static recommendations based on a user's past order history and attribute information, and are unable to consider the customer's emotional state in real time. This makes it difficult to provide the optimal menu based on the customer's current mood, and this has led to the problem of not being able to sufficiently increase customer satisfaction.

[0389] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user attribute information based on the analyzed attribute information and emotional state, emotion analysis means for analyzing the user's emotional state in real time, and means for acquiring the user's past order history based on the analyzed attribute information and emotional state. This makes it possible to provide the user with a personalized menu in real time that corresponds to their current emotional state.

[0390] "User-operable terminal means" refers to a terminal such as a smartphone, tablet, or smart glasses that allows a user to place an order, check the menu, etc.

[0391] "Means of identifying a user's face using facial recognition software and analyzing attribute information" refers to software that analyzes a user's attributes, such as age and gender, based on an image of the user's face taken with a camera.

[0392] "Means for analyzing user attribute information based on analyzed attribute information and emotional state" refers to a system for analyzing detailed attributes and preferences of a user using the acquired attribute information and emotional state.

[0393] "Emotion analysis means for analyzing the user's emotional state in real time" refers to a system for analyzing emotions such as joy, surprise, sadness, and anger based on the user's facial expressions.

[0394] "Server means for acquiring a user's past order history based on the analyzed attribute information and emotional state" refers to a system in which a server acquires a user's past order data based on the user's attribute information and emotional information.

[0395] "Server means for generating recommended menus based on acquired order history, attribute information, and emotional state" refers to a system in which a server generates a menu suitable for a user based on past order history, attribute information, and emotional state.

[0396] "Means for displaying the generated recommended menu on a terminal means and on smart glasses or other smart devices" refers to a system that displays the recommended menu generated by the server on a display device (such as smart glasses) in real time.

[0397] "Means for transmitting and storing user order information to server means" refers to a system that transmits order information selected by a user to a server and stores it for use in future recommendations.

[0398] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. Specifically, the system is configured and operates as follows.

[0399] User Interfaces and Terminals

[0400] Users use devices such as smart glasses, smartphones, or tablets, each with a built-in camera and capable of running facial recognition software, and an ordering application installed, which provides an interface for customers to select menu items and enter their order information.

[0401] Facial Recognition and Attribute Analysis

[0402] When a user puts on the smart glasses, the built-in camera captures the user's face, which is then analyzed by facial recognition software to determine the user's age, gender, and other demographic information, which is then analyzed in real time and sent to a server.

[0403] Emotional state analysis

[0404] Facial expression data obtained from facial images is input into the emotion engine, which analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). This makes it possible to understand the user's current mood and emotions. The analysis results are also sent to the server.

[0405] Server Processing

[0406] The server receives attribute information and emotional state from the device, and for existing users, retrieves past order history from the database. For new users, the server registers facial information along with attribute information and emotional information, generates a unique ID, and stores it in the database.

[0407] Generate menu recommendations

[0408] The server runs an algorithm that recommends the most suitable menu for the user based on the analyzed attribute information, emotional state, and past order history. If the user is excited, the server will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy options.

[0409] Displaying recommended menu items and ordering

[0410] The generated recommended menu list is displayed in real time on smart glasses or other smart devices. The user selects items from the displayed menu and confirms the order. The device then sends the selected order information to the server, which stores this information in a database and uses it to reflect future recommendations.

[0411] Specific examples

[0412] For example, suppose a 30-year-old male user is wearing smart glasses. The smart glasses' camera captures his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu items are displayed on the smart glasses, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[0413] Prompt Sentence Examples

[0414] When the customer looks at the menu displayed in the smart glasses, menu recommendations tailored to his emotions are displayed.

[0415] "We recommend the following menu items: Specialty Pizza, Carbonara Pasta"

[0416] Simply move your hand to select your order and the glasses will navigate you to the menu item's location in the store.

[0417] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[0418] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0419] Step 1:

[0420] The user puts on the smart glasses and launches the ordering application.

[0421] Input: Smart glasses activation signal.

[0422] Output: The camera starts capturing face images.

[0423] Specific operation: The smart glasses' built-in camera captures an image of the user's face, and the image data is sent to the ordering application.

[0424] Step 2:

[0425] The device uses facial recognition software to analyze the user's face.

[0426] Input: A captured face image.

[0427] Output: User's demographic information such as age and gender.

[0428] Specific operation: Facial recognition software analyzes facial images, obtains attribute information such as age and gender, and temporarily stores this information in the device database.

[0429] Step 3:

[0430] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[0431] Input: A captured face image.

[0432] Output: The user's emotional state (e.g., happy, surprised, sad, angry, etc.).

[0433] Specific operation: The emotion engine analyzes the facial expression data, identifies the emotional state, and temporarily stores that information in the device database.

[0434] Step 4:

[0435] The terminal transmits the acquired attribute information and emotion information to the server.

[0436] Input: attribute information, emotion information.

[0437] Output: Attribute information and emotion information sent to the server.

[0438] Specific operation: The terminal transmits the analyzed attribute information and emotion information to the server as a data packet.

[0439] Step 5:

[0440] The server acquires the user's past order history based on the received information.

[0441] Input: attribute information, emotion information.

[0442] Output: Past order history or register as a new user.

[0443] Specific operation: Using the attribute information and emotion information received by the server, the server searches the database for past order history for existing users, and for new users, it registers them along with their facial information and generates a unique ID.

[0444] Step 6:

[0445] The server generates a recommended menu based on the analyzed information.

[0446] Input: Past order history, attribute information, emotional state.

[0447] Output: Recommended menu list.

[0448] Specific operation: The server uses a generative AI model to analyze past order history, attribute information, and current emotional state, and generates the most suitable recommended menu for the user.

[0449] Step 7:

[0450] The server sends the generated recommended menu list to the terminal.

[0451] Input: Recommended menu list.

[0452] Output: A list of recommended menus sent to the device.

[0453] Specific operation: The recommended menu list generated by the server is sent to the terminal as a data packet.

[0454] Step 8:

[0455] The device will display a list of recommended menu items and display them on the smart glasses.

[0456] Input: Recommended menu list.

[0457] Output: Recommendation menu displayed on smart glasses.

[0458] Specific operation: The recommended menu list received by the device is displayed in real time on the smart glasses display.

[0459] Step 9:

[0460] The user selects a product from the recommended menu and confirms the order.

[0461] Input: Recommended menu list.

[0462] Output: Selected order information.

[0463] Specific operation: The user selects a product from the menu displayed on the smart glasses and presses the order confirmation button.

[0464] Step 10:

[0465] The terminal transmits the selected order information to the server, which stores the information in a database.

[0466] Enter: Selected order information.

[0467] Output: Order information stored on the server.

[0468] Specific operation: The terminal sends the user's selected order information to the server, which stores the information in a database so that it can be reflected in future recommendations.

[0469] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0471] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0472] [Second embodiment]

[0473] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0474] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0476] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0480] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0481] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0482] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0483] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0484] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0485] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. The following describes in detail an embodiment of the present invention.

[0486] Facial Recognition and Attribute Analysis

[0487] 1. Tablet startup and face recognition

[0488] The user picks up the tablet and launches the ordering application.

[0489] The device uses a built-in camera to capture the user's face.

[0490] The device runs facial recognition software that analyzes the captured image to generate demographic information about the user, such as their age and gender.

[0491] 2. Processing of Attribute Information by the Server

[0492] The terminal transmits the acquired attribute information to the server.

[0493] The server checks whether the user is already registered based on the received attribute information.

[0494] In the case of an existing user, the server retrieves the user's past order history from the database.

[0495] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[0496] Generate menu recommendations

[0497] 3. View your past order history

[0498] For existing users, the server analyzes the user's preferences based on past order history.

[0499] For new users, the server generates a general recommendation menu based on their attribute information.

[0500] 4. Generate recommended menu

[0501] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[0502] The server sends the generated recommended menu list to the terminal.

[0503] Displaying recommended menu items and ordering

[0504] 5. Display of recommended menu items

[0505] The terminal displays the recommended menu list received from the server on the ordering application.

[0506] The recommended menu is highlighted to make it easier for users to make a selection.

[0507] 6. Placing orders and collecting feedback

[0508] The user selects the desired menu item and presses the order confirmation button.

[0509] The terminal transmits the selected order information to the server.

[0510] The server stores the received order information in a database and uses it to reflect future recommendations.

[0511] Specific examples

[0512] For example, when a 30-year-old male user opens an ordering application on a tablet, the device captures his face using the camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server then stores the order information in a database and uses it as data for future recommendations.

[0513] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[0514] The processing flow will be explained below.

[0515] Step 1:

[0516] The user picks up the tablet and launches the ordering application.

[0517] Step 2:

[0518] The device uses its built-in camera to capture the user's face.

[0519] Step 3:

[0520] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[0521] Step 4:

[0522] The terminal transmits the acquired attribute information to the server.

[0523] Step 5:

[0524] Based on the attribute information received by the server, the server searches the database to check whether the user is already registered.

[0525] Step 6:

[0526] If the server is an existing user, it retrieves the user's past order history from the database.

[0527] Step 7:

[0528] When the server receives a new user, it registers face information and attribute information, generates a unique ID, and stores it in the database.

[0529] Step 8:

[0530] The server analyzes the user's preferences based on the past order history and attribute information acquired.

[0531] Step 9:

[0532] The server runs an algorithm to generate recommended menu items based on the user's attribute information and past order data.

[0533] Step 10:

[0534] The server sends the generated recommended menu list to the terminal.

[0535] Step 11:

[0536] The terminal displays the recommended menu list received from the server on the ordering application.

[0537] Step 12:

[0538] The user selects a product from the displayed menu and presses the order confirmation button.

[0539] Step 13:

[0540] The terminal transmits the selected order information to the server.

[0541] Step 14:

[0542] The server stores the received order information in a database.

[0543] Step 15:

[0544] The server uses the stored order information as data to reflect in future recommendations.

[0545] Example 1

[0546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0547] Existing systems have the problem that when providing personalized recommendations to users, the users themselves must perform many manual operations, making it difficult to automatically provide menus that meet the user's preferences. Another issue is that they cannot efficiently utilize user attribute information or past order history, making it difficult to provide highly accurate recommendations.

[0548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0549] In this invention, the server includes an input device that can be operated by a user, a terminal that uses facial recognition software to identify the user's face and analyze their attribute information, a server that acquires the user's past order history based on the analyzed attribute information, a server that generates recommended menu items based on the acquired order history and attribute information, a display that displays the generated recommended menu items on the terminal, and a recording that transmits the user's order information to the server and stores it. This allows the user to receive menu recommendations that suit their preferences with minimal manual operation. Furthermore, highly accurate recommendations can be realized by efficiently utilizing attribute information and past order history.

[0550] The "user-operable input device means" is a device that allows a user to input information into the system by operating it himself.

[0551] "Facial recognition software" is a program that analyzes a user's facial image to identify the individual and extract attribute information.

[0552] "Terminal means" refers to a device that executes face recognition software, captures the user's face, and generates attribute information.

[0553] "Server means" refers to a central computer that stores and manages user attribute information and order history, and performs advanced calculation processing.

[0554] A "display means" is a device for visually presenting information to a user, or a partial function thereof.

[0555] "Storage means" refers to a device or function for storing data or information generated within the system.

[0556] "Information analysis means" refers to a program or function for analyzing collected data and extracting useful attribute information.

[0557] An "algorithm" is a set of calculations or procedures for solving a particular problem.

[0558] The present invention is directed to a system including a server and an input device operable by a user, which utilizes facial recognition and order history to recommend personalized menus. Hereinafter, embodiments of the present invention will be described in detail.

[0559] Facial Recognition and Attribute Analysis

[0560] 1. Tablet startup and face recognition

[0561] The user picks up the tablet and launches the ordering application. After turning on the tablet and tapping the application icon to launch the application, the message "Starting face recognition" is displayed.

[0562] The device captures the user's face using the built-in camera. When the camera is activated, it automatically recognizes the position of the user's face and captures it. This captured image is then analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition).

[0563] The device runs facial recognition software, which analyzes the captured image to generate demographic information about the user, such as their age and gender. Image analysis extracts the user's facial features and runs a model to estimate their age and gender. This process generates demographic information (e.g., "30-year-old male").

[0564] Server-based attribute processing

[0565] 2. Processing of Attribute Information by the Server

[0566] The device sends the acquired attribute information to the server. The generated attribute information is sent to the server in JSON format using the HTTP POST method.

[0567] The server queries a database (e.g., MySQL or PostgreSQL) based on the received attribute information to see if the user is already registered, and verifies the user's existence based on the face ID and attribute information.

[0568] If the user is an existing user, the server retrieves the user's past order history from the database. It uses an SQL query to retrieve the past order history associated with the user ID and makes it available within the application.

[0569] For a new user, the server registers the face information along with the attribute information, generates a new unique ID, and saves it in the database. The server registers the user's attribute information and the captured face image in the database and generates a new unique user ID.

[0570] Generate menu recommendations

[0571] 3. View your past order history

[0572] The server analyzes the user's preferences based on the user's past order history. It uses machine learning models (e.g., K-Means clustering and logistic regression) to analyze the user's past order history and identify the user's preferences.

[0573] 4. Generate recommended menu

[0574] The server runs a recommendation algorithm (e.g., collaborative filtering or content-based filtering) based on the user's attribute information and past order history. It combines the facial recognition results, attribute information, and past order history to generate a personalized menu using a pre-defined algorithm.

[0575] The server sends the generated recommended menu list to the device. It generates the recommended menu list in JSON format and returns it as an HTTP response.

[0576] Displaying recommended menu items and ordering

[0577] 5. Display of recommended menu items

[0578] The terminal displays the recommended menu list received from the server on the ordering application. The terminal parses the received data and displays it appropriately in the user interface. Recommended menu items are highlighted with eye-catching colors and fonts.

[0579] 6. Placing orders and collecting feedback

[0580] The user selects the desired menu item and presses the order confirmation button. By touching the screen, the user selects the menu item and taps the confirmation button. The terminal sends the selected order information to the server.

[0581] The terminal sends the selected order information to the server. Information such as the selected menu ID is sent in JSON format to the server.

[0582] The server stores the received order information in a database and uses it for future recommendations. The server also stores the transmitted data in a database and records it as an order history.

[0583] Specific examples

[0584] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[0585] Example prompts for generative AI models

[0586] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[0587] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[0588] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0589] System program processing flow

[0590] Step 1: Boot up the tablet and start the ordering application

[0591] The user picks up the tablet and launches the ordering application. Specifically, they turn on the tablet and tap the application icon. The message "Starting face recognition" appears on the screen. The input is the user's touch operation, and the output is the initial screen of the application.

[0592] Step 2: Obtaining user attribute information through facial recognition

[0593] The device captures the user's face using the built-in camera and analyzes the image using facial recognition software (e.g., OpenCV, Amazon Rekognition). The camera activates, identifies the user's face, and acquires image data. The input is the captured facial image, and the output is attribute information such as age and gender. Specific operations include extracting facial feature points and estimating age and gender.

[0594] Step 3: Send attribute information to the server and confirm with the user

[0595] The device sends the acquired attribute information in JSON format to the server. The server queries the database and checks whether the user is already registered based on the received attribute information. The input is the JSON data of the attribute information, and the output is the result of determining whether the user is new or existing. Specifically, the database search and face ID matching are performed.

[0596] Step 4: Get the order history of an existing user

[0597] If the user is an existing user, the server retrieves the user's past order history from the database. The input is the user ID, and the output is the user's past order history. The specific operation is to retrieve the order history from the database using an SQL query.

[0598] Step 5: Register a new user

[0599] When a user is new, the server registers their face information along with their attribute information in the database and generates a new unique user ID. The input is attribute information and face image data, and the output is the new user ID. Specifically, the new user data is inserted into the database and a unique ID is generated.

[0600] Step 6: Menu recommendations based on order history and attribute information

[0601] The server generates recommended menus based on the user's past order history and attribute information. It runs a recommendation algorithm to create a personalized menu list. The input is attribute information and order history, and the output is a list of recommended menus. Specific operations include collaborative filtering and content-based filtering.

[0602] Step 7: Displaying recommended menu items

[0603] The server generates a recommended menu list and sends it to the terminal, which then displays it on the ordering application. The input is the recommended menu list, and the output is the menu displayed on the ordering application. Specifically, the received data is parsed and displayed on the user interface.

[0604] Step 8: User confirms order

[0605] The user selects the menu item they want and presses the order confirmation button. The input is the user's selection and touch operation, and the output is the order data. The specific operation involves selecting a menu item and pressing the confirmation button.

[0606] Step 9: Send and store order information on the server

[0607] The terminal sends the selected order information in JSON format to the server. The server saves the order information in a database. The input is the order information, and the output is updating the database. Specifically, the order data is sent to the server and saved in the database.

[0608] Specific examples

[0609] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[0610] Example prompts for generative AI models

[0611] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[0612] (Application example 1)

[0613] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] Conventional personalized menu and product recommendation systems mainly use terminals or PCs, which poses challenges in improving user convenience and experience. Furthermore, there is a lack of means for providing recommendation information in real time within the store, requiring users to actively operate their terminals. Therefore, there is a need for a more intuitive and convenient way to provide personalized information to users.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0616] In this invention, the server includes a terminal means operable by a user, a means for identifying a user's face and analyzing attribute information using face recognition software, a server means for acquiring the user's past order history based on the analyzed attribute information, a server means for generating recommended menu items based on the acquired order history and attribute information, a means for displaying the generated recommended menu items on the terminal means, a means for transmitting and storing the user's order information to the server means, and an information display means for performing face recognition and displaying recommended menu items using smart glasses. This enables users to intuitively and conveniently receive personalized menu and product recommendations in a store.

[0617] "User-operable terminal means" refers to an electronic terminal device that can be operated by the user himself / herself to input and display information.

[0618] "Means of identifying a user's face and analyzing attribute information using facial recognition software" refers to a combination of software and hardware that uses a camera and facial recognition algorithm to capture a user's face, and then extracts and analyzes attribute information such as age and gender from the video data.

[0619] The "server means for acquiring the user's past order history based on the analyzed attribute information" is a server that has the function of searching a database based on the user's attribute information and acquiring the user's past order history.

[0620] The "server means for generating recommended menus based on the acquired order history and attribute information" refers to a server that has an algorithm for analyzing the acquired user order history and attribute information and automatically generating recommended menus based thereon.

[0621] The "means for displaying the generated recommended menu on the terminal means" refers to software and hardware that has the function of displaying the recommended menu generated by the server on the user's terminal.

[0622] The "means for transmitting user order information to server means and storing it" refers to communication means and storage means for transmitting the menu selected by the user to the server and storing that information in a database.

[0623] "Information display means for performing facial recognition and displaying recommended menus using smart glasses" refers to a device and system that performs facial recognition through smart glasses worn by the user and displays personalized recommended menus based on the results on the glasses' display.

[0624] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. A specific embodiment of the present invention will be described.

[0625] Facial Recognition and Attribute Analysis

[0626] 1. Tablet startup and face recognition

[0627] When a user wears the smart glasses and enters a store, the smart glasses automatically activate their camera and capture the user's face.

[0628] Facial recognition software built into the smart glasses analyzes this image and generates demographic information about the user, such as their age and gender.

[0629] 2. Processing of Attribute Information by the Server

[0630] The smart glasses transmit the acquired attribute information to the server.

[0631] The server checks whether the user is already registered based on the received attribute information.

[0632] In the case of an existing user, the server retrieves the user's past order history from the database.

[0633] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[0634] Generate menu recommendations

[0635] 3. View your past order history

[0636] For existing users, the server analyzes the user's preferences based on past order history.

[0637] For new users, the server generates a general recommendation menu based on their attribute information.

[0638] 4. Generate recommended menu

[0639] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[0640] The server sends the generated recommended menu list to the smart glasses.

[0641] Displaying recommended menu items and ordering

[0642] 5. Display of recommended menu items

[0643] The smart glasses display the recommended menu list received from the server on the glasses' display.

[0644] The recommended menu is highlighted to make it easier for users to make a selection.

[0645] 6. Placing orders and collecting feedback

[0646] The user operates the smart glasses to select the desired menu item and confirm the order.

[0647] The smart glasses transmit the selected order information to the server.

[0648] The server stores the received order information in a database and uses it to reflect future recommendations.

[0649] To implement this system, the following hardware and software are used:

[0650] Hardware

[0651] Smart glasses (e.g. Google Glass)

[0652] Built-in camera (integrated into smart glasses)

[0653] software

[0654] OpenCV library (face recognition)

[0655] REST API (sending and receiving face images and attribute information)

[0656] Smart Glasses SDK

[0657] Specific examples

[0658] For example, when a 30-year-old male user wears smart glasses and enters a store, the camera in the smart glasses captures his face and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. As a result, it is determined that he has previously preferred casual sneakers, and the latest sneaker collection is generated as a recommended menu. The recommended menu is displayed on the smart glasses, and the user selects new sneakers from the menu and confirms the purchase. The server saves the order information in a database and uses it as data for future recommendations.

[0659] Prompt Sentence Examples

[0660] "You are an AI that recommends products in a store. When a user enters the store, it recognizes their face and suggests products based on their past purchase history. For example, if the user has liked sneakers in the past, it will recommend a new sneaker collection."

[0661] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0662] Step 1: Capture the user's face and generate attribute information

[0663] When a user wearing smart glasses enters a store, the glasses automatically activate the camera. The camera captures the user's face, and facial recognition software analyzes the facial image to generate attribute information such as age and gender. The input is the camera's video data, and the output is the user's attribute information.

[0664] Step 2: Sending attribute information to the server

[0665] The smart glasses send the attribute information generated by the facial recognition software to the server. The input is the attribute information, and the output is the attribute information sent to the server. The smart glasses' communication module is used.

[0666] Step 3: Get past order history

[0667] The server checks whether the user is already registered based on the attribute information it receives. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers the user's face information along with the attribute information, generates a unique ID, and saves it in the database. The input is attribute information, and the output is the user's past order history or new user registration information.

[0668] Step 4: Generate a recommendation menu

[0669] The server runs an algorithm that generates menu recommendations based on the user's attribute information and past order history. The input is attribute information and order history, and the output is a list of recommended menu items. A generative AI model is used to analyze past data and make predictions.

[0670] Step 5: Send recommendations to your smart glasses

[0671] The server sends the generated recommended menu list to the smart glasses. The input is the recommended menu list, and the output is the recommended menu list sent to the smart glasses. A communication module is used.

[0672] Step 6: Displaying recommended menu items

[0673] The smart glasses display the recommended menu list received from the server on the glasses' display. The input is the recommended menu list, and the output is the menu displayed on the display. The smart glasses' display control module is used.

[0674] Step 7: Confirm your order

[0675] The user operates the smart glasses to select the desired menu and confirm the order. The input is the user's selection information, and the output is the selected order information. The interface of the smart glasses is used.

[0676] Step 8: Send and store order information on the server

[0677] The smart glasses send the selected order information to the server, which stores the information in a database. The input is the order information, and the output is the order information stored on the server. A communication module and a database management system are used.

[0678] At each step, specific data processing and calculations are performed, enabling users to receive intuitive, convenient, and personalized menu and product recommendations in-store, improving the user experience.

[0679] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0680] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. The following describes in detail embodiments of the present invention.

[0681] Facial Recognition and Attribute Analysis

[0682] 1. Tablet startup and face recognition

[0683] The user picks up the tablet and launches the ordering application.

[0684] The device uses a built-in camera to capture the user's face.

[0685] The device runs facial recognition software and analyzes the user's age, gender and other demographic information from the captured image.

[0686] 2. Emotion Recognition by Emotion Engine

[0687] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state (e.g., joy, surprise, sadness, anger, etc.).

[0688] 3. Processing of attribute information and emotion information by the server

[0689] The terminal transmits the acquired attribute information and emotion information to the server.

[0690] Based on the received information, the server searches the database to check whether the user is already registered.

[0691] In the case of an existing user, the server retrieves the user's past order history from the database.

[0692] For new users, the server registers their facial information along with their attribute and emotion information, generates a unique ID, and stores it in the database.

[0693] Generate menu recommendations

[0694] 4. View your past order history

[0695] For existing users, the server analyzes the user's preferences based on past order history.

[0696] For new users, the server generates a general recommendation menu based on attribute information and emotional information.

[0697] 5. Generate recommended menu

[0698] The server analyzes the user's attribute information, emotional information, and past order data and runs an algorithm to generate menu recommendations.

[0699] If the user is excited, the system will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy menus, making recommendations based on the user's emotional state.

[0700] The server sends the generated recommended menu list to the terminal.

[0701] Displaying recommended menu items and ordering

[0702] 6. Display recommended menu items

[0703] The terminal displays the recommended menu list received from the server on the ordering application.

[0704] The recommended menu is highlighted to make it easier for users to make a selection.

[0705] 7. Confirming orders and collecting feedback

[0706] The user selects a product from the displayed menu and presses the order confirmation button.

[0707] The terminal transmits the selected order information to the server.

[0708] The server stores the received order information in a database and uses it to reflect future recommendations.

[0709] Specific examples

[0710] For example, when a 30-year-old male user opens an ordering application on a tablet, the device uses a camera to capture his face and recognize his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu is displayed on the device, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[0711] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[0712] The processing flow will be explained below.

[0713] Step 1:

[0714] The user picks up the tablet and launches the ordering application.

[0715] Step 2:

[0716] The device uses its built-in camera to capture the user's face.

[0717] Step 3:

[0718] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[0719] Step 4:

[0720] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[0721] Step 5:

[0722] The terminal transmits the acquired attribute information and emotion information to the server.

[0723] Step 6:

[0724] Based on the attribute information and emotion information received by the server, the server searches the database to check whether the user is already registered.

[0725] Step 7:

[0726] If the server is an existing user, it retrieves the user's past order history from the database.

[0727] Step 8:

[0728] When the server receives a new user, it registers face information, attribute information, and emotion information, generates a unique ID, and stores it in the database.

[0729] Step 9:

[0730] The server analyzes the user's preferences based on the past order history, attribute information, and emotional information acquired by the server.

[0731] Step 10:

[0732] The server runs an algorithm to generate menu recommendations based on the user's attribute information, emotional information, and past order data.

[0733] Step 11:

[0734] The system adjusts the recommended menu according to the user's emotional state; for example, if the user is excited, it will recommend hearty dishes or sweets, and if the user is relaxed, it will recommend light meals or healthy options.

[0735] Step 12:

[0736] The server sends the generated recommended menu list to the terminal.

[0737] Step 13:

[0738] The terminal displays the recommended menu list received from the server on the ordering application.

[0739] Step 14:

[0740] The user selects a product from the displayed menu and presses the order confirmation button.

[0741] Step 15:

[0742] The terminal transmits the selected order information to the server.

[0743] Step 16:

[0744] The server stores the received order information in a database.

[0745] Step 17:

[0746] The server uses the stored order information as data to reflect in future recommendations.

[0747] Example 2

[0748] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0749] To provide personalized services at restaurants and other establishments, it is important to take into account not only the user's attribute information and past ordering history, but also their emotional state at the time. However, conventional systems have not been able to properly analyze the user's emotional state and reflect it in recommendations. This has made it difficult to provide even more personalized services.

[0750] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes terminal means operable by the user, means for identifying the user's face and analyzing attribute information using face recognition software, means for analyzing the user's emotional state using the analyzed attribute information and an emotion engine, means for acquiring the user's past order history based on the analyzed attribute information and emotion information, means for generating recommended menus based on the acquired order history, attribute information, and emotion information, means for displaying the generated recommended menus on the terminal means, and means for transmitting the user's order information to the server means and storing it. This makes it possible to analyze the user's emotional state and provide personalized recommended menus based on the analyzed emotional state.

[0751] "User-operable terminal means" refers to an electronic device that can be physically operated by a user, and that can execute a specific application to display information or input data.

[0752] "Facial recognition software" refers to a program or algorithm that analyzes facial images captured by a camera or other imaging device to detect, identify, and analyze certain facial features.

[0753] "Attribute information" refers to personal characteristic data such as a user's age and gender, and includes data obtained by facial recognition or other analytical means.

[0754] An "emotion engine" refers to an algorithm or program that determines a user's current emotional state from their facial expressions, voice, etc.

[0755] "Server means" refers to a computer system or network device for storing, processing, and transmitting data, and includes programs and services for managing and executing a series of operations.

[0756] "Analyzed attribute information and emotion information" refers to the user's personal characteristic data and emotional state data obtained and analyzed by the facial recognition software and emotion engine.

[0757] "Past order history" refers to data that records the details of orders previously placed by a user, and is used to analyze the user's preferences.

[0758] A "recommended menu" refers to a list of dishes and drinks suggested to the user based on the acquired data, and is generated by a specific algorithm.

[0759] "Order information" refers to information about the menu items selected by the user, and includes data for specifically recording the contents of the order.

[0760] The present invention is a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. This system utilizes facial recognition and emotion analysis of the user to recommend personalized menus. Specific embodiments for implementing the present invention are described below.

[0761] First, a terminal that can be operated by the user is used. Examples of such terminals include tablets and smartphones. This terminal has an ordering application installed and is activated by the user's operation.

[0762] The device uses the built-in camera to capture the user's face. This process uses a built-in camera module (e.g., a standard webcam). After the facial image is captured, facial recognition software (e.g., a common facial recognition API) is used to identify the user's face and analyze attribute information such as age and gender. This analysis is performed using algorithms built into the facial recognition software.

[0763] Next, the device inputs the facial expression data obtained from the facial image into an emotion engine. The emotion engine (e.g., emotion analysis API) analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). After obtaining the analysis results, the device transmits this attribute information and emotion information to the server.

[0764] The server checks whether the user is already registered in the database based on the received attribute and emotion information. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers new face, attribute, and emotion information, and generates and stores a unique ID. This registration process uses database manipulation software (e.g., a standard SQL database).

[0765] The server analyzes user preferences based on past data and newly acquired information. For existing users, it generates personalized menu recommendations based on past order history. For new users, it generates general menu recommendations based on attribute information and emotional information. This menu generation uses machine learning algorithms (e.g., a typical recommender system).

[0766] The generated recommended menu is sent from the server to the device and displayed on the ordering application. The user selects products based on this and confirms the order. The order information is sent from the device to the server, which stores it in a database and reflects it in the next recommendation.

[0767] As a concrete example, consider the case where a 30-year-old male user opens an ordering application on a tablet. The device uses a camera to capture his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state of "happiness" from his facial expression. The server confirms that the user's facial information has already been registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta. The server generates recommended menu items such as specialty pizza and carbonara pasta, and recommends more of certain menu items considering his state of happiness. The recommended menu items are displayed on the device, and the user makes a selection and confirms the order. The server stores this order information in a database and uses it for future recommendations.

[0768] The above is a specific embodiment of the present invention. This system allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide more personalized services that take emotional states into consideration.

[0769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0770] Step 1:

[0771] The user turns on the tablet and launches the dedicated ordering application.

[0772] Input: User actions

[0773] Specific action: Tap the ordering application icon on the tablet's home screen.

[0774] Output: The ordering application is launched.

[0775] Step 2:

[0776] The device activates its built-in camera to capture the user's face.

[0777] Input: A face image captured from the camera

[0778] Specific operation: The tablet's front camera takes a picture of the user's face, and facial image data is generated.

[0779] Output: Face image data

[0780] Step 3:

[0781] The device uses facial recognition software to analyze the user's age, gender, and other attribute information from the captured facial image.

[0782] Input: Facial image data

[0783] Specific operation: Calls a facial recognition API and runs an image processing algorithm to estimate age and gender.

[0784] Output: Attribute information (age, gender, etc.)

[0785] Step 4:

[0786] The terminal inputs the acquired facial image data and attribute information into an emotion engine to analyze the user's emotional state.

[0787] Input: Facial image data, attribute information

[0788] Specific operation: Calls the emotion analysis API and quantifies the emotional state (e.g., joy, surprise, sadness, anger, etc.) from the facial image.

[0789] Output: Emotional information

[0790] Step 5:

[0791] The terminal transmits the analyzed attribute information and emotion information to the server.

[0792] Input: attribute information, emotion information

[0793] Specific operation: Data is sent to the server using the HTTPS protocol.

[0794] Output: Data sent to the server

[0795] Step 6:

[0796] Based on the information received by the server, it checks whether the user is registered in the database.

[0797] Input: attribute information, emotion information

[0798] Specific behavior: Runs an SQL database query to check the user's enrollment status based on facial information.

[0799] Output: Information on whether the user is an existing user or a new user

[0800] Step 7:

[0801] If the user is an existing user, the server retrieves the user's past order history from the database, and if the user is a new user, the server registers new face information, attribute information, and emotion information, and generates and saves a unique ID.

[0802] Input: User registration status, attribute information, emotional information

[0803] Specific operation: For existing users, retrieve past order data based on the user ID, and for new users, create a new record and insert the data.

[0804] Output: If an existing user, the retrieved past order history. If a new user, the registration completion ID.

[0805] Step 8:

[0806] Based on the data acquired by the server and newly analyzed information, the server analyzes the user's preferences and generates recommended menus.

[0807] Input: past order history, attribute information, emotional information

[0808] What it does: It uses machine learning algorithms to analyze and generate a menu tailored to the user's preferences.

[0809] Output: Recommended menu list

[0810] Step 9:

[0811] The server sends the generated recommended menu list to the terminal.

[0812] Input: Recommended menu list

[0813] Specific operation: Sends data to the terminal using the HTTPS protocol.

[0814] Output: Recommended menu list sent to the device

[0815] Step 10:

[0816] The terminal displays the recommended menu list received from the server on the ordering application.

[0817] Input: Recommended menu list

[0818] What it does: Displays the list on the ordering application screen so that the user can visually confirm it.

[0819] Output: Screen showing recommended menu

[0820] Step 11:

[0821] The user selects a product from the displayed menu and presses the order confirmation button.

[0822] Input: Select from the recommended menu

[0823] Specific actions: Tap to select a menu item and tap the order confirmation button.

[0824] Output: Selected menu and order confirmation information

[0825] Step 12:

[0826] The terminal transmits the selected order information to the server.

[0827] Input: Selected menu and order confirmation information

[0828] Specific operation: Order data is sent to the server using the HTTPS protocol.

[0829] Output: Order information sent to the server

[0830] Step 13:

[0831] The server stores the received order information in a database and uses it to make future recommendations.

[0832] Input: Order Information

[0833] Specific operation: Inserts order information into the database and saves the new data.

[0834] Output: Order information stored in the database

[0835] This concludes the overall processing flow of the system, which provides a personalized menu based on the user's emotional state and preferences.

[0836] (Application example 2)

[0837] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0838] Conventional personalized menu recommendation systems generally make static recommendations based on a user's past order history and attribute information, and are unable to consider the customer's emotional state in real time. This makes it difficult to provide the optimal menu based on the customer's current mood, and this has led to the problem of not being able to sufficiently increase customer satisfaction.

[0839] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user attribute information based on the analyzed attribute information and emotional state, emotion analysis means for analyzing the user's emotional state in real time, and means for acquiring the user's past order history based on the analyzed attribute information and emotional state. This makes it possible to provide the user with a personalized menu in real time that corresponds to their current emotional state.

[0840] "User-operable terminal means" refers to a terminal such as a smartphone, tablet, or smart glasses that allows a user to place an order, check the menu, etc.

[0841] "Means of identifying a user's face using facial recognition software and analyzing attribute information" refers to software that analyzes a user's attributes, such as age and gender, based on an image of the user's face taken with a camera.

[0842] "Means for analyzing user attribute information based on analyzed attribute information and emotional state" refers to a system for analyzing detailed attributes and preferences of a user using the acquired attribute information and emotional state.

[0843] "Emotion analysis means for analyzing the user's emotional state in real time" refers to a system for analyzing emotions such as joy, surprise, sadness, and anger based on the user's facial expressions.

[0844] "Server means for acquiring a user's past order history based on the analyzed attribute information and emotional state" refers to a system in which a server acquires a user's past order data based on the user's attribute information and emotional information.

[0845] "Server means for generating recommended menus based on acquired order history, attribute information, and emotional state" refers to a system in which a server generates a menu suitable for a user based on past order history, attribute information, and emotional state.

[0846] "Means for displaying the generated recommended menu on a terminal means and on smart glasses or other smart devices" refers to a system that displays the recommended menu generated by the server on a display device (such as smart glasses) in real time.

[0847] "Means for transmitting and storing user order information to server means" refers to a system that transmits order information selected by a user to a server and stores it for use in future recommendations.

[0848] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. Specifically, the system is configured and operates as follows.

[0849] User Interfaces and Terminals

[0850] Users use devices such as smart glasses, smartphones, or tablets, each with a built-in camera and capable of running facial recognition software, and an ordering application installed, which provides an interface for customers to select menu items and enter their order information.

[0851] Facial Recognition and Attribute Analysis

[0852] When a user puts on the smart glasses, the built-in camera captures the user's face, which is then analyzed by facial recognition software to determine the user's age, gender, and other demographic information, which is then analyzed in real time and sent to a server.

[0853] Emotional state analysis

[0854] Facial expression data obtained from facial images is input into the emotion engine, which analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). This makes it possible to understand the user's current mood and emotions. The analysis results are also sent to the server.

[0855] Server Processing

[0856] The server receives attribute information and emotional state from the device, and for existing users, retrieves past order history from the database. For new users, the server registers facial information along with attribute information and emotional information, generates a unique ID, and stores it in the database.

[0857] Generate menu recommendations

[0858] The server runs an algorithm that recommends the most suitable menu for the user based on the analyzed attribute information, emotional state, and past order history. If the user is excited, the server will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy options.

[0859] Displaying recommended menu items and ordering

[0860] The generated recommended menu list is displayed in real time on smart glasses or other smart devices. The user selects items from the displayed menu and confirms the order. The device then sends the selected order information to the server, which stores this information in a database and uses it to reflect future recommendations.

[0861] Specific examples

[0862] For example, suppose a 30-year-old male user is wearing smart glasses. The smart glasses' camera captures his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu items are displayed on the smart glasses, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[0863] Prompt Sentence Examples

[0864] When the customer looks at the menu displayed in the smart glasses, menu recommendations tailored to his emotions are displayed.

[0865] "We recommend the following menu items: Specialty Pizza, Carbonara Pasta"

[0866] Simply move your hand to select your order and the glasses will navigate you to the menu item's location in the store.

[0867] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[0868] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0869] Step 1:

[0870] The user puts on the smart glasses and launches the ordering application.

[0871] Input: Smart glasses activation signal.

[0872] Output: The camera starts capturing face images.

[0873] Specific operation: The smart glasses' built-in camera captures an image of the user's face, and the image data is sent to the ordering application.

[0874] Step 2:

[0875] The device uses facial recognition software to analyze the user's face.

[0876] Input: A captured face image.

[0877] Output: User's demographic information such as age and gender.

[0878] Specific operation: Facial recognition software analyzes facial images, obtains attribute information such as age and gender, and temporarily stores this information in the device database.

[0879] Step 3:

[0880] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[0881] Input: A captured face image.

[0882] Output: The user's emotional state (e.g., happy, surprised, sad, angry, etc.).

[0883] Specific operation: The emotion engine analyzes the facial expression data, identifies the emotional state, and temporarily stores that information in the device database.

[0884] Step 4:

[0885] The terminal transmits the acquired attribute information and emotion information to the server.

[0886] Input: attribute information, emotion information.

[0887] Output: Attribute information and emotion information sent to the server.

[0888] Specific operation: The terminal transmits the analyzed attribute information and emotion information to the server as a data packet.

[0889] Step 5:

[0890] The server acquires the user's past order history based on the received information.

[0891] Input: attribute information, emotion information.

[0892] Output: Past order history or register as a new user.

[0893] Specific operation: Using the attribute information and emotion information received by the server, the server searches the database for past order history for existing users, and for new users, it registers them along with their facial information and generates a unique ID.

[0894] Step 6:

[0895] The server generates a recommended menu based on the analyzed information.

[0896] Input: Past order history, attribute information, emotional state.

[0897] Output: Recommended menu list.

[0898] Specific operation: The server uses a generative AI model to analyze past order history, attribute information, and current emotional state, and generates the most suitable recommended menu for the user.

[0899] Step 7:

[0900] The server sends the generated recommended menu list to the terminal.

[0901] Input: Recommended menu list.

[0902] Output: A list of recommended menus sent to the device.

[0903] Specific operation: The recommended menu list generated by the server is sent to the terminal as a data packet.

[0904] Step 8:

[0905] The device will display a list of recommended menu items and display them on the smart glasses.

[0906] Input: Recommended menu list.

[0907] Output: Recommendation menu displayed on smart glasses.

[0908] Specific operation: The recommended menu list received by the device is displayed in real time on the smart glasses display.

[0909] Step 9:

[0910] The user selects a product from the recommended menu and confirms the order.

[0911] Input: Recommended menu list.

[0912] Output: Selected order information.

[0913] Specific operation: The user selects a product from the menu displayed on the smart glasses and presses the order confirmation button.

[0914] Step 10:

[0915] The terminal transmits the selected order information to the server, which stores the information in a database.

[0916] Enter: Selected order information.

[0917] Output: Order information stored on the server.

[0918] Specific operation: The terminal sends the user's selected order information to the server, which stores the information in a database so that it can be reflected in future recommendations.

[0919] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0920] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0921] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0922] [Third embodiment]

[0923] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0924] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0925] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0926] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0927] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0928] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0929] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0930] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0931] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0932] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0933] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0934] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0935] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. The following describes in detail an embodiment of the present invention.

[0936] Facial Recognition and Attribute Analysis

[0937] 1. Tablet startup and face recognition

[0938] The user picks up the tablet and launches the ordering application.

[0939] The device uses a built-in camera to capture the user's face.

[0940] The device runs facial recognition software that analyzes the captured image to generate demographic information about the user, such as their age and gender.

[0941] 2. Processing of Attribute Information by the Server

[0942] The terminal transmits the acquired attribute information to the server.

[0943] The server checks whether the user is already registered based on the received attribute information.

[0944] In the case of an existing user, the server retrieves the user's past order history from the database.

[0945] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[0946] Generate menu recommendations

[0947] 3. View your past order history

[0948] For existing users, the server analyzes the user's preferences based on past order history.

[0949] For new users, the server generates a general recommendation menu based on their attribute information.

[0950] 4. Generate recommended menu

[0951] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[0952] The server sends the generated recommended menu list to the terminal.

[0953] Displaying recommended menu items and ordering

[0954] 5. Display of recommended menu items

[0955] The terminal displays the recommended menu list received from the server on the ordering application.

[0956] The recommended menu is highlighted to make it easier for users to make a selection.

[0957] 6. Placing orders and collecting feedback

[0958] The user selects the desired menu item and presses the order confirmation button.

[0959] The terminal transmits the selected order information to the server.

[0960] The server stores the received order information in a database and uses it to reflect future recommendations.

[0961] Specific examples

[0962] For example, when a 30-year-old male user opens an ordering application on a tablet, the device captures his face using the camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server then stores the order information in a database and uses it as data for future recommendations.

[0963] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[0964] The processing flow will be explained below.

[0965] Step 1:

[0966] The user picks up the tablet and launches the ordering application.

[0967] Step 2:

[0968] The device uses its built-in camera to capture the user's face.

[0969] Step 3:

[0970] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[0971] Step 4:

[0972] The terminal transmits the acquired attribute information to the server.

[0973] Step 5:

[0974] Based on the attribute information received by the server, the server searches the database to check whether the user is already registered.

[0975] Step 6:

[0976] If the server is an existing user, it retrieves the user's past order history from the database.

[0977] Step 7:

[0978] When the server receives a new user, it registers face information and attribute information, generates a unique ID, and stores it in the database.

[0979] Step 8:

[0980] The server analyzes the user's preferences based on the past order history and attribute information acquired.

[0981] Step 9:

[0982] The server runs an algorithm to generate recommended menu items based on the user's attribute information and past order data.

[0983] Step 10:

[0984] The server sends the generated recommended menu list to the terminal.

[0985] Step 11:

[0986] The terminal displays the recommended menu list received from the server on the ordering application.

[0987] Step 12:

[0988] The user selects a product from the displayed menu and presses the order confirmation button.

[0989] Step 13:

[0990] The terminal transmits the selected order information to the server.

[0991] Step 14:

[0992] The server stores the received order information in a database.

[0993] Step 15:

[0994] The server uses the stored order information as data to reflect in future recommendations.

[0995] Example 1

[0996] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0997] Existing systems have the problem that when providing personalized recommendations to users, the users themselves must perform many manual operations, making it difficult to automatically provide menus that meet the user's preferences. Another issue is that they cannot efficiently utilize user attribute information or past order history, making it difficult to provide highly accurate recommendations.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0999] In this invention, the server includes an input device that can be operated by a user, a terminal that uses facial recognition software to identify the user's face and analyze their attribute information, a server that acquires the user's past order history based on the analyzed attribute information, a server that generates recommended menu items based on the acquired order history and attribute information, a display that displays the generated recommended menu items on the terminal, and a recording that transmits the user's order information to the server and stores it. This allows the user to receive menu recommendations that suit their preferences with minimal manual operation. Furthermore, highly accurate recommendations can be realized by efficiently utilizing attribute information and past order history.

[1000] The "user-operable input device means" is a device that allows a user to input information into the system by operating it himself.

[1001] "Facial recognition software" is a program that analyzes a user's facial image to identify the individual and extract attribute information.

[1002] "Terminal means" refers to a device that executes face recognition software, captures the user's face, and generates attribute information.

[1003] "Server means" refers to a central computer that stores and manages user attribute information and order history, and performs advanced calculation processing.

[1004] A "display means" is a device for visually presenting information to a user, or a partial function thereof.

[1005] "Storage means" refers to a device or function for storing data or information generated within the system.

[1006] "Information analysis means" refers to a program or function for analyzing collected data and extracting useful attribute information.

[1007] An "algorithm" is a set of calculations or procedures for solving a particular problem.

[1008] The present invention is directed to a system including a server and an input device operable by a user, which utilizes facial recognition and order history to recommend personalized menus. Hereinafter, embodiments of the present invention will be described in detail.

[1009] Facial Recognition and Attribute Analysis

[1010] 1. Tablet startup and face recognition

[1011] The user picks up the tablet and launches the ordering application. After turning on the tablet and tapping the application icon to launch the application, the message "Starting face recognition" is displayed.

[1012] The device captures the user's face using the built-in camera. When the camera is activated, it automatically recognizes the position of the user's face and captures it. This captured image is then analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition).

[1013] The device runs facial recognition software, which analyzes the captured image to generate demographic information about the user, such as their age and gender. Image analysis extracts the user's facial features and runs a model to estimate their age and gender. This process generates demographic information (e.g., "30-year-old male").

[1014] Server-based attribute processing

[1015] 2. Processing of Attribute Information by the Server

[1016] The device sends the acquired attribute information to the server. The generated attribute information is sent to the server in JSON format using the HTTP POST method.

[1017] The server queries a database (e.g., MySQL or PostgreSQL) based on the received attribute information to see if the user is already registered, and verifies the user's existence based on the face ID and attribute information.

[1018] If the user is an existing user, the server retrieves the user's past order history from the database. It uses an SQL query to retrieve the past order history associated with the user ID and makes it available within the application.

[1019] For a new user, the server registers the face information along with the attribute information, generates a new unique ID, and saves it in the database. The server registers the user's attribute information and the captured face image in the database and generates a new unique user ID.

[1020] Generate menu recommendations

[1021] 3. View your past order history

[1022] The server analyzes the user's preferences based on the user's past order history. It uses machine learning models (e.g., K-Means clustering and logistic regression) to analyze the user's past order history and identify the user's preferences.

[1023] 4. Generate recommended menu

[1024] The server runs a recommendation algorithm (e.g., collaborative filtering or content-based filtering) based on the user's attribute information and past order history. It combines the facial recognition results, attribute information, and past order history to generate a personalized menu using a pre-defined algorithm.

[1025] The server sends the generated recommended menu list to the device. It generates the recommended menu list in JSON format and returns it as an HTTP response.

[1026] Displaying recommended menu items and ordering

[1027] 5. Display of recommended menu items

[1028] The terminal displays the recommended menu list received from the server on the ordering application. The terminal parses the received data and displays it appropriately in the user interface. Recommended menu items are highlighted with eye-catching colors and fonts.

[1029] 6. Placing orders and collecting feedback

[1030] The user selects the desired menu item and presses the order confirmation button. By touching the screen, the user selects the menu item and taps the confirmation button. The terminal sends the selected order information to the server.

[1031] The terminal sends the selected order information to the server. Information such as the selected menu ID is sent in JSON format to the server.

[1032] The server stores the received order information in a database and uses it for future recommendations. The server also stores the transmitted data in a database and records it as an order history.

[1033] Specific examples

[1034] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[1035] Example prompts for generative AI models

[1036] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[1037] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[1038] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1039] System program processing flow

[1040] Step 1: Boot up the tablet and start the ordering application

[1041] The user picks up the tablet and launches the ordering application. Specifically, they turn on the tablet and tap the application icon. The message "Starting face recognition" appears on the screen. The input is the user's touch operation, and the output is the initial screen of the application.

[1042] Step 2: Obtaining user attribute information through facial recognition

[1043] The device captures the user's face using the built-in camera and analyzes the image using facial recognition software (e.g., OpenCV, Amazon Rekognition). The camera activates, identifies the user's face, and acquires image data. The input is the captured facial image, and the output is attribute information such as age and gender. Specific operations include extracting facial feature points and estimating age and gender.

[1044] Step 3: Send attribute information to the server and confirm with the user

[1045] The device sends the acquired attribute information in JSON format to the server. The server queries the database and checks whether the user is already registered based on the received attribute information. The input is the JSON data of the attribute information, and the output is the result of determining whether the user is new or existing. Specifically, the database search and face ID matching are performed.

[1046] Step 4: Get the order history of an existing user

[1047] If the user is an existing user, the server retrieves the user's past order history from the database. The input is the user ID, and the output is the user's past order history. The specific operation is to retrieve the order history from the database using an SQL query.

[1048] Step 5: Register a new user

[1049] When a user is new, the server registers their face information along with their attribute information in the database and generates a new unique user ID. The input is attribute information and face image data, and the output is the new user ID. Specifically, the new user data is inserted into the database and a unique ID is generated.

[1050] Step 6: Menu recommendations based on order history and attribute information

[1051] The server generates recommended menus based on the user's past order history and attribute information. It runs a recommendation algorithm to create a personalized menu list. The input is attribute information and order history, and the output is a list of recommended menus. Specific operations include collaborative filtering and content-based filtering.

[1052] Step 7: Displaying recommended menu items

[1053] The server generates a recommended menu list and sends it to the terminal, which then displays it on the ordering application. The input is the recommended menu list, and the output is the menu displayed on the ordering application. Specifically, the received data is parsed and displayed on the user interface.

[1054] Step 8: User confirms order

[1055] The user selects the menu item they want and presses the order confirmation button. The input is the user's selection and touch operation, and the output is the order data. The specific operation involves selecting a menu item and pressing the confirmation button.

[1056] Step 9: Send and store order information on the server

[1057] The terminal sends the selected order information in JSON format to the server. The server saves the order information in a database. The input is the order information, and the output is updating the database. Specifically, the order data is sent to the server and saved in the database.

[1058] Specific examples

[1059] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[1060] Example prompts for generative AI models

[1061] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[1062] (Application example 1)

[1063] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1064] Conventional personalized menu and product recommendation systems mainly use terminals or PCs, which poses challenges in improving user convenience and experience. Furthermore, there is a lack of means for providing recommendation information in real time within the store, requiring users to actively operate their terminals. Therefore, there is a need for a more intuitive and convenient way to provide personalized information to users.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1066] In this invention, the server includes a terminal means operable by a user, a means for identifying a user's face and analyzing attribute information using face recognition software, a server means for acquiring the user's past order history based on the analyzed attribute information, a server means for generating recommended menu items based on the acquired order history and attribute information, a means for displaying the generated recommended menu items on the terminal means, a means for transmitting and storing the user's order information to the server means, and an information display means for performing face recognition and displaying recommended menu items using smart glasses. This enables users to intuitively and conveniently receive personalized menu and product recommendations in a store.

[1067] "User-operable terminal means" refers to an electronic terminal device that can be operated by the user himself / herself to input and display information.

[1068] "Means of identifying a user's face and analyzing attribute information using facial recognition software" refers to a combination of software and hardware that uses a camera and facial recognition algorithm to capture a user's face, and then extracts and analyzes attribute information such as age and gender from the video data.

[1069] The "server means for acquiring the user's past order history based on the analyzed attribute information" is a server that has the function of searching a database based on the user's attribute information and acquiring the user's past order history.

[1070] The "server means for generating recommended menus based on the acquired order history and attribute information" refers to a server that has an algorithm for analyzing the acquired user order history and attribute information and automatically generating recommended menus based thereon.

[1071] The "means for displaying the generated recommended menu on the terminal means" refers to software and hardware that has the function of displaying the recommended menu generated by the server on the user's terminal.

[1072] The "means for transmitting user order information to server means and storing it" refers to communication means and storage means for transmitting the menu selected by the user to the server and storing that information in a database.

[1073] "Information display means for performing facial recognition and displaying recommended menus using smart glasses" refers to a device and system that performs facial recognition through smart glasses worn by the user and displays personalized recommended menus based on the results on the glasses' display.

[1074] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. A specific embodiment of the present invention will be described.

[1075] Facial Recognition and Attribute Analysis

[1076] 1. Tablet startup and face recognition

[1077] When a user wears the smart glasses and enters a store, the smart glasses automatically activate their camera and capture the user's face.

[1078] Facial recognition software built into the smart glasses analyzes this image and generates demographic information about the user, such as their age and gender.

[1079] 2. Processing of Attribute Information by the Server

[1080] The smart glasses transmit the acquired attribute information to the server.

[1081] The server checks whether the user is already registered based on the received attribute information.

[1082] In the case of an existing user, the server retrieves the user's past order history from the database.

[1083] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[1084] Generate menu recommendations

[1085] 3. View your past order history

[1086] For existing users, the server analyzes the user's preferences based on past order history.

[1087] For new users, the server generates a general recommendation menu based on their attribute information.

[1088] 4. Generate recommended menu

[1089] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[1090] The server sends the generated recommended menu list to the smart glasses.

[1091] Displaying recommended menu items and ordering

[1092] 5. Display of recommended menu items

[1093] The smart glasses display the recommended menu list received from the server on the glasses' display.

[1094] The recommended menu is highlighted to make it easier for users to make a selection.

[1095] 6. Placing orders and collecting feedback

[1096] The user operates the smart glasses to select the desired menu item and confirm the order.

[1097] The smart glasses transmit the selected order information to the server.

[1098] The server stores the received order information in a database and uses it to reflect future recommendations.

[1099] To implement this system, the following hardware and software are used:

[1100] Hardware

[1101] Smart glasses (e.g. Google Glass)

[1102] Built-in camera (integrated into smart glasses)

[1103] software

[1104] OpenCV library (face recognition)

[1105] REST API (sending and receiving face images and attribute information)

[1106] Smart Glasses SDK

[1107] Specific examples

[1108] For example, when a 30-year-old male user wears smart glasses and enters a store, the camera in the smart glasses captures his face and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. As a result, it is determined that he has previously preferred casual sneakers, and the latest sneaker collection is generated as a recommended menu. The recommended menu is displayed on the smart glasses, and the user selects new sneakers from the menu and confirms the purchase. The server saves the order information in a database and uses it as data for future recommendations.

[1109] Prompt Sentence Examples

[1110] "You are an AI that recommends products in a store. When a user enters the store, it recognizes their face and suggests products based on their past purchase history. For example, if the user has liked sneakers in the past, it will recommend a new sneaker collection."

[1111] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1112] Step 1: Capture the user's face and generate attribute information

[1113] When a user wearing smart glasses enters a store, the glasses automatically activate the camera. The camera captures the user's face, and facial recognition software analyzes the facial image to generate attribute information such as age and gender. The input is the camera's video data, and the output is the user's attribute information.

[1114] Step 2: Sending attribute information to the server

[1115] The smart glasses send the attribute information generated by the facial recognition software to the server. The input is the attribute information, and the output is the attribute information sent to the server. The smart glasses' communication module is used.

[1116] Step 3: Get past order history

[1117] The server checks whether the user is already registered based on the attribute information it receives. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers the user's face information along with the attribute information, generates a unique ID, and saves it in the database. The input is attribute information, and the output is the user's past order history or new user registration information.

[1118] Step 4: Generate a recommendation menu

[1119] The server runs an algorithm that generates menu recommendations based on the user's attribute information and past order history. The input is attribute information and order history, and the output is a list of recommended menu items. A generative AI model is used to analyze past data and make predictions.

[1120] Step 5: Send recommendations to your smart glasses

[1121] The server sends the generated recommended menu list to the smart glasses. The input is the recommended menu list, and the output is the recommended menu list sent to the smart glasses. A communication module is used.

[1122] Step 6: Displaying recommended menu items

[1123] The smart glasses display the recommended menu list received from the server on the glasses' display. The input is the recommended menu list, and the output is the menu displayed on the display. The smart glasses' display control module is used.

[1124] Step 7: Confirm your order

[1125] The user operates the smart glasses to select the desired menu and confirm the order. The input is the user's selection information, and the output is the selected order information. The interface of the smart glasses is used.

[1126] Step 8: Send and store order information on the server

[1127] The smart glasses send the selected order information to the server, which stores the information in a database. The input is the order information, and the output is the order information stored on the server. A communication module and a database management system are used.

[1128] At each step, specific data processing and calculations are performed, enabling users to receive intuitive, convenient, and personalized menu and product recommendations in-store, improving the user experience.

[1129] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1130] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. The following describes in detail embodiments of the present invention.

[1131] Facial Recognition and Attribute Analysis

[1132] 1. Tablet startup and face recognition

[1133] The user picks up the tablet and launches the ordering application.

[1134] The device uses a built-in camera to capture the user's face.

[1135] The device runs facial recognition software and analyzes the user's age, gender and other demographic information from the captured image.

[1136] 2. Emotion Recognition by Emotion Engine

[1137] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state (e.g., joy, surprise, sadness, anger, etc.).

[1138] 3. Processing of attribute information and emotion information by the server

[1139] The terminal transmits the acquired attribute information and emotion information to the server.

[1140] Based on the received information, the server searches the database to check whether the user is already registered.

[1141] In the case of an existing user, the server retrieves the user's past order history from the database.

[1142] For new users, the server registers their facial information along with their attribute and emotion information, generates a unique ID, and stores it in the database.

[1143] Generate menu recommendations

[1144] 4. View your past order history

[1145] For existing users, the server analyzes the user's preferences based on past order history.

[1146] For new users, the server generates a general recommendation menu based on attribute information and emotional information.

[1147] 5. Generate recommended menu

[1148] The server analyzes the user's attribute information, emotional information, and past order data and runs an algorithm to generate menu recommendations.

[1149] If the user is excited, the system will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy menus, making recommendations based on the user's emotional state.

[1150] The server sends the generated recommended menu list to the terminal.

[1151] Displaying recommended menu items and ordering

[1152] 6. Display recommended menu items

[1153] The terminal displays the recommended menu list received from the server on the ordering application.

[1154] The recommended menu is highlighted to make it easier for users to make a selection.

[1155] 7. Confirming orders and collecting feedback

[1156] The user selects a product from the displayed menu and presses the order confirmation button.

[1157] The terminal transmits the selected order information to the server.

[1158] The server stores the received order information in a database and uses it to reflect future recommendations.

[1159] Specific examples

[1160] For example, when a 30-year-old male user opens an ordering application on a tablet, the device uses a camera to capture his face and recognize his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu is displayed on the device, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[1161] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[1162] The processing flow will be explained below.

[1163] Step 1:

[1164] The user picks up the tablet and launches the ordering application.

[1165] Step 2:

[1166] The device uses its built-in camera to capture the user's face.

[1167] Step 3:

[1168] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[1169] Step 4:

[1170] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[1171] Step 5:

[1172] The terminal transmits the acquired attribute information and emotion information to the server.

[1173] Step 6:

[1174] Based on the attribute information and emotion information received by the server, the server searches the database to check whether the user is already registered.

[1175] Step 7:

[1176] If the server is an existing user, it retrieves the user's past order history from the database.

[1177] Step 8:

[1178] When the server receives a new user, it registers face information, attribute information, and emotion information, generates a unique ID, and stores it in the database.

[1179] Step 9:

[1180] The server analyzes the user's preferences based on the past order history, attribute information, and emotional information acquired by the server.

[1181] Step 10:

[1182] The server runs an algorithm to generate menu recommendations based on the user's attribute information, emotional information, and past order data.

[1183] Step 11:

[1184] The system adjusts the recommended menu according to the user's emotional state; for example, if the user is excited, it will recommend hearty dishes or sweets, and if the user is relaxed, it will recommend light meals or healthy options.

[1185] Step 12:

[1186] The server sends the generated recommended menu list to the terminal.

[1187] Step 13:

[1188] The terminal displays the recommended menu list received from the server on the ordering application.

[1189] Step 14:

[1190] The user selects a product from the displayed menu and presses the order confirmation button.

[1191] Step 15:

[1192] The terminal transmits the selected order information to the server.

[1193] Step 16:

[1194] The server stores the received order information in a database.

[1195] Step 17:

[1196] The server uses the stored order information as data to reflect in future recommendations.

[1197] Example 2

[1198] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1199] To provide personalized services at restaurants and other establishments, it is important to take into account not only the user's attribute information and past ordering history, but also their emotional state at the time. However, conventional systems have not been able to properly analyze the user's emotional state and reflect it in recommendations. This has made it difficult to provide even more personalized services.

[1200] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes terminal means operable by the user, means for identifying the user's face and analyzing attribute information using face recognition software, means for analyzing the user's emotional state using the analyzed attribute information and an emotion engine, means for acquiring the user's past order history based on the analyzed attribute information and emotion information, means for generating recommended menus based on the acquired order history, attribute information, and emotion information, means for displaying the generated recommended menus on the terminal means, and means for transmitting the user's order information to the server means and storing it. This makes it possible to analyze the user's emotional state and provide personalized recommended menus based on the analyzed emotional state.

[1201] "User-operable terminal means" refers to an electronic device that can be physically operated by a user, and that can execute a specific application to display information or input data.

[1202] "Facial recognition software" refers to a program or algorithm that analyzes facial images captured by a camera or other imaging device to detect, identify, and analyze certain facial features.

[1203] "Attribute information" refers to personal characteristic data such as a user's age and gender, and includes data obtained by facial recognition or other analytical means.

[1204] An "emotion engine" refers to an algorithm or program that determines a user's current emotional state from their facial expressions, voice, etc.

[1205] "Server means" refers to a computer system or network device for storing, processing, and transmitting data, and includes programs and services for managing and executing a series of operations.

[1206] "Analyzed attribute information and emotion information" refers to the user's personal characteristic data and emotional state data obtained and analyzed by the facial recognition software and emotion engine.

[1207] "Past order history" refers to data that records the details of orders previously placed by a user, and is used to analyze the user's preferences.

[1208] A "recommended menu" refers to a list of dishes and drinks suggested to the user based on the acquired data, and is generated by a specific algorithm.

[1209] "Order information" refers to information about the menu items selected by the user, and includes data for specifically recording the contents of the order.

[1210] The present invention is a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. This system utilizes facial recognition and emotion analysis of the user to recommend personalized menus. Specific embodiments for implementing the present invention are described below.

[1211] First, a terminal that can be operated by the user is used. Examples of such terminals include tablets and smartphones. This terminal has an ordering application installed and is activated by the user's operation.

[1212] The device uses the built-in camera to capture the user's face. This process uses a built-in camera module (e.g., a standard webcam). After the facial image is captured, facial recognition software (e.g., a common facial recognition API) is used to identify the user's face and analyze attribute information such as age and gender. This analysis is performed using algorithms built into the facial recognition software.

[1213] Next, the device inputs the facial expression data obtained from the facial image into an emotion engine. The emotion engine (e.g., emotion analysis API) analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). After obtaining the analysis results, the device transmits this attribute information and emotion information to the server.

[1214] The server checks whether the user is already registered in the database based on the received attribute and emotion information. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers new face, attribute, and emotion information, and generates and stores a unique ID. This registration process uses database manipulation software (e.g., a standard SQL database).

[1215] The server analyzes user preferences based on past data and newly acquired information. For existing users, it generates personalized menu recommendations based on past order history. For new users, it generates general menu recommendations based on attribute information and emotional information. This menu generation uses machine learning algorithms (e.g., a typical recommender system).

[1216] The generated recommended menu is sent from the server to the device and displayed on the ordering application. The user selects products based on this and confirms the order. The order information is sent from the device to the server, which stores it in a database and reflects it in the next recommendation.

[1217] As a concrete example, consider the case where a 30-year-old male user opens an ordering application on a tablet. The device uses a camera to capture his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state of "happiness" from his facial expression. The server confirms that the user's facial information has already been registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta. The server generates recommended menu items such as specialty pizza and carbonara pasta, and recommends more of certain menu items considering his state of happiness. The recommended menu items are displayed on the device, and the user makes a selection and confirms the order. The server stores this order information in a database and uses it for future recommendations.

[1218] The above is a specific embodiment of the present invention. This system allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide more personalized services that take emotional states into consideration.

[1219] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1220] Step 1:

[1221] The user turns on the tablet and launches the dedicated ordering application.

[1222] Input: User actions

[1223] Specific action: Tap the ordering application icon on the tablet's home screen.

[1224] Output: The ordering application is launched.

[1225] Step 2:

[1226] The device activates its built-in camera to capture the user's face.

[1227] Input: A face image captured from the camera

[1228] Specific operation: The tablet's front camera takes a picture of the user's face, and facial image data is generated.

[1229] Output: Face image data

[1230] Step 3:

[1231] The device uses facial recognition software to analyze the user's age, gender, and other attribute information from the captured facial image.

[1232] Input: Facial image data

[1233] Specific operation: Calls a facial recognition API and runs an image processing algorithm to estimate age and gender.

[1234] Output: Attribute information (age, gender, etc.)

[1235] Step 4:

[1236] The terminal inputs the acquired facial image data and attribute information into an emotion engine to analyze the user's emotional state.

[1237] Input: Facial image data, attribute information

[1238] Specific operation: Calls the emotion analysis API and quantifies the emotional state (e.g., joy, surprise, sadness, anger, etc.) from the facial image.

[1239] Output: Emotional information

[1240] Step 5:

[1241] The terminal transmits the analyzed attribute information and emotion information to the server.

[1242] Input: attribute information, emotion information

[1243] Specific operation: Data is sent to the server using the HTTPS protocol.

[1244] Output: Data sent to the server

[1245] Step 6:

[1246] Based on the information received by the server, it checks whether the user is registered in the database.

[1247] Input: attribute information, emotion information

[1248] Specific behavior: Runs an SQL database query to check the user's enrollment status based on facial information.

[1249] Output: Information on whether the user is an existing user or a new user

[1250] Step 7:

[1251] If the user is an existing user, the server retrieves the user's past order history from the database, and if the user is a new user, the server registers new face information, attribute information, and emotion information, and generates and saves a unique ID.

[1252] Input: User registration status, attribute information, emotional information

[1253] Specific operation: For existing users, retrieve past order data based on the user ID, and for new users, create a new record and insert the data.

[1254] Output: If an existing user, the retrieved past order history. If a new user, the registration completion ID.

[1255] Step 8:

[1256] Based on the data acquired by the server and newly analyzed information, the server analyzes the user's preferences and generates recommended menus.

[1257] Input: past order history, attribute information, emotional information

[1258] What it does: It uses machine learning algorithms to analyze and generate a menu tailored to the user's preferences.

[1259] Output: Recommended menu list

[1260] Step 9:

[1261] The server sends the generated recommended menu list to the terminal.

[1262] Input: Recommended menu list

[1263] Specific operation: Sends data to the terminal using the HTTPS protocol.

[1264] Output: Recommended menu list sent to the device

[1265] Step 10:

[1266] The terminal displays the recommended menu list received from the server on the ordering application.

[1267] Input: Recommended menu list

[1268] What it does: Displays the list on the ordering application screen so that the user can visually confirm it.

[1269] Output: Screen showing recommended menu

[1270] Step 11:

[1271] The user selects a product from the displayed menu and presses the order confirmation button.

[1272] Input: Select from the recommended menu

[1273] Specific actions: Tap to select a menu item and tap the order confirmation button.

[1274] Output: Selected menu and order confirmation information

[1275] Step 12:

[1276] The terminal transmits the selected order information to the server.

[1277] Input: Selected menu and order confirmation information

[1278] Specific operation: Order data is sent to the server using the HTTPS protocol.

[1279] Output: Order information sent to the server

[1280] Step 13:

[1281] The server stores the received order information in a database and uses it to make future recommendations.

[1282] Input: Order Information

[1283] Specific operation: Inserts order information into the database and saves the new data.

[1284] Output: Order information stored in the database

[1285] This concludes the overall processing flow of the system, which provides a personalized menu based on the user's emotional state and preferences.

[1286] (Application example 2)

[1287] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1288] Conventional personalized menu recommendation systems generally make static recommendations based on a user's past order history and attribute information, and are unable to consider the customer's emotional state in real time. This makes it difficult to provide the optimal menu based on the customer's current mood, and this has led to the problem of not being able to sufficiently increase customer satisfaction.

[1289] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user attribute information based on the analyzed attribute information and emotional state, emotion analysis means for analyzing the user's emotional state in real time, and means for acquiring the user's past order history based on the analyzed attribute information and emotional state. This makes it possible to provide the user with a personalized menu in real time that corresponds to their current emotional state.

[1290] "User-operable terminal means" refers to a terminal such as a smartphone, tablet, or smart glasses that allows a user to place an order, check the menu, etc.

[1291] "Means of identifying a user's face using facial recognition software and analyzing attribute information" refers to software that analyzes a user's attributes, such as age and gender, based on an image of the user's face taken with a camera.

[1292] "Means for analyzing user attribute information based on analyzed attribute information and emotional state" refers to a system for analyzing detailed attributes and preferences of a user using the acquired attribute information and emotional state.

[1293] "Emotion analysis means for analyzing the user's emotional state in real time" refers to a system for analyzing emotions such as joy, surprise, sadness, and anger based on the user's facial expressions.

[1294] "Server means for acquiring a user's past order history based on the analyzed attribute information and emotional state" refers to a system in which a server acquires a user's past order data based on the user's attribute information and emotional information.

[1295] "Server means for generating recommended menus based on acquired order history, attribute information, and emotional state" refers to a system in which a server generates a menu suitable for a user based on past order history, attribute information, and emotional state.

[1296] "Means for displaying the generated recommended menu on a terminal means and on smart glasses or other smart devices" refers to a system that displays the recommended menu generated by the server on a display device (such as smart glasses) in real time.

[1297] "Means for transmitting and storing user order information to server means" refers to a system that transmits order information selected by a user to a server and stores it for use in future recommendations.

[1298] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. Specifically, the system is configured and operates as follows.

[1299] User Interfaces and Terminals

[1300] Users use devices such as smart glasses, smartphones, or tablets, each with a built-in camera and capable of running facial recognition software, and an ordering application installed, which provides an interface for customers to select menu items and enter their order information.

[1301] Facial Recognition and Attribute Analysis

[1302] When a user puts on the smart glasses, the built-in camera captures the user's face, which is then analyzed by facial recognition software to determine the user's age, gender, and other demographic information, which is then analyzed in real time and sent to a server.

[1303] Emotional state analysis

[1304] Facial expression data obtained from facial images is input into the emotion engine, which analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). This makes it possible to understand the user's current mood and emotions. The analysis results are also sent to the server.

[1305] Server Processing

[1306] The server receives attribute information and emotional state from the device, and for existing users, retrieves past order history from the database. For new users, the server registers facial information along with attribute information and emotional information, generates a unique ID, and stores it in the database.

[1307] Generate menu recommendations

[1308] The server runs an algorithm that recommends the most suitable menu for the user based on the analyzed attribute information, emotional state, and past order history. If the user is excited, the server will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy options.

[1309] Displaying recommended menu items and ordering

[1310] The generated recommended menu list is displayed in real time on smart glasses or other smart devices. The user selects items from the displayed menu and confirms the order. The device then sends the selected order information to the server, which stores this information in a database and uses it to reflect future recommendations.

[1311] Specific examples

[1312] For example, suppose a 30-year-old male user is wearing smart glasses. The smart glasses' camera captures his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu items are displayed on the smart glasses, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[1313] Prompt Sentence Examples

[1314] When the customer looks at the menu displayed in the smart glasses, menu recommendations tailored to his emotions are displayed.

[1315] "We recommend the following menu items: Specialty Pizza, Carbonara Pasta"

[1316] Simply move your hand to select your order and the glasses will navigate you to the menu item's location in the store.

[1317] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[1318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1319] Step 1:

[1320] The user puts on the smart glasses and launches the ordering application.

[1321] Input: Smart glasses activation signal.

[1322] Output: The camera starts capturing face images.

[1323] Specific operation: The smart glasses' built-in camera captures an image of the user's face, and the image data is sent to the ordering application.

[1324] Step 2:

[1325] The device uses facial recognition software to analyze the user's face.

[1326] Input: A captured face image.

[1327] Output: User's demographic information such as age and gender.

[1328] Specific operation: Facial recognition software analyzes facial images, obtains attribute information such as age and gender, and temporarily stores this information in the device database.

[1329] Step 3:

[1330] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[1331] Input: A captured face image.

[1332] Output: The user's emotional state (e.g., happy, surprised, sad, angry, etc.).

[1333] Specific operation: The emotion engine analyzes the facial expression data, identifies the emotional state, and temporarily stores that information in the device database.

[1334] Step 4:

[1335] The terminal transmits the acquired attribute information and emotion information to the server.

[1336] Input: attribute information, emotion information.

[1337] Output: Attribute information and emotion information sent to the server.

[1338] Specific operation: The terminal transmits the analyzed attribute information and emotion information to the server as a data packet.

[1339] Step 5:

[1340] The server acquires the user's past order history based on the received information.

[1341] Input: attribute information, emotion information.

[1342] Output: Past order history or register as a new user.

[1343] Specific operation: Using the attribute information and emotion information received by the server, the server searches the database for past order history for existing users, and for new users, it registers them along with their facial information and generates a unique ID.

[1344] Step 6:

[1345] The server generates a recommended menu based on the analyzed information.

[1346] Input: Past order history, attribute information, emotional state.

[1347] Output: Recommended menu list.

[1348] Specific operation: The server uses a generative AI model to analyze past order history, attribute information, and current emotional state, and generates the most suitable recommended menu for the user.

[1349] Step 7:

[1350] The server sends the generated recommended menu list to the terminal.

[1351] Input: Recommended menu list.

[1352] Output: A list of recommended menus sent to the device.

[1353] Specific operation: The recommended menu list generated by the server is sent to the terminal as a data packet.

[1354] Step 8:

[1355] The device will display a list of recommended menu items and display them on the smart glasses.

[1356] Input: Recommended menu list.

[1357] Output: Recommendation menu displayed on smart glasses.

[1358] Specific operation: The recommended menu list received by the device is displayed in real time on the smart glasses display.

[1359] Step 9:

[1360] The user selects a product from the recommended menu and confirms the order.

[1361] Input: Recommended menu list.

[1362] Output: Selected order information.

[1363] Specific operation: The user selects a product from the menu displayed on the smart glasses and presses the order confirmation button.

[1364] Step 10:

[1365] The terminal transmits the selected order information to the server, which stores the information in a database.

[1366] Enter: Selected order information.

[1367] Output: Order information stored on the server.

[1368] Specific operation: The terminal sends the user's selected order information to the server, which stores the information in a database so that it can be reflected in future recommendations.

[1369] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1370] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1371] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1372] [Fourth embodiment]

[1373] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1374] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1375] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1376] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1377] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1378] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1379] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1380] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1381] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1382] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1383] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1384] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1385] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1386] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. The following describes in detail an embodiment of the present invention.

[1387] Facial Recognition and Attribute Analysis

[1388] 1. Tablet startup and face recognition

[1389] The user picks up the tablet and launches the ordering application.

[1390] The device uses a built-in camera to capture the user's face.

[1391] The device runs facial recognition software that analyzes the captured image to generate demographic information about the user, such as their age and gender.

[1392] 2. Processing of Attribute Information by the Server

[1393] The terminal transmits the acquired attribute information to the server.

[1394] The server checks whether the user is already registered based on the received attribute information.

[1395] In the case of an existing user, the server retrieves the user's past order history from the database.

[1396] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[1397] Generate menu recommendations

[1398] 3. View your past order history

[1399] For existing users, the server analyzes the user's preferences based on past order history.

[1400] For new users, the server generates a general recommendation menu based on their attribute information.

[1401] 4. Generate recommended menu

[1402] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[1403] The server sends the generated recommended menu list to the terminal.

[1404] Displaying recommended menu items and ordering

[1405] 5. Display of recommended menu items

[1406] The terminal displays the recommended menu list received from the server on the ordering application.

[1407] The recommended menu is highlighted to make it easier for users to make a selection.

[1408] 6. Placing orders and collecting feedback

[1409] The user selects the desired menu item and presses the order confirmation button.

[1410] The terminal transmits the selected order information to the server.

[1411] The server stores the received order information in a database and uses it to reflect future recommendations.

[1412] Specific examples

[1413] For example, when a 30-year-old male user opens an ordering application on a tablet, the device captures his face using the camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server then stores the order information in a database and uses it as data for future recommendations.

[1414] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[1415] The processing flow will be explained below.

[1416] Step 1:

[1417] The user picks up the tablet and launches the ordering application.

[1418] Step 2:

[1419] The device uses its built-in camera to capture the user's face.

[1420] Step 3:

[1421] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[1422] Step 4:

[1423] The terminal transmits the acquired attribute information to the server.

[1424] Step 5:

[1425] Based on the attribute information received by the server, the server searches the database to check whether the user is already registered.

[1426] Step 6:

[1427] If the server is an existing user, it retrieves the user's past order history from the database.

[1428] Step 7:

[1429] When the server receives a new user, it registers face information and attribute information, generates a unique ID, and stores it in the database.

[1430] Step 8:

[1431] The server analyzes the user's preferences based on the past order history and attribute information acquired.

[1432] Step 9:

[1433] The server runs an algorithm to generate recommended menu items based on the user's attribute information and past order data.

[1434] Step 10:

[1435] The server sends the generated recommended menu list to the terminal.

[1436] Step 11:

[1437] The terminal displays the recommended menu list received from the server on the ordering application.

[1438] Step 12:

[1439] The user selects a product from the displayed menu and presses the order confirmation button.

[1440] Step 13:

[1441] The terminal transmits the selected order information to the server.

[1442] Step 14:

[1443] The server stores the received order information in a database.

[1444] Step 15:

[1445] The server uses the stored order information as data to reflect in future recommendations.

[1446] Example 1

[1447] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1448] Existing systems have the problem that when providing personalized recommendations to users, the users themselves must perform many manual operations, making it difficult to automatically provide menus that meet the user's preferences. Another issue is that they cannot efficiently utilize user attribute information or past order history, making it difficult to provide highly accurate recommendations.

[1449] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1450] In this invention, the server includes an input device that can be operated by a user, a terminal that uses facial recognition software to identify the user's face and analyze their attribute information, a server that acquires the user's past order history based on the analyzed attribute information, a server that generates recommended menu items based on the acquired order history and attribute information, a display that displays the generated recommended menu items on the terminal, and a recording that transmits the user's order information to the server and stores it. This allows the user to receive menu recommendations that suit their preferences with minimal manual operation. Furthermore, highly accurate recommendations can be realized by efficiently utilizing attribute information and past order history.

[1451] The "user-operable input device means" is a device that allows a user to input information into the system by operating it himself.

[1452] "Facial recognition software" is a program that analyzes a user's facial image to identify the individual and extract attribute information.

[1453] "Terminal means" refers to a device that executes face recognition software, captures the user's face, and generates attribute information.

[1454] "Server means" refers to a central computer that stores and manages user attribute information and order history, and performs advanced calculation processing.

[1455] A "display means" is a device for visually presenting information to a user, or a partial function thereof.

[1456] "Storage means" refers to a device or function for storing data or information generated within the system.

[1457] "Information analysis means" refers to a program or function for analyzing collected data and extracting useful attribute information.

[1458] An "algorithm" is a set of calculations or procedures for solving a particular problem.

[1459] The present invention is directed to a system including a server and an input device operable by a user, which utilizes facial recognition and order history to recommend personalized menus. Hereinafter, embodiments of the present invention will be described in detail.

[1460] Facial Recognition and Attribute Analysis

[1461] 1. Tablet startup and face recognition

[1462] The user picks up the tablet and launches the ordering application. After turning on the tablet and tapping the application icon to launch the application, the message "Starting face recognition" is displayed.

[1463] The device captures the user's face using the built-in camera. When the camera is activated, it automatically recognizes the position of the user's face and captures it. This captured image is then analyzed by facial recognition software (e.g., OpenCV or Amazon Rekognition).

[1464] The device runs facial recognition software, which analyzes the captured image to generate demographic information about the user, such as their age and gender. Image analysis extracts the user's facial features and runs a model to estimate their age and gender. This process generates demographic information (e.g., "30-year-old male").

[1465] Server-based attribute processing

[1466] 2. Processing of Attribute Information by the Server

[1467] The device sends the acquired attribute information to the server. The generated attribute information is sent to the server in JSON format using the HTTP POST method.

[1468] The server queries a database (e.g., MySQL or PostgreSQL) based on the received attribute information to see if the user is already registered, and verifies the user's existence based on the face ID and attribute information.

[1469] If the user is an existing user, the server retrieves the user's past order history from the database. It uses an SQL query to retrieve the past order history associated with the user ID and makes it available within the application.

[1470] For a new user, the server registers the face information along with the attribute information, generates a new unique ID, and saves it in the database. The server registers the user's attribute information and the captured face image in the database and generates a new unique user ID.

[1471] Generate menu recommendations

[1472] 3. View your past order history

[1473] The server analyzes the user's preferences based on the user's past order history. It uses machine learning models (e.g., K-Means clustering and logistic regression) to analyze the user's past order history and identify the user's preferences.

[1474] 4. Generate recommended menu

[1475] The server runs a recommendation algorithm (e.g., collaborative filtering or content-based filtering) based on the user's attribute information and past order history. It combines the facial recognition results, attribute information, and past order history to generate a personalized menu using a pre-defined algorithm.

[1476] The server sends the generated recommended menu list to the device. It generates the recommended menu list in JSON format and returns it as an HTTP response.

[1477] Displaying recommended menu items and ordering

[1478] 5. Display of recommended menu items

[1479] The terminal displays the recommended menu list received from the server on the ordering application. The terminal parses the received data and displays it appropriately in the user interface. Recommended menu items are highlighted with eye-catching colors and fonts.

[1480] 6. Placing orders and collecting feedback

[1481] The user selects the desired menu item and presses the order confirmation button. By touching the screen, the user selects the menu item and taps the confirmation button. The terminal sends the selected order information to the server.

[1482] The terminal sends the selected order information to the server. Information such as the selected menu ID is sent in JSON format to the server.

[1483] The server stores the received order information in a database and uses it for future recommendations. The server also stores the transmitted data in a database and records it as an order history.

[1484] Specific examples

[1485] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[1486] Example prompts for generative AI models

[1487] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[1488] In this way, the system of the present invention allows users to easily select menu items that suit them, and allows restaurants to efficiently provide personalized services.

[1489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1490] System program processing flow

[1491] Step 1: Boot up the tablet and start the ordering application

[1492] The user picks up the tablet and launches the ordering application. Specifically, they turn on the tablet and tap the application icon. The message "Starting face recognition" appears on the screen. The input is the user's touch operation, and the output is the initial screen of the application.

[1493] Step 2: Obtaining user attribute information through facial recognition

[1494] The device captures the user's face using the built-in camera and analyzes the image using facial recognition software (e.g., OpenCV, Amazon Rekognition). The camera activates, identifies the user's face, and acquires image data. The input is the captured facial image, and the output is attribute information such as age and gender. Specific operations include extracting facial feature points and estimating age and gender.

[1495] Step 3: Send attribute information to the server and confirm with the user

[1496] The device sends the acquired attribute information in JSON format to the server. The server queries the database and checks whether the user is already registered based on the received attribute information. The input is the JSON data of the attribute information, and the output is the result of determining whether the user is new or existing. Specifically, the database search and face ID matching are performed.

[1497] Step 4: Get the order history of an existing user

[1498] If the user is an existing user, the server retrieves the user's past order history from the database. The input is the user ID, and the output is the user's past order history. The specific operation is to retrieve the order history from the database using an SQL query.

[1499] Step 5: Register a new user

[1500] When a user is new, the server registers their face information along with their attribute information in the database and generates a new unique user ID. The input is attribute information and face image data, and the output is the new user ID. Specifically, the new user data is inserted into the database and a unique ID is generated.

[1501] Step 6: Menu recommendations based on order history and attribute information

[1502] The server generates recommended menus based on the user's past order history and attribute information. It runs a recommendation algorithm to create a personalized menu list. The input is attribute information and order history, and the output is a list of recommended menus. Specific operations include collaborative filtering and content-based filtering.

[1503] Step 7: Displaying recommended menu items

[1504] The server generates a recommended menu list and sends it to the terminal, which then displays it on the ordering application. The input is the recommended menu list, and the output is the menu displayed on the ordering application. Specifically, the received data is parsed and displayed on the user interface.

[1505] Step 8: User confirms order

[1506] The user selects the menu item they want and presses the order confirmation button. The input is the user's selection and touch operation, and the output is the order data. The specific operation involves selecting a menu item and pressing the confirmation button.

[1507] Step 9: Send and store order information on the server

[1508] The terminal sends the selected order information in JSON format to the server. The server saves the order information in a database. The input is the order information, and the output is updating the database. Specifically, the order data is sent to the server and saved in the database.

[1509] Specific examples

[1510] For example, suppose a 30-year-old male user opens an ordering application on a tablet. The device captures his face using a camera and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. Since it is determined that he has previously preferred pizza and pasta, a special pizza and carbonara pasta are generated as recommended menu items. The device displays the recommended menu, and the user selects the pesto pasta and confirms his order. The server stores the order information in a database and uses it as data for future recommendations.

[1511] Example prompts for generative AI models

[1512] Prompt: "If a 30-year-old male user has previously preferred pizza and pasta, please recommend a menu item for him."

[1513] (Application example 1)

[1514] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] Conventional personalized menu and product recommendation systems mainly use terminals or PCs, which poses challenges in improving user convenience and experience. Furthermore, there is a lack of means for providing recommendation information in real time within the store, requiring users to actively operate their terminals. Therefore, there is a need for a more intuitive and convenient way to provide personalized information to users.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1517] In this invention, the server includes a terminal means operable by a user, a means for identifying a user's face and analyzing attribute information using face recognition software, a server means for acquiring the user's past order history based on the analyzed attribute information, a server means for generating recommended menu items based on the acquired order history and attribute information, a means for displaying the generated recommended menu items on the terminal means, a means for transmitting and storing the user's order information to the server means, and an information display means for performing face recognition and displaying recommended menu items using smart glasses. This enables users to intuitively and conveniently receive personalized menu and product recommendations in a store.

[1518] "User-operable terminal means" refers to an electronic terminal device that can be operated by the user himself / herself to input and display information.

[1519] "Means of identifying a user's face and analyzing attribute information using facial recognition software" refers to a combination of software and hardware that uses a camera and facial recognition algorithm to capture a user's face, and then extracts and analyzes attribute information such as age and gender from the video data.

[1520] The "server means for acquiring the user's past order history based on the analyzed attribute information" is a server that has the function of searching a database based on the user's attribute information and acquiring the user's past order history.

[1521] The "server means for generating recommended menus based on the acquired order history and attribute information" refers to a server that has an algorithm for analyzing the acquired user order history and attribute information and automatically generating recommended menus based thereon.

[1522] The "means for displaying the generated recommended menu on the terminal means" refers to software and hardware that has the function of displaying the recommended menu generated by the server on the user's terminal.

[1523] The "means for transmitting user order information to server means and storing it" refers to communication means and storage means for transmitting the menu selected by the user to the server and storing that information in a database.

[1524] "Information display means for performing facial recognition and displaying recommended menus using smart glasses" refers to a device and system that performs facial recognition through smart glasses worn by the user and displays personalized recommended menus based on the results on the glasses' display.

[1525] The present invention is directed to a system including a user-operable terminal and a server that utilizes facial recognition and order history to recommend personalized menu items. A specific embodiment of the present invention will be described.

[1526] Facial Recognition and Attribute Analysis

[1527] 1. Tablet startup and face recognition

[1528] When a user wears the smart glasses and enters a store, the smart glasses automatically activate their camera and capture the user's face.

[1529] Facial recognition software built into the smart glasses analyzes this image and generates demographic information about the user, such as their age and gender.

[1530] 2. Processing of Attribute Information by the Server

[1531] The smart glasses transmit the acquired attribute information to the server.

[1532] The server checks whether the user is already registered based on the received attribute information.

[1533] In the case of an existing user, the server retrieves the user's past order history from the database.

[1534] For new users, the server registers their facial information along with their attributes, generates a unique ID, and stores it in the database.

[1535] Generate menu recommendations

[1536] 3. View your past order history

[1537] For existing users, the server analyzes the user's preferences based on past order history.

[1538] For new users, the server generates a general recommendation menu based on their attribute information.

[1539] 4. Generate recommended menu

[1540] The server analyzes the user's attribute information and past order history and runs an algorithm to generate recommended menu items.

[1541] The server sends the generated recommended menu list to the smart glasses.

[1542] Displaying recommended menu items and ordering

[1543] 5. Display of recommended menu items

[1544] The smart glasses display the recommended menu list received from the server on the glasses' display.

[1545] The recommended menu is highlighted to make it easier for users to make a selection.

[1546] 6. Placing orders and collecting feedback

[1547] The user operates the smart glasses to select the desired menu item and confirm the order.

[1548] The smart glasses transmit the selected order information to the server.

[1549] The server stores the received order information in a database and uses it to reflect future recommendations.

[1550] To implement this system, the following hardware and software are used:

[1551] Hardware

[1552] Smart glasses (e.g. Google Glass)

[1553] Built-in camera (integrated into smart glasses)

[1554] software

[1555] OpenCV library (face recognition)

[1556] REST API (sending and receiving face images and attribute information)

[1557] Smart Glasses SDK

[1558] Specific examples

[1559] For example, when a 30-year-old male user wears smart glasses and enters a store, the camera in the smart glasses captures his face and recognizes his age and gender as "30-year-old male." The server confirms that the user's face information is already registered and references his past order history. As a result, it is determined that he has previously preferred casual sneakers, and the latest sneaker collection is generated as a recommended menu. The recommended menu is displayed on the smart glasses, and the user selects new sneakers from the menu and confirms the purchase. The server saves the order information in a database and uses it as data for future recommendations.

[1560] Prompt Sentence Examples

[1561] "You are an AI that recommends products in a store. When a user enters the store, it recognizes their face and suggests products based on their past purchase history. For example, if the user has liked sneakers in the past, it will recommend a new sneaker collection."

[1562] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1563] Step 1: Capture the user's face and generate attribute information

[1564] When a user wearing smart glasses enters a store, the glasses automatically activate the camera. The camera captures the user's face, and facial recognition software analyzes the facial image to generate attribute information such as age and gender. The input is the camera's video data, and the output is the user's attribute information.

[1565] Step 2: Sending attribute information to the server

[1566] The smart glasses send the attribute information generated by the facial recognition software to the server. The input is the attribute information, and the output is the attribute information sent to the server. The smart glasses' communication module is used.

[1567] Step 3: Get past order history

[1568] The server checks whether the user is already registered based on the attribute information it receives. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers the user's face information along with the attribute information, generates a unique ID, and saves it in the database. The input is attribute information, and the output is the user's past order history or new user registration information.

[1569] Step 4: Generate a recommendation menu

[1570] The server runs an algorithm that generates menu recommendations based on the user's attribute information and past order history. The input is attribute information and order history, and the output is a list of recommended menu items. A generative AI model is used to analyze past data and make predictions.

[1571] Step 5: Send recommendations to your smart glasses

[1572] The server sends the generated recommended menu list to the smart glasses. The input is the recommended menu list, and the output is the recommended menu list sent to the smart glasses. A communication module is used.

[1573] Step 6: Displaying recommended menu items

[1574] The smart glasses display the recommended menu list received from the server on the glasses' display. The input is the recommended menu list, and the output is the menu displayed on the display. The smart glasses' display control module is used.

[1575] Step 7: Confirm your order

[1576] The user operates the smart glasses to select the desired menu and confirm the order. The input is the user's selection information, and the output is the selected order information. The interface of the smart glasses is used.

[1577] Step 8: Send and store order information on the server

[1578] The smart glasses send the selected order information to the server, which stores the information in a database. The input is the order information, and the output is the order information stored on the server. A communication module and a database management system are used.

[1579] At each step, specific data processing and calculations are performed, enabling users to receive intuitive, convenient, and personalized menu and product recommendations in-store, improving the user experience.

[1580] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1581] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. The following describes in detail embodiments of the present invention.

[1582] Facial Recognition and Attribute Analysis

[1583] 1. Tablet startup and face recognition

[1584] The user picks up the tablet and launches the ordering application.

[1585] The device uses a built-in camera to capture the user's face.

[1586] The device runs facial recognition software and analyzes the user's age, gender and other demographic information from the captured image.

[1587] 2. Emotion Recognition by Emotion Engine

[1588] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state (e.g., joy, surprise, sadness, anger, etc.).

[1589] 3. Processing of attribute information and emotion information by the server

[1590] The terminal transmits the acquired attribute information and emotion information to the server.

[1591] Based on the received information, the server searches the database to check whether the user is already registered.

[1592] In the case of an existing user, the server retrieves the user's past order history from the database.

[1593] For new users, the server registers their facial information along with their attribute and emotion information, generates a unique ID, and stores it in the database.

[1594] Generate menu recommendations

[1595] 4. View your past order history

[1596] For existing users, the server analyzes the user's preferences based on past order history.

[1597] For new users, the server generates a general recommendation menu based on attribute information and emotional information.

[1598] 5. Generate recommended menu

[1599] The server analyzes the user's attribute information, emotional information, and past order data and runs an algorithm to generate menu recommendations.

[1600] If the user is excited, the system will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy menus, making recommendations based on the user's emotional state.

[1601] The server sends the generated recommended menu list to the terminal.

[1602] Displaying recommended menu items and ordering

[1603] 6. Display recommended menu items

[1604] The terminal displays the recommended menu list received from the server on the ordering application.

[1605] The recommended menu is highlighted to make it easier for users to make a selection.

[1606] 7. Confirming orders and collecting feedback

[1607] The user selects a product from the displayed menu and presses the order confirmation button.

[1608] The terminal transmits the selected order information to the server.

[1609] The server stores the received order information in a database and uses it to reflect future recommendations.

[1610] Specific examples

[1611] For example, when a 30-year-old male user opens an ordering application on a tablet, the device uses a camera to capture his face and recognize his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu is displayed on the device, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[1612] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[1613] The processing flow will be explained below.

[1614] Step 1:

[1615] The user picks up the tablet and launches the ordering application.

[1616] Step 2:

[1617] The device uses its built-in camera to capture the user's face.

[1618] Step 3:

[1619] The device runs facial recognition software and analyzes the user's age, gender, and other attributes from the captured image.

[1620] Step 4:

[1621] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[1622] Step 5:

[1623] The terminal transmits the acquired attribute information and emotion information to the server.

[1624] Step 6:

[1625] Based on the attribute information and emotion information received by the server, the server searches the database to check whether the user is already registered.

[1626] Step 7:

[1627] If the server is an existing user, it retrieves the user's past order history from the database.

[1628] Step 8:

[1629] When the server receives a new user, it registers face information, attribute information, and emotion information, generates a unique ID, and stores it in the database.

[1630] Step 9:

[1631] The server analyzes the user's preferences based on the past order history, attribute information, and emotional information acquired by the server.

[1632] Step 10:

[1633] The server runs an algorithm to generate menu recommendations based on the user's attribute information, emotional information, and past order data.

[1634] Step 11:

[1635] The system adjusts the recommended menu according to the user's emotional state; for example, if the user is excited, it will recommend hearty dishes or sweets, and if the user is relaxed, it will recommend light meals or healthy options.

[1636] Step 12:

[1637] The server sends the generated recommended menu list to the terminal.

[1638] Step 13:

[1639] The terminal displays the recommended menu list received from the server on the ordering application.

[1640] Step 14:

[1641] The user selects a product from the displayed menu and presses the order confirmation button.

[1642] Step 15:

[1643] The terminal transmits the selected order information to the server.

[1644] Step 16:

[1645] The server stores the received order information in a database.

[1646] Step 17:

[1647] The server uses the stored order information as data to reflect in future recommendations.

[1648] Example 2

[1649] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1650] To provide personalized services at restaurants and other establishments, it is important to take into account not only the user's attribute information and past ordering history, but also their emotional state at the time. However, conventional systems have not been able to properly analyze the user's emotional state and reflect it in recommendations. This has made it difficult to provide even more personalized services.

[1651] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes terminal means operable by the user, means for identifying the user's face and analyzing attribute information using face recognition software, means for analyzing the user's emotional state using the analyzed attribute information and an emotion engine, means for acquiring the user's past order history based on the analyzed attribute information and emotion information, means for generating recommended menus based on the acquired order history, attribute information, and emotion information, means for displaying the generated recommended menus on the terminal means, and means for transmitting the user's order information to the server means and storing it. This makes it possible to analyze the user's emotional state and provide personalized recommended menus based on the analyzed emotional state.

[1652] "User-operable terminal means" refers to an electronic device that can be physically operated by a user, and that can execute a specific application to display information or input data.

[1653] "Facial recognition software" refers to a program or algorithm that analyzes facial images captured by a camera or other imaging device to detect, identify, and analyze certain facial features.

[1654] "Attribute information" refers to personal characteristic data such as a user's age and gender, and includes data obtained by facial recognition or other analytical means.

[1655] An "emotion engine" refers to an algorithm or program that determines a user's current emotional state from their facial expressions, voice, etc.

[1656] "Server means" refers to a computer system or network device for storing, processing, and transmitting data, and includes programs and services for managing and executing a series of operations.

[1657] "Analyzed attribute information and emotion information" refers to the user's personal characteristic data and emotional state data obtained and analyzed by the facial recognition software and emotion engine.

[1658] "Past order history" refers to data that records the details of orders previously placed by a user, and is used to analyze the user's preferences.

[1659] A "recommended menu" refers to a list of dishes and drinks suggested to the user based on the acquired data, and is generated by a specific algorithm.

[1660] "Order information" refers to information about the menu items selected by the user, and includes data for specifically recording the contents of the order.

[1661] The present invention is a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. This system utilizes facial recognition and emotion analysis of the user to recommend personalized menus. Specific embodiments for implementing the present invention are described below.

[1662] First, a terminal that can be operated by the user is used. Examples of such terminals include tablets and smartphones. This terminal has an ordering application installed and is activated by the user's operation.

[1663] The device uses the built-in camera to capture the user's face. This process uses a built-in camera module (e.g., a standard webcam). After the facial image is captured, facial recognition software (e.g., a common facial recognition API) is used to identify the user's face and analyze attribute information such as age and gender. This analysis is performed using algorithms built into the facial recognition software.

[1664] Next, the device inputs the facial expression data obtained from the facial image into an emotion engine. The emotion engine (e.g., emotion analysis API) analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). After obtaining the analysis results, the device transmits this attribute information and emotion information to the server.

[1665] The server checks whether the user is already registered in the database based on the received attribute and emotion information. If the user is an existing user, the server retrieves the user's past order history from the database. If the user is a new user, the server registers new face, attribute, and emotion information, and generates and stores a unique ID. This registration process uses database manipulation software (e.g., a standard SQL database).

[1666] The server analyzes user preferences based on past data and newly acquired information. For existing users, it generates personalized menu recommendations based on past order history. For new users, it generates general menu recommendations based on attribute information and emotional information. This menu generation uses machine learning algorithms (e.g., a typical recommender system).

[1667] The generated recommended menu is sent from the server to the device and displayed on the ordering application. The user selects products based on this and confirms the order. The order information is sent from the device to the server, which stores it in a database and reflects it in the next recommendation.

[1668] As a concrete example, consider the case where a 30-year-old male user opens an ordering application on a tablet. The device uses a camera to capture his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state of "happiness" from his facial expression. The server confirms that the user's facial information has already been registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta. The server generates recommended menu items such as specialty pizza and carbonara pasta, and recommends more of certain menu items considering his state of happiness. The recommended menu items are displayed on the device, and the user makes a selection and confirms the order. The server stores this order information in a database and uses it for future recommendations.

[1669] The above is a specific embodiment of the present invention. This system allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide more personalized services that take emotional states into consideration.

[1670] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1671] Step 1:

[1672] The user turns on the tablet and launches the dedicated ordering application.

[1673] Input: User actions

[1674] Specific action: Tap the ordering application icon on the tablet's home screen.

[1675] Output: The ordering application is launched.

[1676] Step 2:

[1677] The device activates its built-in camera to capture the user's face.

[1678] Input: A face image captured from the camera

[1679] Specific operation: The tablet's front camera takes a picture of the user's face, and facial image data is generated.

[1680] Output: Face image data

[1681] Step 3:

[1682] The device uses facial recognition software to analyze the user's age, gender, and other attribute information from the captured facial image.

[1683] Input: Facial image data

[1684] Specific operation: Calls a facial recognition API and runs an image processing algorithm to estimate age and gender.

[1685] Output: Attribute information (age, gender, etc.)

[1686] Step 4:

[1687] The terminal inputs the acquired facial image data and attribute information into an emotion engine to analyze the user's emotional state.

[1688] Input: Facial image data, attribute information

[1689] Specific operation: Calls the emotion analysis API and quantifies the emotional state (e.g., joy, surprise, sadness, anger, etc.) from the facial image.

[1690] Output: Emotional information

[1691] Step 5:

[1692] The terminal transmits the analyzed attribute information and emotion information to the server.

[1693] Input: attribute information, emotion information

[1694] Specific operation: Data is sent to the server using the HTTPS protocol.

[1695] Output: Data sent to the server

[1696] Step 6:

[1697] Based on the information received by the server, it checks whether the user is registered in the database.

[1698] Input: attribute information, emotion information

[1699] Specific behavior: Runs an SQL database query to check the user's enrollment status based on facial information.

[1700] Output: Information on whether the user is an existing user or a new user

[1701] Step 7:

[1702] If the user is an existing user, the server retrieves the user's past order history from the database, and if the user is a new user, the server registers new face information, attribute information, and emotion information, and generates and saves a unique ID.

[1703] Input: User registration status, attribute information, emotional information

[1704] Specific operation: For existing users, retrieve past order data based on the user ID, and for new users, create a new record and insert the data.

[1705] Output: If an existing user, the retrieved past order history. If a new user, the registration completion ID.

[1706] Step 8:

[1707] Based on the data acquired by the server and newly analyzed information, the server analyzes the user's preferences and generates recommended menus.

[1708] Input: past order history, attribute information, emotional information

[1709] What it does: It uses machine learning algorithms to analyze and generate a menu tailored to the user's preferences.

[1710] Output: Recommended menu list

[1711] Step 9:

[1712] The server sends the generated recommended menu list to the terminal.

[1713] Input: Recommended menu list

[1714] Specific operation: Sends data to the terminal using the HTTPS protocol.

[1715] Output: Recommended menu list sent to the device

[1716] Step 10:

[1717] The terminal displays the recommended menu list received from the server on the ordering application.

[1718] Input: Recommended menu list

[1719] What it does: Displays the list on the ordering application screen so that the user can visually confirm it.

[1720] Output: Screen showing recommended menu

[1721] Step 11:

[1722] The user selects a product from the displayed menu and presses the order confirmation button.

[1723] Input: Select from the recommended menu

[1724] Specific actions: Tap to select a menu item and tap the order confirmation button.

[1725] Output: Selected menu and order confirmation information

[1726] Step 12:

[1727] The terminal transmits the selected order information to the server.

[1728] Input: Selected menu and order confirmation information

[1729] Specific operation: Order data is sent to the server using the HTTPS protocol.

[1730] Output: Order information sent to the server

[1731] Step 13:

[1732] The server stores the received order information in a database and uses it to make future recommendations.

[1733] Input: Order Information

[1734] Specific operation: Inserts order information into the database and saves the new data.

[1735] Output: Order information stored in the database

[1736] This concludes the overall processing flow of the system, which provides a personalized menu based on the user's emotional state and preferences.

[1737] (Application example 2)

[1738] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1739] Conventional personalized menu recommendation systems generally make static recommendations based on a user's past order history and attribute information, and are unable to consider the customer's emotional state in real time. This makes it difficult to provide the optimal menu based on the customer's current mood, and this has led to the problem of not being able to sufficiently increase customer satisfaction.

[1740] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user attribute information based on the analyzed attribute information and emotional state, emotion analysis means for analyzing the user's emotional state in real time, and means for acquiring the user's past order history based on the analyzed attribute information and emotional state. This makes it possible to provide the user with a personalized menu in real time that corresponds to their current emotional state.

[1741] "User-operable terminal means" refers to a terminal such as a smartphone, tablet, or smart glasses that allows a user to place an order, check the menu, etc.

[1742] "Means of identifying a user's face using facial recognition software and analyzing attribute information" refers to software that analyzes a user's attributes, such as age and gender, based on an image of the user's face taken with a camera.

[1743] "Means for analyzing user attribute information based on analyzed attribute information and emotional state" refers to a system for analyzing detailed attributes and preferences of a user using the acquired attribute information and emotional state.

[1744] "Emotion analysis means for analyzing the user's emotional state in real time" refers to a system for analyzing emotions such as joy, surprise, sadness, and anger based on the user's facial expressions.

[1745] "Server means for acquiring a user's past order history based on the analyzed attribute information and emotional state" refers to a system in which a server acquires a user's past order data based on the user's attribute information and emotional information.

[1746] "Server means for generating recommended menus based on acquired order history, attribute information, and emotional state" refers to a system in which a server generates a menu suitable for a user based on past order history, attribute information, and emotional state.

[1747] "Means for displaying the generated recommended menu on a terminal means and on smart glasses or other smart devices" refers to a system that displays the recommended menu generated by the server on a display device (such as smart glasses) in real time.

[1748] "Means for transmitting and storing user order information to server means" refers to a system that transmits order information selected by a user to a server and stores it for use in future recommendations.

[1749] The present invention utilizes facial recognition and emotion analysis to recommend personalized menus using a system including a user-operable terminal, a server, facial recognition software, and an emotion engine. Specifically, the system is configured and operates as follows.

[1750] User Interfaces and Terminals

[1751] Users use devices such as smart glasses, smartphones, or tablets, each with a built-in camera and capable of running facial recognition software, and an ordering application installed, which provides an interface for customers to select menu items and enter their order information.

[1752] Facial Recognition and Attribute Analysis

[1753] When a user puts on the smart glasses, the built-in camera captures the user's face, which is then analyzed by facial recognition software to determine the user's age, gender, and other demographic information, which is then analyzed in real time and sent to a server.

[1754] Emotional state analysis

[1755] Facial expression data obtained from facial images is input into the emotion engine, which analyzes the user's emotional state (e.g., joy, surprise, sadness, anger, etc.). This makes it possible to understand the user's current mood and emotions. The analysis results are also sent to the server.

[1756] Server Processing

[1757] The server receives attribute information and emotional state from the device, and for existing users, retrieves past order history from the database. For new users, the server registers facial information along with attribute information and emotional information, generates a unique ID, and stores it in the database.

[1758] Generate menu recommendations

[1759] The server runs an algorithm that recommends the most suitable menu for the user based on the analyzed attribute information, emotional state, and past order history. If the user is excited, the server will recommend hearty dishes and sweets, and if the user is relaxed, it will recommend light meals and healthy options.

[1760] Displaying recommended menu items and ordering

[1761] The generated recommended menu list is displayed in real time on smart glasses or other smart devices. The user selects items from the displayed menu and confirms the order. The device then sends the selected order information to the server, which stores this information in a database and uses it to reflect future recommendations.

[1762] Specific examples

[1763] For example, suppose a 30-year-old male user is wearing smart glasses. The smart glasses' camera captures his face and recognizes his age and gender as "30-year-old male." At the same time, the emotion engine determines his emotional state, such as "happiness," from his facial expression. The server confirms that the user's facial information is already registered and references his past order history. As a result, it is determined that he has previously preferred pizza and pasta, and a special pizza and carbonara pasta are generated as recommended menu items. Furthermore, taking into account his state of happiness, the system recommends more of his favorite dishes. The recommended menu items are displayed on the smart glasses, and the user selects the pesto pasta and confirms his order. The server saves the order information in a database and uses it as data for future recommendations.

[1764] Prompt Sentence Examples

[1765] When the customer looks at the menu displayed in the smart glasses, menu recommendations tailored to his emotions are displayed.

[1766] "We recommend the following menu items: Specialty Pizza, Carbonara Pasta"

[1767] Simply move your hand to select your order and the glasses will navigate you to the menu item's location in the store.

[1768] In this way, the system of the present invention allows users to easily select a menu that suits their emotional state at the time, and allows restaurants to provide even more personalized services that take emotional states into consideration.

[1769] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1770] Step 1:

[1771] The user puts on the smart glasses and launches the ordering application.

[1772] Input: Smart glasses activation signal.

[1773] Output: The camera starts capturing face images.

[1774] Specific operation: The smart glasses' built-in camera captures an image of the user's face, and the image data is sent to the ordering application.

[1775] Step 2:

[1776] The device uses facial recognition software to analyze the user's face.

[1777] Input: A captured face image.

[1778] Output: User's demographic information such as age and gender.

[1779] Specific operation: Facial recognition software analyzes facial images, obtains attribute information such as age and gender, and temporarily stores this information in the device database.

[1780] Step 3:

[1781] The device inputs facial expression data obtained from the facial image into an emotion engine to analyze the user's emotional state.

[1782] Input: A captured face image.

[1783] Output: The user's emotional state (e.g., happy, surprised, sad, angry, etc.).

[1784] Specific operation: The emotion engine analyzes the facial expression data, identifies the emotional state, and temporarily stores that information in the device database.

[1785] Step 4:

[1786] The terminal transmits the acquired attribute information and emotion information to the server.

[1787] Input: attribute information, emotion information.

[1788] Output: Attribute information and emotion information sent to the server.

[1789] Specific operation: The terminal transmits the analyzed attribute information and emotion information to the server as a data packet.

[1790] Step 5:

[1791] The server acquires the user's past order history based on the received information.

[1792] Input: attribute information, emotion information.

[1793] Output: Past order history or register as a new user.

[1794] Specific operation: Using the attribute information and emotion information received by the server, the server searches the database for past order history for existing users, and for new users, it registers them along with their facial information and generates a unique ID.

[1795] Step 6:

[1796] The server generates a recommended menu based on the analyzed information.

[1797] Input: Past order history, attribute information, emotional state.

[1798] Output: Recommended menu list.

[1799] Specific operation: The server uses a generative AI model to analyze past order history, attribute information, and current emotional state, and generates the most suitable recommended menu for the user.

[1800] Step 7:

[1801] The server sends the generated recommended menu list to the terminal.

[1802] Input: Recommended menu list.

[1803] Output: A list of recommended menus sent to the device.

[1804] Specific operation: The recommended menu list generated by the server is sent to the terminal as a data packet.

[1805] Step 8:

[1806] The device will display a list of recommended menu items and display them on the smart glasses.

[1807] Input: Recommended menu list.

[1808] Output: Recommendation menu displayed on smart glasses.

[1809] Specific operation: The recommended menu list received by the device is displayed in real time on the smart glasses display.

[1810] Step 9:

[1811] The user selects a product from the recommended menu and confirms the order.

[1812] Input: Recommended menu list.

[1813] Output: Selected order information.

[1814] Specific operation: The user selects a product from the menu displayed on the smart glasses and presses the order confirmation button.

[1815] Step 10:

[1816] The terminal transmits the selected order information to the server, which stores the information in a database.

[1817] Enter: Selected order information.

[1818] Output: Order information stored on the server.

[1819] Specific operation: The terminal sends the user's selected order information to the server, which stores the information in a database so that it can be reflected in future recommendations.

[1820] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1821] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1822] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1823] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1824] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1825] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1826] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1827] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1828] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1829] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1830] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1831] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1832] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1833] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1834] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1835] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1836] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1837] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1838] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1839] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1840] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1841] The following is further disclosed regarding the above embodiment.

[1842] (Claim 1)

[1843] a terminal means operable by a user;

[1844] A means for identifying a user's face and analyzing attribute information using facial recognition software;

[1845] A server means for acquiring the user's past order history based on the analyzed attribute information;

[1846] A server means for generating recommended menus based on the acquired order history and attribute information;

[1847] A means for displaying the generated recommendation menu on a terminal means;

[1848] means for transmitting user order information to a server means and storing the information;

[1849] A system including:

[1850] (Claim 2)

[1851] 10. The system of claim 1, further comprising means for identifying age and gender when analyzing user attribute information using facial recognition software.

[1852] (Claim 3)

[1853] 2. The system according to claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on a user's attribute information and order history.

[1854] "Example 1"

[1855] (Claim 1)

[1856] an input device means operable by a user;

[1857] A terminal means for identifying a user's face and analyzing attribute information using face recognition software;

[1858] A server means for acquiring the user's past order history based on the analyzed attribute information;

[1859] A server means for generating recommended menus based on the acquired order history and attribute information;

[1860] a display means for displaying the generated recommendation menu on a terminal means;

[1861] a recording means for transmitting user order information to a server means and storing the information;

[1862] A system including:

[1863] (Claim 2)

[1864] 2. The system according to claim 1, further comprising information analysis means for identifying age and gender when analyzing user attribute information using face recognition software.

[1865] (Claim 3)

[1866] 2. The system according to claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on a user's attribute information and order history.

[1867] "Application Example 1"

[1868] (Claim 1)

[1869] a terminal means operable by a user;

[1870] A means for identifying a user's face and analyzing attribute information using facial recognition software;

[1871] A server means for acquiring the user's past order history based on the analyzed attribute information;

[1872] A server means for generating recommended menus based on the acquired order history and attribute information;

[1873] A means for displaying the generated recommendation menu on a terminal means;

[1874] means for transmitting user order information to a server means and storing the information;

[1875] An information display means that uses smart glasses to perform face recognition and display recommended menus;

[1876] A system including:

[1877] (Claim 2)

[1878] 10. The system of claim 1, further comprising means for identifying age and gender when analyzing user attribute information using facial recognition software.

[1879] (Claim 3)

[1880] 2. The system according to claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on a user's attribute information and order history.

[1881] "Example 2: Combining Emotion Engines"

[1882] (Claim 1)

[1883] a terminal means operable by a user;

[1884] A means for identifying a user's face and analyzing attribute information using facial recognition software;

[1885] means for analyzing the emotional state of a user using the analyzed attribute information and an emotion engine;

[1886] a server means for acquiring the user's past order history based on the analyzed attribute information and emotion information;

[1887] a server means for generating recommended menus based on the acquired order history, attribute information, and emotion information;

[1888] A means for displaying the generated recommendation menu on a terminal means;

[1889] means for transmitting user order information to a server means and storing the information;

[1890] A system including:

[1891] (Claim 2)

[1892] 10. The system of claim 1, further comprising means for identifying age, gender, and emotional state when analyzing the user's demographic information and emotional state using facial recognition software and an emotion engine.

[1893] (Claim 3)

[1894] 2. The system according to claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on a user's attribute information, emotional state, and order history.

[1895] "Application example 2 when combining emotion engines"

[1896] (Claim 1)

[1897] a terminal means operable by a user;

[1898] A means for identifying a user's face and analyzing attribute information using facial recognition software;

[1899] means for analyzing the user's attribute information based on the analyzed attribute information and emotional state;

[1900] emotion analysis means for analyzing the user's emotional state in real time;

[1901] a server means for acquiring the user's past order history based on the analyzed attribute information and emotional state;

[1902] a server means for generating recommended menus based on the acquired order history, attribute information, and emotional state;

[1903] A means for displaying the generated recommendation menu on a terminal means and a means for displaying the recommendation menu on smart glasses or other smart devices;

[1904] means for transmitting user order information to a server means and storing the information;

[1905] A system including:

[1906] (Claim 2)

[1907] 10. The system of claim 1, further comprising means for identifying age and gender when analyzing user attribute information using facial recognition software.

[1908] (Claim 3)

[1909] 10. The system of claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on a user's attribute information, emotional state, and order history. [Explanation of symbols]

[1910] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a terminal means operable by a user; A means for identifying a user's face and analyzing attribute information using facial recognition software; A server means for acquiring the user's past order history based on the analyzed attribute information; A server means for generating recommended menus based on the acquired order history and attribute information; A means for displaying the generated recommendation menu on a terminal means; means for transmitting user order information to a server means and storing the information; A system including:

2. The system according to claim 1, further comprising means for identifying age and gender when analyzing the user's attribute information using face recognition software.

3. The system according to claim 1, further comprising a server means including an algorithm for recommending an optimal menu item based on the user's attribute information and order history.

Images