System

Abstract

A system is provided.SOLUTION: This system is provided with a means for collecting menu conditions inputted by a user, a means for generating a menu by a language generation AI on the basis of the menu conditions, a means for selecting a relevant advertisement on the basis of information inputted by the user, and a means for transmitting the generated menu and the selected advertisement to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Planning a daily menu is a burden for housewives and other household members in many households, especially when deciding on breakfast, lunch, and dinner menus, as well as side dishes for bento lunches. Fixed menus often lead to nutritional imbalances and a sense of monotony. Furthermore, there is a demand for reducing the time required for preparing ingredients and cooking. For this reason, there is a need for a system that can easily and conveniently suggest menus. [Means for solving the problem]

[0005] This invention provides a system that collects menu requirements entered by users and generates menus using language generation AI based on those requirements. The system also includes a means for selecting relevant advertisements based on the information entered by the user and sending them to the user's device along with the generated menu. Furthermore, the system also includes a function for acquiring the user's current location, collecting flyer information from nearby commercial facilities, analyzing the results using image analysis AI, and reflecting the results in the menu, thereby improving user convenience and the accuracy of menu suggestions. This system allows for efficient daily menu planning, reduces the effort required for ingredient preparation, and enables the provision of nutritionally balanced meals.

[0006] "Menu conditions" is information including requirements such as the user's desired cuisine genre, theme (hearty, stylish, health-conscious, etc.), ingredients, etc.

[0007] "Language generation AI" is an artificial intelligence that uses natural language processing technology to generate specific menus based on the user's menu requirements.

[0008] "Advertisements" refer to promotions for related products and services based on information entered by the user.

[0009] A "terminal" is a computer device (such as a smartphone, tablet, or PC) that a user directly operates to input or receive information.

[0010] "Flyer information" is advertising information that lists prices and special offers for products offered by commercial facilities.

[0011] "Image analysis AI" is an artificial intelligence that analyzes image data and extracts and analyzes necessary information.

[0012] "Product information" refers to information such as the name, price, and features of specific products obtained from flyers or stores. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system that can automatically propose meal menus based on user-specified criteria and display related advertisements. Specifically, it collects menu conditions entered by the user and generates menus based on those conditions using language generation AI. Furthermore, it uses the user's current location information to collect flyers from nearby commercial facilities and analyzes them using image analysis AI to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the necessary ingredients.

[0035] Program processing explanation

[0036] User input of information

[0037] 1. User Action:

[0038] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[0039] Server processing of input information

[0040] 2. Server processing:

[0041] The server receives the menu requirements sent by the user, formats the received information appropriately, and passes it to the language generation AI.

[0042] Menu generation using language generation AI

[0043] 3. Server Operation:

[0044] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the AI ​​will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0045] Advertisement selection and display

[0046] 4. Server Operation:

[0047] The server selects relevant advertisements based on the user's input information. For example, it selects advertisements for health foods for health-conscious menus, and advertisements for fast food for hearty menus.

[0048] 5. Server Operation:

[0049] The server combines the generated menu with the selected advertisement and transmits it to the user's terminal.

[0050] 6. Device Operation:

[0051] The user's terminal receives the response from the server and displays the menu and advertisements on the screen.

[0052] Implementing the extension

[0053] 7. User Actions:

[0054] When a user wants to use the extended function, the user transmits current location information from the terminal to the server.

[0055] 8. Server Operation:

[0056] The server collects flyer information for nearby commercial facilities based on current location information.

[0057] 9. Server Operation:

[0058] The server uses image analysis AI to analyze flyer information and extract appropriate product information, reflecting the results in the menu and providing recommendations.

[0059] Detailed recipe provided

[0060] 10. Server Operation:

[0061] The server generates a link to the detailed recipe associated with the suggested meal.

[0062] 11. Terminal Operation:

[0063] When the user clicks on the link, the device will transition to the recipe site to display the detailed recipe.

[0064] Specific examples

[0065] Example 1: Healthy Japanese menu

[0066] 1. User input:

[0067] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[0068] 2. Server processing:

[0069] The server sends this information to the language generation AI.

[0070] 3. Results of language generation AI:

[0071] The language generation AI suggests "grilled salted salmon, boiled broccoli, and brown rice."

[0072] 4. Advertisement Selection:

[0073] The server selects advertisements for health foods and beauty salons and sends them to the user's terminal.

[0074] 5. Displaying the results:

[0075] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[0076] 6. Use of extensions:

[0077] When a user submits their current location information, the server collects flyer information from nearby commercial facilities and makes additional suggestions based on the analysis results.

[0078] This allows users to easily plan their daily menu, and by utilizing advertising and flyer information, they can also obtain information on how to shop at bargain prices.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[0082] Step 2:

[0083] The user clicks the "Submit" button to send the entered menu requirements to the server.

[0084] Step 3:

[0085] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[0086] Step 4:

[0087] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[0088] Step 5:

[0089] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[0090] Step 6:

[0091] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[0092] Step 7:

[0093] The server selects relevant advertisements based on the information entered by the user. For example, advertisements for health foods and beauty salons are selected for health-conscious criteria, and advertisements for fast food are selected for hearty meals.

[0094] Step 8:

[0095] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[0096] Step 9:

[0097] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0098] Step 10:

[0099] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0100] Step 11:

[0101] The server uses the user's current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[0102] Step 12:

[0103] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[0104] Step 13:

[0105] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0106] Step 14:

[0107] The server sends the newly generated recommendations to the user's device.

[0108] Step 15:

[0109] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0110] Step 16:

[0111] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0112] Step 17:

[0113] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0114] By following the above steps, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] The present invention aims to provide a system that allows users to efficiently decide on a menu and obtain related information. Specifically, the system automatically generates an appropriate menu based on various menu conditions specified by the user, collects sales information from nearby commercial facilities using the user's current location information, and uses image analysis technology to suggest optimal product information, thereby enabling users to easily plan their daily meals.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for collecting menu conditions input by the user, means for generating a menu using a generative AI model, means for selecting relevant advertisements based on the information input by the user, means for transmitting the generated menu and selected advertisements to the user's terminal, means for transmitting current location information operated by the user, means for collecting sales information of nearby commercial facilities based on the current location information, means for analyzing the sales information using image analysis technology, and means for recommending product information based on the analyzed sales information. This allows users to obtain menus automatically generated based on a variety of menu conditions, and further enables them to obtain optimal product information by utilizing sales information based on their current location.

[0120] The "means for collecting menu conditions" is a function that collects information such as the cooking genre, theme, type of menu, and ingredients desired to be used that the user inputs within the application.

[0121] "Means for generating menus using a generative AI model" is a function that uses artificial intelligence to automatically generate appropriate menus based on collected menu conditions.

[0122] The "means for selecting advertisements" is a function by which the server searches and selects relevant advertisements based on the information entered by the user and the generated menu.

[0123] "Means for transmitting menu and selected advertisement to user's terminal" is a function in which the server compiles the generated menu information and selected advertisement information and transmits them as data to the user's terminal.

[0124] The "means for transmitting current location information" is a function that allows the user to obtain current location information through an operation and transmit it to the server.

[0125] The "means for collecting sales information" is a function that allows the server to obtain sales information from nearby commercial facilities based on the user's current location information.

[0126] The "means for analyzing sales information using image analysis technology" is a function for analyzing collected sales information using image analysis technology and extracting useful information.

[0127] "Means for recommending product information" is a function that suggests optimal product information to users based on analyzed sales information.

[0128] The present invention is a system that automatically proposes meal plans based on user-specified criteria and displays related advertisements. Specifically, it collects menu plan conditions entered by the user and generates menu plans based on those conditions using a generative AI model. It also utilizes the user's current location information to collect sales information from nearby commercial facilities and analyzes it using image analysis technology to provide optimal product information. A specific embodiment of this system is described below.

[0129] User input of information

[0130] First, the user launches the application and inputs the menu requirements. Specifically, the user enters information into a form, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). This input information is collected on the device and then sent to the server.

[0131] Processing of input information and menu generation by the server

[0132] The server receives the menu requirements sent by the user and converts them into an appropriate format. It then passes the requirements to a generative AI model (such as OpenAI's GPT-4) to generate an appropriate menu. The generative AI model analyzes the input requirements and suggests an appropriate menu. For example, given the requirements of "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and ingredients of "salmon, broccoli, brown rice," it would generate a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0133] Ad selection and display

[0134] Based on the generated menu, the server searches and selects relevant advertisements. For example, for a health-conscious menu, advertisements for healthy foods are selected. These advertisements are sent to the user's terminal together with the generated menu and displayed on the terminal.

[0135] Utilizing current location information and collecting sales information

[0136] Furthermore, if the user provides their current location information, the device sends that information to a server. Based on the current location information, the server collects sales information from nearby commercial facilities via APIs and other means. The collected sales information is analyzed using image analysis technology (for example, Google Cloud Vision API) to extract useful product information.

[0137] Product recommendations and detailed recipes

[0138] The extracted product information is recommended to the user and reflected in the menu. The server also generates a link to a detailed recipe related to the suggested menu and sends it to the terminal in a format that the user can use. When the user clicks the link, the terminal is redirected to a detailed recipe site where the detailed recipe is displayed.

[0139] Specific examples

[0140] For example, if a user inputs "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients, the generative AI model will suggest "grilled salmon with salt, boiled broccoli, and brown rice." The server will then use image analysis technology to analyze health food advertisements and related product information sold at nearby commercial facilities, and make additional suggestions based on the results. This allows users to easily decide on a menu and obtain the most appropriate information when shopping.

[0141] Prompt Sentence Examples

[0142] "Please suggest a menu for tonight's dinner. I'd like to use salmon, broccoli, and brown rice as healthy ingredients for a Japanese meal."

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1: The user launches the application and enters menu requirements.

[0145] Specific operation: The user moves from the application's home screen to the menu condition input form. The form has input fields for cooking genre, theme, menu type, and desired ingredients. The user enters the conditions in these fields and presses the submit button.

[0146] Input: cuisine genre, theme, type of menu, ingredients you want to use

[0147] Output: Menu conditions entered by the user

[0148] Step 2: The terminal sends the input data from the user to the server.

[0149] What happens: The device collects the data entered by the user and sends it to the server as an HTTP request. The data is encoded in JSON format.

[0150] Input: Menu conditions entered by the user

[0151] Output: Menu conditions sent to the server

[0152] Step 3: The server receives the menu requirements sent by the user and converts them into an appropriate format.

[0153] What it does: The server parses the received data and formats it for internal processing, in a format that is easy for the generative AI model to understand (e.g., JSON).

[0154] Input: Menu conditions received from the terminal

[0155] Output: Formatted menu items

[0156] Step 4: The server sends the formatted menu requirements to the generative AI model.

[0157] Specific operation: The server sends the formatted data as a request to the API of the generative AI model (e.g., OpenAI's GPT-4).

[0158] Input: Formatted menu requirements

[0159] Output: The request sent to the generative AI model

[0160] Step 5: The generative AI model analyzes the conditions and generates the optimal menu.

[0161] Specific operation: The generative AI model analyzes the received conditions and generates an appropriate menu that meets the conditions. The result is sent back to the server in JSON format.

[0162] Input: The request sent to the generative AI model

[0163] Output: Generated menu

[0164] Step 6: The server receives the generated menu and selects relevant advertisements.

[0165] Specific operation: The server searches the advertisement database based on the generated menu information and selects relevant advertisements. For example, for a health-conscious menu, it selects advertisements for health foods.

[0166] Input: Generated menu

[0167] Output: Selected ads

[0168] Step 7: The server integrates the generated menu with the selected advertisements and sends it to the user's terminal.

[0169] Specific operation: The server combines the generated menu data and the selected advertising data into a single JSON object and sends it to the user's device as an HTTP response.

[0170] Input: Generated menu and selected advertisement

[0171] Output: The response sent to the user's device

[0172] Step 8: The user's device receives the response from the server and displays the menu and advertisements on the screen.

[0173] Specific operation: The device analyzes the received response data and displays menu information and advertising information in the appropriate position on the screen.

[0174] Input: Response from the server

[0175] Output: Menu and advertisements displayed on the screen

[0176] Step 9: If the user provides current location information, the terminal sends the information to the server.

[0177] Specific operation: The user presses the "Send current location" button within the application, and the device obtains GPS data and sends it to the server as an HTTP request.

[0178] Input: User's current location information

[0179] Output: Current location information sent to the server

[0180] Step 10: The server collects sales information from nearby commercial facilities based on the current location information.

[0181] Specific operation: The server uses the collected location information to obtain sales information from the commercial facility's API or database.

[0182] Input: User's current location information

[0183] Output: Retrieved sales information

[0184] Step 11: The server analyzes the collected sales information using image analysis technology.

[0185] Specific operation: The server analyzes the collected sales information using image analysis technology (e.g., Google Cloud Vision API) and extracts useful product information.

[0186] Input: Collected sales information

[0187] Output: Parsed product information

[0188] Step 12: Based on the analyzed product information, the server recommends the most suitable product information to the user.

[0189] Specific operation: Based on the analyzed product information, the server generates recommendations for the user and sends them to the user's device.

[0190] Input: Parsed product information

[0191] Output: Recommendations sent to the user's device

[0192] (Application example 1)

[0193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0194] Conventional food delivery applications require users to select the menu themselves, and they lack the ability to provide menu suggestions and related information. Furthermore, there is a need for applications that utilize the user's current location to collect special offers and menu information from nearby restaurants and efficiently present optimized options.

[0195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0196] In this invention, the server includes means for collecting menu requirements entered by the user, means for generating a menu using language generation AI, means for selecting relevant advertisements based on the information entered by the user, means for acquiring the user's current location information and collecting special offers and menu information from nearby restaurants, and means for transmitting the generated menu and selected advertisements to the user's terminal. This allows the user to efficiently receive optimal menu suggestions and conveniently take advantage of special offer information from nearby restaurants.

[0197] "Menu conditions" refers to information necessary when deciding on a menu, such as the type of cuisine the user desires, the theme, the budget, and the ingredients they wish to use.

[0198] "Language generation AI" is artificial intelligence that uses natural language processing technology to generate text based on specified conditions.

[0199] "Advertisement" means information or messages provided to users for the purpose of promoting a particular product or service.

[0200] "Current location information" refers to location information obtained from a user terminal and is used to identify the geographic location of the user.

[0201] "Restaurant" refers to a facility or store that serves food and drinks.

[0202] "Special offers" refers to special offers such as discounts and campaigns offered by restaurants.

[0203] "Menu Information" refers to the list of food and drinks served at a restaurant and their detailed information.

[0204] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts specific information and features from it.

[0205] "Product information" refers to detailed information about ingredients and related products used in a dish.

[0206] A "menu" refers to the menu, composition, and combination of meals, and is a list of dishes suggested based on the user's conditions.

[0207] A "recipe" is a set of instructions that describes how to cook a dish, the steps involved, and the amounts of ingredients to use.

[0208] "Calorie information" refers to information that indicates the amount of energy contained in food or dishes, and is used for nutritional management and maintaining health.

[0209] This invention is a food delivery system that automatically suggests meal plans based on user-specified criteria and displays related advertisements. The system collects menu options entered by the user and generates menus based on those criteria using language generation AI. It uses the user's current location information to collect special offers and menu information from nearby restaurants, providing efficiently optimized options. It also provides detailed recipes and calorie information for the menus selected by the user.

[0210] Hardware and software used

[0211] Hardware: Smartphone (iOS, Android)

[0212] Software: Food delivery app, cloud server, location information API, image analysis AI, language generation AI

[0213] Program processing explanation

[0214] User operations

[0215] Users open a food delivery app and input their menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.).

[0216] Server Operations

[0217] The server receives the menu requirements sent by the user, formats them appropriately, and sends them to the language generation AI. The language generation AI analyzes these requirements and generates an appropriate menu. For example, if the budget for a Japanese dinner is 2,000 yen and the specified ingredients are "tomatoes and chicken," the language generation AI will suggest a menu such as "teriyaki chicken, tomato salad, and miso soup."

[0218] Advertisement selection and display

[0219] The server selects relevant advertisements based on the user's input information. It obtains the user's current location information and collects special offers and menu information from nearby restaurants. The server analyzes this information using image analysis AI and extracts appropriate product information.

[0220] Sending and displaying results

[0221] The server compiles the generated menu, related advertisements, special offers, and menu information and sends it to the user's device. The user's device receives this information and displays it on the food delivery app. The user can then view the menu suggestions, related advertisements, and special offers.

[0222] Specific examples

[0223] Example: Smartphone food delivery app

[0224] Below is an example of a specific prompt sentence that suggests the optimal menu when the user enters "Japanese food," "dinner," "budget 2,000 yen," and "tomato, chicken."

[0225] Prompt Sentence Examples

[0226] Suggest the best menu for a user when they input "Japanese food," "dinner," "budget 2000 yen," "tomato, chicken." Display the suggested menu with detailed recipes and calorie information.

[0227] The present invention is expected to enable users to efficiently receive optimal menu suggestions and take advantage of special offer information from nearby restaurants, thereby improving their food delivery experience.

[0228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0229] Step 1:

[0230] The user opens the food delivery app and inputs the menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.). The input information is sent from the device to the server.

[0231] Step 2:

[0232] The server receives the menu requirements sent by the user. It formats the received information into an appropriate format and sends it to the language generation AI. At this time, the input data includes the cuisine genre, theme, budget, and ingredient information. The formatted data is generated as the output.

[0233] Step 3:

[0234] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a Japanese dinner has a budget of 2,000 yen and the specified ingredients are tomatoes and chicken, it will suggest a menu such as "Teriyaki chicken, tomato salad, and miso soup." The proposed menu is generated as the output.

[0235] Step 4:

[0236] The server selects relevant advertisements based on the user's input information. The server takes into consideration the user's cuisine genre, theme, budget, etc., when selecting advertisements. It also obtains the user's current location information and collects special offers and menu information from nearby restaurants. The input data includes the user's current location information, and the output generates relevant advertisements and special offer information from restaurants.

[0237] Step 5:

[0238] The server uses image analysis AI to analyze the collected restaurant special offer information. Through the analysis, restaurant special offer information and new menu details are extracted. The input data includes restaurant flyers and offer images, and the analyzed special offer information is generated as output.

[0239] Step 6:

[0240] The server aggregates the generated menu, related advertisements, special offers, and menu information and sends it to the user's device, allowing the user to view this information in a single interface. The input data includes the generated menu information, related advertisements, and special offer information, and the aggregated information is sent to the user's device as output.

[0241] Step 7:

[0242] The device displays the received information. The user can view and use menu suggestions, advertisements, special offers, and menu information in the application. Output includes information displayed in the application.

[0243] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0244] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining it with an emotion engine that recognizes the user's emotions. Furthermore, it displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the ingredients they need.

[0245] Program processing explanation

[0246] User input of information

[0247] 1. User Action:

[0248] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[0249] Server processing of input information

[0250] 2. Server processing:

[0251] The server receives and formats the menu requirements sent by the user.

[0252] 3. Server processing:

[0253] The server then sends the formatted information to a language generation AI to generate an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the system will suggest a menu such as "grilled salmon, boiled broccoli, and brown rice."

[0254] Using the Emotion Engine

[0255] 4. Device operation:

[0256] The device recognizes the user's current emotion using an emotion engine, which analyzes the user's facial expressions, voice tone, and input text.

[0257] 5. Server Processing:

[0258] The server receives emotional data from the emotion engine and adjusts the menu suggestions accordingly. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0259] Advertisement selection and display

[0260] 6. Server Processing:

[0261] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user desires relaxation, advertisements for products related to relaxation will be displayed.

[0262] 7. Server Processing:

[0263] The server combines the generated menu with the selected advertisements and transmits them to the user's terminal.

[0264] 8. Terminal operation:

[0265] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0266] Implementing the extension

[0267] 9. User Actions:

[0268] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0269] 10. Server Processing:

[0270] The server collects flyer information for nearby commercial facilities based on current location information.

[0271] 11. Server Processing:

[0272] The server uses image analysis AI to analyze the flyer information and extract appropriate product information.

[0273] 12. Server Processing:

[0274] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0275] 13. Server Processing:

[0276] The server sends the newly generated recommendations to the user's device.

[0277] 14. Terminal Operation:

[0278] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0279] Detailed recipe provided

[0280] 15. Server Processing:

[0281] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0282] 16. Terminal Operation:

[0283] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0284] Specific examples

[0285] Example 1: Healthy Japanese menu and use of emotion engine

[0286] 1. User input:

[0287] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[0288] 2. Server processing:

[0289] The server sends this information to a language generation AI to generate a menu, suggesting, for example, "grilled salmon, boiled broccoli, and brown rice."

[0290] 3. Use of Emotion Engine:

[0291] The device recognizes the user's emotions and transmits them to the server. For example, if the device detects that the user is seeking relaxation, the server will select advertisements related to relaxation.

[0292] 4. Advertisement Selection:

[0293] The server sends the selected advertisements and menus to the user's terminal.

[0294] 5. Displaying the results:

[0295] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[0296] By utilizing this system, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals. Furthermore, by combining it with an emotion engine, it becomes possible to make more personalized suggestions based on the user's emotions.

[0297] The processing flow will be explained below.

[0298] Step 1:

[0299] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[0300] Step 2:

[0301] The user clicks the "Submit" button to send the entered menu requirements to the server.

[0302] Step 3:

[0303] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[0304] Step 4:

[0305] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[0306] Step 5:

[0307] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[0308] Step 6:

[0309] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[0310] Step 7:

[0311] The device activates an emotion engine to recognize the user's current emotion, which analyzes emotions from the user's facial expressions, voice tone, and input text.

[0312] Step 8:

[0313] The device sends emotional data analyzed by the emotion engine to the server. For example, the user's emotion is recognized as "stress."

[0314] Step 9:

[0315] The server receives the emotion data and adjusts the menu suggestions accordingly, for example adding ingredients and recipes that have a relaxing effect for a user who is feeling stressed.

[0316] Step 10:

[0317] The server then tailors the selected advertisements based on the emotional data, for example, selecting advertisements for products or services that are expected to have a relaxing effect.

[0318] Step 11:

[0319] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[0320] Step 12:

[0321] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0322] Step 13:

[0323] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0324] Step 14:

[0325] The server uses the current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[0326] Step 15:

[0327] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[0328] Step 16:

[0329] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0330] Step 17:

[0331] The server sends the newly generated recommendations to the user's device.

[0332] Step 18:

[0333] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0334] Step 19:

[0335] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0336] Step 20:

[0337] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0338] By following the above steps, users can easily decide on their daily menu, and by using the emotion engine, it is possible to propose more personalized menus based on the user's emotions.In addition, by utilizing advertisements and flyer information, users can optimally select the ingredients they need and obtain information on deals.

[0339] Example 2

[0340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] Conventional menu suggestion systems lack personalized suggestions that take into account the user's emotions and information about nearby commercial facilities. Furthermore, they are unable to reflect the user's current emotional state when providing detailed recipe links or selecting related advertisements, resulting in a suboptimal user experience.

[0342] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0343] In this invention, the server includes: means for collecting menu conditions input by the user; means for generating a menu using a generative AI model; means for using an emotion engine to recognize the user's emotions; means for adjusting the menu based on the user's emotion data; means for selecting related advertisements; means for transmitting the generated menu and advertisements to the user's terminal; means for acquiring the user's current location information and collecting information on nearby commercial facilities; and means for generating product information that analyzes the collected information using image analysis AI and reflects it in the menu. This enables more personalized menu suggestions and advertisement selection based on the user's emotions and current location.

[0344] The "means for collecting menu conditions" is an element of the system that collects information about the menu entered by the user, such as the food genre, theme, type of menu, and ingredients desired to be used, through a digital form or the like.

[0345] A "generative AI model" is a type of artificial intelligence that uses natural language processing technology to generate appropriate menus based on conditions provided by the user.

[0346] The "means for using an emotion engine" is a component of the system that has the function of recognizing emotions by analyzing the user's emotional state from facial expressions, voice tone, and input text.

[0347] The "menu adjustment means" is a system element that has the function of adjusting the generated menu based on the user's emotional data and changing it to a form that suits the user's emotions.

[0348] The "means for selecting advertisements" refers to a system element for selecting and displaying highly relevant advertisements based on user input information and emotional data.

[0349] The "transmission means" is a system element that has the function of transmitting the generated menu and selected advertisements in digital form to the user's terminal.

[0350] The "means for collecting information" is an element of the system that has the function of acquiring information about the user's current location and using that information to collect flyer information for nearby commercial facilities, etc.

[0351] The "means for generating product information" is an element of the system that has the function of analyzing collected flyer information using image analysis AI and generating product information related to the menu based on the results.

[0352] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining them with an emotion engine that recognizes the user's emotions. It also displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information.

[0353] System Overview

[0354] This system includes a user device, a server, a generative AI model, an emotion engine, and an image analysis AI. Each component and its function are explained below.

[0355] Collecting user input information

[0356] Users access the application using a device such as a smartphone or PC. The application provides a form for entering the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple dish, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). The device collects this information and sends it to the server.

[0357] Menu generation

[0358] The server receives the information sent by the user and organizes the data. The organized information is then sent to a generative AI model (e.g., GPT-4). The generative AI model creates a prompt based on the information provided and generates an appropriate menu. For example, a prompt might be, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." Based on this prompt, the generative AI model generates a menu such as "grilled salmon, boiled broccoli, and brown rice."

[0359] Using the Emotion Engine

[0360] The user's device uses an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's facial expressions, voice tone, and input text to identify the user's emotions. For example, it may use Microsoft Azure's Emotion API. The analyzed emotion data is sent to the server. The server adjusts the generated menu based on this emotion data. For example, if the user is feeling stressed, it may change the recipe to one that has a relaxing effect.

[0361] Advertisement selection and display

[0362] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for relaxation-related products. The selected advertisements and generated menus are sent from the server to the user's device, where they are displayed.

[0363] Collection and analysis of flyer information

[0364] When a user uses the extension, the device sends the user's current location information to the server. The server uses the current location information to collect flyers from nearby commercial facilities and analyzes the flyer information using image analysis AI (e.g., Google Cloud Vision API). Based on the analysis results, recommendations including information on great deals are generated and reflected in the user's menu suggestions.

[0365] Specific examples

[0366] If a user inputs the conditions "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice," the system sends a prompt to the generative AI model: "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." The generative AI model generates a menu of "grilled salmon with salt, broccoli in ohitashi sauce, and brown rice" and responds to the server. If the emotion engine's analysis determines that the user is seeking relaxation, the server selects relaxation-related advertisements and sends them to the user's device. The user's device displays this information, and the user can view the proposed menu and related advertisements.

[0367] By using this system, users can easily plan their daily menu and receive personalized suggestions that take into account their emotions and special offers.

[0368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0369] Step 1:

[0370] The user opens the application and enters the cooking genre, theme, type of menu, and ingredients they want to use in the provided form. Specifically, the user enters "Japanese cuisine," "healthy," "staple food / main dish / side dish," and "salmon, broccoli, brown rice." This generates input data on the user's device.

[0371] Step 2:

[0372] When the device receives user input information, it sends it to the server. The server then formats the received data in a way that makes it easier to format. Specifically, the input data is organized by converting each input item into a specific data structure (e.g., JSON format).

[0373] Step 3:

[0374] Based on the prepared information, the server generates a prompt to send to the generative AI model. For example, it generates a prompt such as, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." This generates a prompt.

[0375] Step 4:

[0376] The server sends the generated prompt to the generative AI model to obtain an appropriate menu. The generative AI model (e.g., GPT-4) analyzes the prompt and generates a menu that matches the requested conditions. For example, a menu such as "grilled salted salmon, boiled broccoli, and brown rice" is generated. This generates the menu data.

[0377] Step 5:

[0378] The device recognizes the user's current emotion using an emotion engine (e.g., Microsoft Azure's Emotion API) that analyzes emotions from the user's facial expressions, voice tone, and input text, generating the user's emotion data.

[0379] Step 6:

[0380] The server receives emotional data from the device and adjusts the menu suggestions based on the analysis results. Specifically, it changes ingredients and recipes to have a relaxing effect based on the user's emotional state (e.g., stress). This results in adjusted menu data.

[0381] Step 7:

[0382] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user is looking to relax, the server selects advertisements related to relaxation goods. This generates selected advertisement data.

[0383] Step 8:

[0384] The server combines the generated menu with the selected advertisement and sends it to the user's terminal. The server then compiles the data, generates a response, and sends it to the terminal. This sends the menu and advertisement data to the user's terminal.

[0385] Step 9:

[0386] The device displays the response received from the server. Specifically, it displays menu information and related advertisements on the screen. The user can then check the suggested menu and advertisements. This provides the user with the information they need.

[0387] Step 10:

[0388] When a user uses the extended function, the terminal transmits the current location information to the server, which transmits the current location data to the server.

[0389] Step 11:

[0390] The server collects flyer information from nearby commercial facilities based on the current location information. For example, it obtains supermarket flyer data using an online API. This allows the collected flyer data to be obtained.

[0391] Step 12:

[0392] The server analyzes the flyer information using image analysis AI. Specifically, it uses image analysis AI (e.g., Google Cloud Vision API) to extract product information from the flyer data. This provides the analyzed product information.

[0393] Step 13:

[0394] Based on the analysis results, the server generates recommendations that reflect additional information in the menu suggestions. Specifically, it generates menu suggestions that include information on special sales and deals. This generates recommendation data.

[0395] Step 14:

[0396] The server sends the newly generated recommendations to the user's device, which then sends the recommendation data to the user's device.

[0397] Step 15:

[0398] The device displays new recommendations, and users can check the suggested menu along with the optimal product information, allowing them to efficiently select ingredients by taking advantage of the discount information.

[0399] Step 16:

[0400] The server generates a link to a detailed recipe related to the proposed menu and sends it to the user's terminal, thereby generating a detailed recipe link.

[0401] Step 17:

[0402] When the user clicks on the link, the device will be redirected to a recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] In today's busy society, it is difficult to efficiently plan menus and optimally select the necessary ingredients. Furthermore, there are few systems that can provide personalized suggestions based on the user's emotions and moods. Furthermore, there is a lack of systems for effectively purchasing related products and mechanisms that integrate advertising displays and electronic payments. The purpose of this invention is to solve these problems.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for collecting menu conditions entered by the user, a means for generating menus using a language generation AI, and a means for selecting related advertisements based on the information entered by the user. This enables efficient and personalized menu suggestions and related product purchases.

[0407] The server also includes a means for analyzing the user's facial expressions and voice tone to acquire emotional data, a means for adjusting the menu based on the emotional data, and a means for allowing the user to purchase suggested products in cooperation with an electronic payment service, thereby enabling more personalized suggestions based on the user's emotions and enabling related products to be purchased immediately.

[0408] "Menu conditions" refers to information such as an outline or theme of the dish desired by the user, ingredients that the user wants to use, and the like.

[0409] "Language generation AI" is an artificial intelligence technology that generates sentences and data based on input information.

[0410] "Advertisement" refers to the promotion of related products selected based on the user's interests and input information.

[0411] "Emotion data" is emotional information analyzed from the user's facial expressions and voice tone.

[0412] "Adjusting the menu" means changing the contents of the menu to be suggested based on the acquired emotional data.

[0413] An "electronic payment service" is a service that allows for online monetary transactions using the Internet.

[0414] "Flyer information" is product advertisement information collected from nearby commercial facilities.

[0415] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts information.

[0416] "Product information" refers to detailed information about individual products extracted from flyer information using image analysis AI.

[0417] The present invention is a system that automatically proposes meal menus based on user-entered criteria and further adjusts the proposals by recognizing the user's emotions. Specifically, the system configuration and operation are as follows:

[0418] Overall system configuration

[0419] 1. User operations

[0420] First, users open the smartphone app and input the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[0421] 2. Menu generation

[0422] The server receives the menu requirements sent by the user and formats them. This is then sent to a language generation AI (e.g., GPT-4) to generate an appropriate menu. For example, if a user is health-conscious and prefers Japanese cuisine, and the specified ingredients are salmon, broccoli, and brown rice, the generation AI will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0423] 3. Emotion recognition

[0424] Using the smartphone's camera and microphone, the user's facial expressions and vocal tone are analyzed by an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to a server.

[0425] 4. Menu adjustment

[0426] The server then adjusts the menu based on the user's emotional data. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0427] 5. Advertisement selection and display

[0428] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for related products. The server then sends the generated menu and the selected advertisements to the user's smartphone for display.

[0429] 6. Electronic payment integration

[0430] The server connects to electronic payment services (e.g., PayPal, Stripe) so that users can instantly purchase the suggested products (ingredients, cooking utensils, etc.) within the app. Users can easily complete the purchase procedure within the app.

[0431] Specific examples

[0432] Example 1: Healthy Japanese menu and use of emotion engine

[0433] 1. The user inputs "Japanese food," "healthy food," "staple food / main dish / side dish," and ingredients "salmon, broccoli, brown rice."

[0434] 2. The server sends this information to a language generation AI to generate a menu. For example, it suggests "grilled salmon, boiled broccoli, and brown rice."

[0435] 3. Emotion recognition: It is analyzed that the user is seeking relaxation.

[0436] 4. Suggest relaxing recipes and display related advertisements (relaxing products, herbal tea, etc.).

[0437] 5. Make ingredients and products available for purchase via electronic payment services.

[0438] Example prompt sentence:

[0439] "The user wants to relax and enjoy healthy Japanese food, with salmon, broccoli, and brown rice. Please suggest a menu based on this."

[0440] This allows users to easily find efficient and personalized menus and even purchase related products directly.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] User operations

[0444] The user opens the smartphone app and inputs the menu requirements, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), type of menu (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[0445] Input: Menu conditions, theme, ingredients

[0446] Output: The input data is sent to the app's server.

[0447] Step 2:

[0448] Server Processing

[0449] The server receives the input menu requirements and formats this information. The formatted data is sent to a language generation AI model (e.g., GPT-4). For example, it might be formatted as "Japanese food for health-conscious people, with salmon, broccoli, and brown rice."

[0450] Input: User's menu requirements, theme, ingredients

[0451] Output: Formatted prompt text

[0452] Step 3:

[0453] Processing of generated AI

[0454] The server sends prompts to the language generation AI model to generate an appropriate menu. For example, a prompt such as "Healthy Japanese food, with salmon, broccoli, and brown rice" is sent, and the AI ​​generates a menu based on this prompt, such as "Grilled salmon, boiled broccoli, and brown rice."

[0455] Input: Formatted prompt text

[0456] Output: Generated menu data

[0457] Step 4:

[0458] Performing emotion recognition

[0459] The user's device uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to the server.

[0460] Input: User's facial expressions and voice tones

[0461] Output: Emotion data

[0462] Step 5:

[0463] Menu adjustment based on emotions

[0464] The server receives the emotion data and adjusts the generated menu accordingly: for example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0465] Input: Emotion data, generated menu data

[0466] Output: Adjusted menu data

[0467] Step 6:

[0468] Ad selection and delivery

[0469] The server selects relevant advertisements based on the user's input information and emotional data. For example, advertisements for relaxation-related products are selected for a user seeking relaxation. The generated menu and the selected advertisements are sent to the user's device.

[0470] Input: User input, emotion data, adjusted menu data

[0471] Output: Selected ads

[0472] Step 7:

[0473] Displaying the final result

[0474] The user's device receives the response from the server and displays the adjusted menu and related advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0475] Input: Response data from the server

[0476] Output: Menu and advertisements displayed on the screen

[0477] Step 8:

[0478] Making electronic payments

[0479] If the user wishes to purchase the suggested products (ingredients, cooking utensils, etc.), the purchase process is carried out in cooperation with an electronic payment service (e.g., PayPal, Stripe). After the purchase process is completed, the purchase information is sent to the server.

[0480] Input: User's purchase request, electronic payment platform

[0481] Output: Purchase completion confirmation data

[0482] This allows users to receive personalized menu suggestions and immediately purchase related products.

[0483] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0484] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0485] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0486] [Second embodiment]

[0487] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0488] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0489] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0490] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0491] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0492] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0493] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0494] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0495] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0496] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0497] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0498] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0499] This invention is a system that can automatically propose meal menus based on user-specified criteria and display related advertisements. Specifically, it collects menu conditions entered by the user and generates menus based on those conditions using language generation AI. Furthermore, it uses the user's current location information to collect flyers from nearby commercial facilities and analyzes them using image analysis AI to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the necessary ingredients.

[0500] Program processing explanation

[0501] User input of information

[0502] 1. User Action:

[0503] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[0504] Server processing of input information

[0505] 2. Server processing:

[0506] The server receives the menu requirements sent by the user, formats the received information appropriately, and passes it to the language generation AI.

[0507] Menu generation using language generation AI

[0508] 3. Server Operation:

[0509] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the AI ​​will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0510] Advertisement selection and display

[0511] 4. Server Operation:

[0512] The server selects relevant advertisements based on the user's input information. For example, it selects advertisements for health foods for health-conscious menus, and advertisements for fast food for hearty menus.

[0513] 5. Server Operation:

[0514] The server combines the generated menu with the selected advertisement and transmits it to the user's terminal.

[0515] 6. Device Operation:

[0516] The user's terminal receives the response from the server and displays the menu and advertisements on the screen.

[0517] Implementing the extension

[0518] 7. User Actions:

[0519] When a user wants to use the extended function, the user transmits current location information from the terminal to the server.

[0520] 8. Server Operation:

[0521] The server collects flyer information for nearby commercial facilities based on current location information.

[0522] 9. Server Operation:

[0523] The server uses image analysis AI to analyze flyer information and extract appropriate product information, reflecting the results in the menu and providing recommendations.

[0524] Detailed recipe provided

[0525] 10. Server Operation:

[0526] The server generates a link to the detailed recipe associated with the suggested meal.

[0527] 11. Terminal Operation:

[0528] When the user clicks on the link, the device will transition to the recipe site to display the detailed recipe.

[0529] Specific examples

[0530] Example 1: Healthy Japanese menu

[0531] 1. User input:

[0532] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[0533] 2. Server processing:

[0534] The server sends this information to the language generation AI.

[0535] 3. Results of language generation AI:

[0536] The language generation AI suggests "grilled salted salmon, boiled broccoli, and brown rice."

[0537] 4. Advertisement Selection:

[0538] The server selects advertisements for health foods and beauty salons and sends them to the user's terminal.

[0539] 5. Displaying the results:

[0540] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[0541] 6. Use of extensions:

[0542] When a user submits their current location information, the server collects flyer information from nearby commercial facilities and makes additional suggestions based on the analysis results.

[0543] This allows users to easily plan their daily menu, and by utilizing advertising and flyer information, they can also obtain information on how to shop at bargain prices.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[0547] Step 2:

[0548] The user clicks the "Submit" button to send the entered menu requirements to the server.

[0549] Step 3:

[0550] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[0551] Step 4:

[0552] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[0553] Step 5:

[0554] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[0555] Step 6:

[0556] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[0557] Step 7:

[0558] The server selects relevant advertisements based on the information entered by the user. For example, advertisements for health foods and beauty salons are selected for health-conscious criteria, and advertisements for fast food are selected for hearty meals.

[0559] Step 8:

[0560] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[0561] Step 9:

[0562] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0563] Step 10:

[0564] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0565] Step 11:

[0566] The server uses the user's current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[0567] Step 12:

[0568] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[0569] Step 13:

[0570] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0571] Step 14:

[0572] The server sends the newly generated recommendations to the user's device.

[0573] Step 15:

[0574] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0575] Step 16:

[0576] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0577] Step 17:

[0578] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0579] By following the above steps, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals.

[0580] Example 1

[0581] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] The present invention aims to provide a system that allows users to efficiently decide on a menu and obtain related information. Specifically, the system automatically generates an appropriate menu based on various menu conditions specified by the user, collects sales information from nearby commercial facilities using the user's current location information, and uses image analysis technology to suggest optimal product information, thereby enabling users to easily plan their daily meals.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0584] In this invention, the server includes means for collecting menu conditions input by the user, means for generating a menu using a generative AI model, means for selecting relevant advertisements based on the information input by the user, means for transmitting the generated menu and selected advertisements to the user's terminal, means for transmitting current location information operated by the user, means for collecting sales information of nearby commercial facilities based on the current location information, means for analyzing the sales information using image analysis technology, and means for recommending product information based on the analyzed sales information. This allows users to obtain menus automatically generated based on a variety of menu conditions, and further enables them to obtain optimal product information by utilizing sales information based on their current location.

[0585] The "means for collecting menu conditions" is a function that collects information such as the cooking genre, theme, type of menu, and ingredients desired to be used that the user inputs within the application.

[0586] "Means for generating menus using a generative AI model" is a function that uses artificial intelligence to automatically generate appropriate menus based on collected menu conditions.

[0587] The "means for selecting advertisements" is a function by which the server searches and selects relevant advertisements based on the information entered by the user and the generated menu.

[0588] "Means for transmitting menu and selected advertisement to user's terminal" is a function in which the server compiles the generated menu information and selected advertisement information and transmits them as data to the user's terminal.

[0589] The "means for transmitting current location information" is a function that allows the user to obtain current location information through an operation and transmit it to the server.

[0590] The "means for collecting sales information" is a function that allows the server to obtain sales information from nearby commercial facilities based on the user's current location information.

[0591] The "means for analyzing sales information using image analysis technology" is a function for analyzing collected sales information using image analysis technology and extracting useful information.

[0592] "Means for recommending product information" is a function that suggests optimal product information to users based on analyzed sales information.

[0593] The present invention is a system that automatically proposes meal plans based on user-specified criteria and displays related advertisements. Specifically, it collects menu plan conditions entered by the user and generates menu plans based on those conditions using a generative AI model. It also utilizes the user's current location information to collect sales information from nearby commercial facilities and analyzes it using image analysis technology to provide optimal product information. A specific embodiment of this system is described below.

[0594] User input of information

[0595] First, the user launches the application and inputs the menu requirements. Specifically, the user enters information into a form, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). This input information is collected on the device and then sent to the server.

[0596] Processing of input information and menu generation by the server

[0597] The server receives the menu requirements sent by the user and converts them into an appropriate format. It then passes the requirements to a generative AI model (such as OpenAI's GPT-4) to generate an appropriate menu. The generative AI model analyzes the input requirements and suggests an appropriate menu. For example, given the requirements of "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and ingredients of "salmon, broccoli, brown rice," it would generate a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0598] Ad selection and display

[0599] Based on the generated menu, the server searches and selects relevant advertisements. For example, for a health-conscious menu, advertisements for healthy foods are selected. These advertisements are sent to the user's terminal together with the generated menu and displayed on the terminal.

[0600] Utilizing current location information and collecting sales information

[0601] Furthermore, if the user provides their current location information, the device sends that information to a server. Based on the current location information, the server collects sales information from nearby commercial facilities via APIs and other means. The collected sales information is analyzed using image analysis technology (for example, Google Cloud Vision API) to extract useful product information.

[0602] Product recommendations and detailed recipes

[0603] The extracted product information is recommended to the user and reflected in the menu. The server also generates a link to a detailed recipe related to the suggested menu and sends it to the terminal in a format that the user can use. When the user clicks the link, the terminal is redirected to a detailed recipe site where the detailed recipe is displayed.

[0604] Specific examples

[0605] For example, if a user inputs "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients, the generative AI model will suggest "grilled salmon with salt, boiled broccoli, and brown rice." The server will then use image analysis technology to analyze health food advertisements and related product information sold at nearby commercial facilities, and make additional suggestions based on the results. This allows users to easily decide on a menu and obtain the most appropriate information when shopping.

[0606] Prompt Sentence Examples

[0607] "Please suggest a menu for tonight's dinner. I'd like to use salmon, broccoli, and brown rice as healthy ingredients for a Japanese meal."

[0608] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0609] Step 1: The user launches the application and enters menu requirements.

[0610] Specific operation: The user moves from the application's home screen to the menu condition input form. The form has input fields for cooking genre, theme, menu type, and desired ingredients. The user enters the conditions in these fields and presses the submit button.

[0611] Input: cuisine genre, theme, type of menu, ingredients you want to use

[0612] Output: Menu conditions entered by the user

[0613] Step 2: The terminal sends the input data from the user to the server.

[0614] What happens: The device collects the data entered by the user and sends it to the server as an HTTP request. The data is encoded in JSON format.

[0615] Input: Menu conditions entered by the user

[0616] Output: Menu conditions sent to the server

[0617] Step 3: The server receives the menu requirements sent by the user and converts them into an appropriate format.

[0618] What it does: The server parses the received data and formats it for internal processing, in a format that is easy for the generative AI model to understand (e.g., JSON).

[0619] Input: Menu conditions received from the terminal

[0620] Output: Formatted menu items

[0621] Step 4: The server sends the formatted menu requirements to the generative AI model.

[0622] Specific operation: The server sends the formatted data as a request to the API of the generative AI model (e.g., OpenAI's GPT-4).

[0623] Input: Formatted menu requirements

[0624] Output: The request sent to the generative AI model

[0625] Step 5: The generative AI model analyzes the conditions and generates the optimal menu.

[0626] Specific operation: The generative AI model analyzes the received conditions and generates an appropriate menu that meets the conditions. The result is sent back to the server in JSON format.

[0627] Input: The request sent to the generative AI model

[0628] Output: Generated menu

[0629] Step 6: The server receives the generated menu and selects relevant advertisements.

[0630] Specific operation: The server searches the advertisement database based on the generated menu information and selects relevant advertisements. For example, for a health-conscious menu, it selects advertisements for health foods.

[0631] Input: Generated menu

[0632] Output: Selected ads

[0633] Step 7: The server integrates the generated menu with the selected advertisements and sends it to the user's terminal.

[0634] Specific operation: The server combines the generated menu data and the selected advertising data into a single JSON object and sends it to the user's device as an HTTP response.

[0635] Input: Generated menu and selected advertisement

[0636] Output: The response sent to the user's device

[0637] Step 8: The user's device receives the response from the server and displays the menu and advertisements on the screen.

[0638] Specific operation: The device analyzes the received response data and displays menu information and advertising information in the appropriate position on the screen.

[0639] Input: Response from the server

[0640] Output: Menu and advertisements displayed on the screen

[0641] Step 9: If the user provides current location information, the terminal sends the information to the server.

[0642] Specific operation: The user presses the "Send current location" button within the application, and the device obtains GPS data and sends it to the server as an HTTP request.

[0643] Input: User's current location information

[0644] Output: Current location information sent to the server

[0645] Step 10: The server collects sales information from nearby commercial facilities based on the current location information.

[0646] Specific operation: The server uses the collected location information to obtain sales information from the commercial facility's API or database.

[0647] Input: User's current location information

[0648] Output: Retrieved sales information

[0649] Step 11: The server analyzes the collected sales information using image analysis technology.

[0650] Specific operation: The server analyzes the collected sales information using image analysis technology (e.g., Google Cloud Vision API) and extracts useful product information.

[0651] Input: Collected sales information

[0652] Output: Parsed product information

[0653] Step 12: Based on the analyzed product information, the server recommends the most suitable product information to the user.

[0654] Specific operation: Based on the analyzed product information, the server generates recommendations for the user and sends them to the user's device.

[0655] Input: Parsed product information

[0656] Output: Recommendations sent to the user's device

[0657] (Application example 1)

[0658] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0659] Conventional food delivery applications require users to select the menu themselves, and they lack the ability to provide menu suggestions and related information. Furthermore, there is a need for applications that utilize the user's current location to collect special offers and menu information from nearby restaurants and efficiently present optimized options.

[0660] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0661] In this invention, the server includes means for collecting menu requirements entered by the user, means for generating a menu using language generation AI, means for selecting relevant advertisements based on the information entered by the user, means for acquiring the user's current location information and collecting special offers and menu information from nearby restaurants, and means for transmitting the generated menu and selected advertisements to the user's terminal. This allows the user to efficiently receive optimal menu suggestions and conveniently take advantage of special offer information from nearby restaurants.

[0662] "Menu conditions" refers to information necessary when deciding on a menu, such as the type of cuisine the user desires, the theme, the budget, and the ingredients they wish to use.

[0663] "Language generation AI" is artificial intelligence that uses natural language processing technology to generate text based on specified conditions.

[0664] "Advertisement" means information or messages provided to users for the purpose of promoting a particular product or service.

[0665] "Current location information" refers to location information obtained from a user terminal and is used to identify the geographic location of the user.

[0666] "Restaurant" refers to a facility or store that serves food and drinks.

[0667] "Special offers" refers to special offers such as discounts and campaigns offered by restaurants.

[0668] "Menu Information" refers to the list of food and drinks served at a restaurant and their detailed information.

[0669] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts specific information and features from it.

[0670] "Product information" refers to detailed information about ingredients and related products used in a dish.

[0671] A "menu" refers to the menu, composition, and combination of meals, and is a list of dishes suggested based on the user's conditions.

[0672] A "recipe" is a set of instructions that describes how to cook a dish, the steps involved, and the amounts of ingredients to use.

[0673] "Calorie information" refers to information that indicates the amount of energy contained in food or dishes, and is used for nutritional management and maintaining health.

[0674] This invention is a food delivery system that automatically suggests meal plans based on user-specified criteria and displays related advertisements. The system collects menu options entered by the user and generates menus based on those criteria using language generation AI. It uses the user's current location information to collect special offers and menu information from nearby restaurants, providing efficiently optimized options. It also provides detailed recipes and calorie information for the menus selected by the user.

[0675] Hardware and software used

[0676] Hardware: Smartphone (iOS, Android)

[0677] Software: Food delivery app, cloud server, location information API, image analysis AI, language generation AI

[0678] Program processing explanation

[0679] User operations

[0680] Users open a food delivery app and input their menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.).

[0681] Server Operations

[0682] The server receives the menu requirements sent by the user, formats them appropriately, and sends them to the language generation AI. The language generation AI analyzes these requirements and generates an appropriate menu. For example, if the budget for a Japanese dinner is 2,000 yen and the specified ingredients are "tomatoes and chicken," the language generation AI will suggest a menu such as "teriyaki chicken, tomato salad, and miso soup."

[0683] Advertisement selection and display

[0684] The server selects relevant advertisements based on the user's input information. It obtains the user's current location information and collects special offers and menu information from nearby restaurants. The server analyzes this information using image analysis AI and extracts appropriate product information.

[0685] Sending and displaying results

[0686] The server compiles the generated menu, related advertisements, special offers, and menu information and sends it to the user's device. The user's device receives this information and displays it on the food delivery app. The user can then view the menu suggestions, related advertisements, and special offers.

[0687] Specific examples

[0688] Example: Smartphone food delivery app

[0689] Below is an example of a specific prompt sentence that suggests the optimal menu when the user enters "Japanese food," "dinner," "budget 2,000 yen," and "tomato, chicken."

[0690] Prompt Sentence Examples

[0691] Suggest the best menu for a user when they input "Japanese food," "dinner," "budget 2000 yen," "tomato, chicken." Display the suggested menu with detailed recipes and calorie information.

[0692] The present invention is expected to enable users to efficiently receive optimal menu suggestions and take advantage of special offer information from nearby restaurants, thereby improving their food delivery experience.

[0693] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0694] Step 1:

[0695] The user opens the food delivery app and inputs the menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.). The input information is sent from the device to the server.

[0696] Step 2:

[0697] The server receives the menu requirements sent by the user. It formats the received information into an appropriate format and sends it to the language generation AI. At this time, the input data includes the cuisine genre, theme, budget, and ingredient information. The formatted data is generated as the output.

[0698] Step 3:

[0699] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a Japanese dinner has a budget of 2,000 yen and the specified ingredients are tomatoes and chicken, it will suggest a menu such as "Teriyaki chicken, tomato salad, and miso soup." The proposed menu is generated as the output.

[0700] Step 4:

[0701] The server selects relevant advertisements based on the user's input information. The server takes into consideration the user's cuisine genre, theme, budget, etc., when selecting advertisements. It also obtains the user's current location information and collects special offers and menu information from nearby restaurants. The input data includes the user's current location information, and the output generates relevant advertisements and special offer information from restaurants.

[0702] Step 5:

[0703] The server uses image analysis AI to analyze the collected restaurant special offer information. Through the analysis, restaurant special offer information and new menu details are extracted. The input data includes restaurant flyers and offer images, and the analyzed special offer information is generated as output.

[0704] Step 6:

[0705] The server aggregates the generated menu, related advertisements, special offers, and menu information and sends it to the user's device, allowing the user to view this information in a single interface. The input data includes the generated menu information, related advertisements, and special offer information, and the aggregated information is sent to the user's device as output.

[0706] Step 7:

[0707] The device displays the received information. The user can view and use menu suggestions, advertisements, special offers, and menu information in the application. Output includes information displayed in the application.

[0708] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0709] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining it with an emotion engine that recognizes the user's emotions. Furthermore, it displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the ingredients they need.

[0710] Program processing explanation

[0711] User input of information

[0712] 1. User Action:

[0713] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[0714] Server processing of input information

[0715] 2. Server processing:

[0716] The server receives and formats the menu requirements sent by the user.

[0717] 3. Server processing:

[0718] The server then sends the formatted information to a language generation AI to generate an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the system will suggest a menu such as "grilled salmon, boiled broccoli, and brown rice."

[0719] Using the Emotion Engine

[0720] 4. Device operation:

[0721] The device recognizes the user's current emotion using an emotion engine, which analyzes the user's facial expressions, voice tone, and input text.

[0722] 5. Server Processing:

[0723] The server receives emotional data from the emotion engine and adjusts the menu suggestions accordingly. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0724] Advertisement selection and display

[0725] 6. Server Processing:

[0726] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user desires relaxation, advertisements for products related to relaxation will be displayed.

[0727] 7. Server Processing:

[0728] The server combines the generated menu with the selected advertisements and transmits them to the user's terminal.

[0729] 8. Terminal operation:

[0730] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0731] Implementing the extension

[0732] 9. User Actions:

[0733] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0734] 10. Server Processing:

[0735] The server collects flyer information for nearby commercial facilities based on current location information.

[0736] 11. Server Processing:

[0737] The server uses image analysis AI to analyze the flyer information and extract appropriate product information.

[0738] 12. Server Processing:

[0739] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0740] 13. Server Processing:

[0741] The server sends the newly generated recommendations to the user's device.

[0742] 14. Terminal Operation:

[0743] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0744] Detailed recipe provided

[0745] 15. Server Processing:

[0746] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0747] 16. Terminal Operation:

[0748] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0749] Specific examples

[0750] Example 1: Healthy Japanese menu and use of emotion engine

[0751] 1. User input:

[0752] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[0753] 2. Server processing:

[0754] The server sends this information to a language generation AI to generate a menu, suggesting, for example, "grilled salmon, boiled broccoli, and brown rice."

[0755] 3. Use of Emotion Engine:

[0756] The device recognizes the user's emotions and transmits them to the server. For example, if the device detects that the user is seeking relaxation, the server will select advertisements related to relaxation.

[0757] 4. Advertisement Selection:

[0758] The server sends the selected advertisements and menus to the user's terminal.

[0759] 5. Displaying the results:

[0760] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[0761] By utilizing this system, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals. Furthermore, by combining it with an emotion engine, it becomes possible to make more personalized suggestions based on the user's emotions.

[0762] The processing flow will be explained below.

[0763] Step 1:

[0764] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[0765] Step 2:

[0766] The user clicks the "Submit" button to send the entered menu requirements to the server.

[0767] Step 3:

[0768] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[0769] Step 4:

[0770] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[0771] Step 5:

[0772] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[0773] Step 6:

[0774] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[0775] Step 7:

[0776] The device activates an emotion engine to recognize the user's current emotion, which analyzes emotions from the user's facial expressions, voice tone, and input text.

[0777] Step 8:

[0778] The device sends emotional data analyzed by the emotion engine to the server. For example, the user's emotion is recognized as "stress."

[0779] Step 9:

[0780] The server receives the emotion data and adjusts the menu suggestions accordingly, for example adding ingredients and recipes that have a relaxing effect for a user who is feeling stressed.

[0781] Step 10:

[0782] The server then tailors the selected advertisements based on the emotional data, for example, selecting advertisements for products or services that are expected to have a relaxing effect.

[0783] Step 11:

[0784] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[0785] Step 12:

[0786] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0787] Step 13:

[0788] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[0789] Step 14:

[0790] The server uses the current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[0791] Step 15:

[0792] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[0793] Step 16:

[0794] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[0795] Step 17:

[0796] The server sends the newly generated recommendations to the user's device.

[0797] Step 18:

[0798] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[0799] Step 19:

[0800] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[0801] Step 20:

[0802] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0803] By following the above steps, users can easily decide on their daily menu, and by using the emotion engine, it is possible to propose more personalized menus based on the user's emotions.In addition, by utilizing advertisements and flyer information, users can optimally select the ingredients they need and obtain information on deals.

[0804] Example 2

[0805] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0806] Conventional menu suggestion systems lack personalized suggestions that take into account the user's emotions and information about nearby commercial facilities. Furthermore, they are unable to reflect the user's current emotional state when providing detailed recipe links or selecting related advertisements, resulting in a suboptimal user experience.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0808] In this invention, the server includes: means for collecting menu conditions input by the user; means for generating a menu using a generative AI model; means for using an emotion engine to recognize the user's emotions; means for adjusting the menu based on the user's emotion data; means for selecting related advertisements; means for transmitting the generated menu and advertisements to the user's terminal; means for acquiring the user's current location information and collecting information on nearby commercial facilities; and means for generating product information that analyzes the collected information using image analysis AI and reflects it in the menu. This enables more personalized menu suggestions and advertisement selection based on the user's emotions and current location.

[0809] The "means for collecting menu conditions" is an element of the system that collects information about the menu entered by the user, such as the food genre, theme, type of menu, and ingredients desired to be used, through a digital form or the like.

[0810] A "generative AI model" is a type of artificial intelligence that uses natural language processing technology to generate appropriate menus based on conditions provided by the user.

[0811] The "means for using an emotion engine" is a component of the system that has the function of recognizing emotions by analyzing the user's emotional state from facial expressions, voice tone, and input text.

[0812] The "menu adjustment means" is a system element that has the function of adjusting the generated menu based on the user's emotional data and changing it to a form that suits the user's emotions.

[0813] The "means for selecting advertisements" refers to a system element for selecting and displaying highly relevant advertisements based on user input information and emotional data.

[0814] The "transmission means" is a system element that has the function of transmitting the generated menu and selected advertisements in digital form to the user's terminal.

[0815] The "means for collecting information" is an element of the system that has the function of acquiring information about the user's current location and using that information to collect flyer information for nearby commercial facilities, etc.

[0816] The "means for generating product information" is an element of the system that has the function of analyzing collected flyer information using image analysis AI and generating product information related to the menu based on the results.

[0817] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining them with an emotion engine that recognizes the user's emotions. It also displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information.

[0818] System Overview

[0819] This system includes a user device, a server, a generative AI model, an emotion engine, and an image analysis AI. Each component and its function are explained below.

[0820] Collecting user input information

[0821] Users access the application using a device such as a smartphone or PC. The application provides a form for entering the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple dish, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). The device collects this information and sends it to the server.

[0822] Menu generation

[0823] The server receives the information sent by the user and organizes the data. The organized information is then sent to a generative AI model (e.g., GPT-4). The generative AI model creates a prompt based on the information provided and generates an appropriate menu. For example, a prompt might be, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." Based on this prompt, the generative AI model generates a menu such as "grilled salmon, boiled broccoli, and brown rice."

[0824] Using the Emotion Engine

[0825] The user's device uses an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's facial expressions, voice tone, and input text to identify the user's emotions. For example, it may use Microsoft Azure's Emotion API. The analyzed emotion data is sent to the server. The server adjusts the generated menu based on this emotion data. For example, if the user is feeling stressed, it may change the recipe to one that has a relaxing effect.

[0826] Advertisement selection and display

[0827] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for relaxation-related products. The selected advertisements and generated menus are sent from the server to the user's device, where they are displayed.

[0828] Collection and analysis of flyer information

[0829] When a user uses the extension, the device sends the user's current location information to the server. The server uses the current location information to collect flyers from nearby commercial facilities and analyzes the flyer information using image analysis AI (e.g., Google Cloud Vision API). Based on the analysis results, recommendations including information on great deals are generated and reflected in the user's menu suggestions.

[0830] Specific examples

[0831] If a user inputs the conditions "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice," the system sends a prompt to the generative AI model: "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." The generative AI model generates a menu of "grilled salmon with salt, broccoli in ohitashi sauce, and brown rice" and responds to the server. If the emotion engine's analysis determines that the user is seeking relaxation, the server selects relaxation-related advertisements and sends them to the user's device. The user's device displays this information, and the user can view the proposed menu and related advertisements.

[0832] By using this system, users can easily plan their daily menu and receive personalized suggestions that take into account their emotions and special offers.

[0833] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0834] Step 1:

[0835] The user opens the application and enters the cooking genre, theme, type of menu, and ingredients they want to use in the provided form. Specifically, the user enters "Japanese cuisine," "healthy," "staple food / main dish / side dish," and "salmon, broccoli, brown rice." This generates input data on the user's device.

[0836] Step 2:

[0837] When the device receives user input information, it sends it to the server. The server then formats the received data in a way that makes it easier to format. Specifically, the input data is organized by converting each input item into a specific data structure (e.g., JSON format).

[0838] Step 3:

[0839] Based on the prepared information, the server generates a prompt to send to the generative AI model. For example, it generates a prompt such as, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." This generates a prompt.

[0840] Step 4:

[0841] The server sends the generated prompt to the generative AI model to obtain an appropriate menu. The generative AI model (e.g., GPT-4) analyzes the prompt and generates a menu that matches the requested conditions. For example, a menu such as "grilled salted salmon, boiled broccoli, and brown rice" is generated. This generates the menu data.

[0842] Step 5:

[0843] The device recognizes the user's current emotion using an emotion engine (e.g., Microsoft Azure's Emotion API) that analyzes emotions from the user's facial expressions, voice tone, and input text, generating the user's emotion data.

[0844] Step 6:

[0845] The server receives emotional data from the device and adjusts the menu suggestions based on the analysis results. Specifically, it changes ingredients and recipes to have a relaxing effect based on the user's emotional state (e.g., stress). This results in adjusted menu data.

[0846] Step 7:

[0847] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user is looking to relax, the server selects advertisements related to relaxation goods. This generates selected advertisement data.

[0848] Step 8:

[0849] The server combines the generated menu with the selected advertisement and sends it to the user's terminal. The server then compiles the data, generates a response, and sends it to the terminal. This sends the menu and advertisement data to the user's terminal.

[0850] Step 9:

[0851] The device displays the response received from the server. Specifically, it displays menu information and related advertisements on the screen. The user can then check the suggested menu and advertisements. This provides the user with the information they need.

[0852] Step 10:

[0853] When a user uses the extended function, the terminal transmits the current location information to the server, which transmits the current location data to the server.

[0854] Step 11:

[0855] The server collects flyer information from nearby commercial facilities based on the current location information. For example, it obtains supermarket flyer data using an online API. This allows the collected flyer data to be obtained.

[0856] Step 12:

[0857] The server analyzes the flyer information using image analysis AI. Specifically, it uses image analysis AI (e.g., Google Cloud Vision API) to extract product information from the flyer data. This provides the analyzed product information.

[0858] Step 13:

[0859] Based on the analysis results, the server generates recommendations that reflect additional information in the menu suggestions. Specifically, it generates menu suggestions that include information on special sales and deals. This generates recommendation data.

[0860] Step 14:

[0861] The server sends the newly generated recommendations to the user's device, which then sends the recommendation data to the user's device.

[0862] Step 15:

[0863] The device displays new recommendations, and users can check the suggested menu along with the optimal product information, allowing them to efficiently select ingredients by taking advantage of the discount information.

[0864] Step 16:

[0865] The server generates a link to a detailed recipe related to the proposed menu and sends it to the user's terminal, thereby generating a detailed recipe link.

[0866] Step 17:

[0867] When the user clicks on the link, the device will be redirected to a recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[0868] (Application example 2)

[0869] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] In today's busy society, it is difficult to efficiently plan menus and optimally select the necessary ingredients. Furthermore, there are few systems that can provide personalized suggestions based on the user's emotions and moods. Furthermore, there is a lack of systems for effectively purchasing related products and mechanisms that integrate advertising displays and electronic payments. The purpose of this invention is to solve these problems.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for collecting menu conditions entered by the user, a means for generating menus using a language generation AI, and a means for selecting related advertisements based on the information entered by the user. This enables efficient and personalized menu suggestions and related product purchases.

[0872] The server also includes a means for analyzing the user's facial expressions and voice tone to acquire emotional data, a means for adjusting the menu based on the emotional data, and a means for allowing the user to purchase suggested products in cooperation with an electronic payment service, thereby enabling more personalized suggestions based on the user's emotions and enabling related products to be purchased immediately.

[0873] "Menu conditions" refers to information such as an outline or theme of the dish desired by the user, ingredients that the user wants to use, and the like.

[0874] "Language generation AI" is an artificial intelligence technology that generates sentences and data based on input information.

[0875] "Advertisement" refers to the promotion of related products selected based on the user's interests and input information.

[0876] "Emotion data" is emotional information analyzed from the user's facial expressions and voice tone.

[0877] "Adjusting the menu" means changing the contents of the menu to be suggested based on the acquired emotional data.

[0878] An "electronic payment service" is a service that allows for online monetary transactions using the Internet.

[0879] "Flyer information" is product advertisement information collected from nearby commercial facilities.

[0880] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts information.

[0881] "Product information" refers to detailed information about individual products extracted from flyer information using image analysis AI.

[0882] The present invention is a system that automatically proposes meal menus based on user-entered criteria and further adjusts the proposals by recognizing the user's emotions. Specifically, the system configuration and operation are as follows:

[0883] Overall system configuration

[0884] 1. User operations

[0885] First, users open the smartphone app and input the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[0886] 2. Menu generation

[0887] The server receives the menu requirements sent by the user and formats them. This is then sent to a language generation AI (e.g., GPT-4) to generate an appropriate menu. For example, if a user is health-conscious and prefers Japanese cuisine, and the specified ingredients are salmon, broccoli, and brown rice, the generation AI will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0888] 3. Emotion recognition

[0889] Using the smartphone's camera and microphone, the user's facial expressions and vocal tone are analyzed by an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to a server.

[0890] 4. Menu adjustment

[0891] The server then adjusts the menu based on the user's emotional data. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0892] 5. Advertisement selection and display

[0893] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for related products. The server then sends the generated menu and the selected advertisements to the user's smartphone for display.

[0894] 6. Electronic payment integration

[0895] The server connects to electronic payment services (e.g., PayPal, Stripe) so that users can instantly purchase the suggested products (ingredients, cooking utensils, etc.) within the app. Users can easily complete the purchase procedure within the app.

[0896] Specific examples

[0897] Example 1: Healthy Japanese menu and use of emotion engine

[0898] 1. The user inputs "Japanese food," "healthy food," "staple food / main dish / side dish," and ingredients "salmon, broccoli, brown rice."

[0899] 2. The server sends this information to a language generation AI to generate a menu. For example, it suggests "grilled salmon, boiled broccoli, and brown rice."

[0900] 3. Emotion recognition: It is analyzed that the user is seeking relaxation.

[0901] 4. Suggest relaxing recipes and display related advertisements (relaxing products, herbal tea, etc.).

[0902] 5. Make ingredients and products available for purchase via electronic payment services.

[0903] Example prompt sentence:

[0904] "The user wants to relax and enjoy healthy Japanese food, with salmon, broccoli, and brown rice. Please suggest a menu based on this."

[0905] This allows users to easily find efficient and personalized menus and even purchase related products directly.

[0906] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0907] Step 1:

[0908] User operations

[0909] The user opens the smartphone app and inputs the menu requirements, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), type of menu (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[0910] Input: Menu conditions, theme, ingredients

[0911] Output: The input data is sent to the app's server.

[0912] Step 2:

[0913] Server Processing

[0914] The server receives the input menu requirements and formats this information. The formatted data is sent to a language generation AI model (e.g., GPT-4). For example, it might be formatted as "Japanese food for health-conscious people, with salmon, broccoli, and brown rice."

[0915] Input: User's menu requirements, theme, ingredients

[0916] Output: Formatted prompt text

[0917] Step 3:

[0918] Processing of generated AI

[0919] The server sends prompts to the language generation AI model to generate an appropriate menu. For example, a prompt such as "Healthy Japanese food, with salmon, broccoli, and brown rice" is sent, and the AI ​​generates a menu based on this prompt, such as "Grilled salmon, boiled broccoli, and brown rice."

[0920] Input: Formatted prompt text

[0921] Output: Generated menu data

[0922] Step 4:

[0923] Performing emotion recognition

[0924] The user's device uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to the server.

[0925] Input: User's facial expressions and voice tones

[0926] Output: Emotion data

[0927] Step 5:

[0928] Menu adjustment based on emotions

[0929] The server receives the emotion data and adjusts the generated menu accordingly: for example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[0930] Input: Emotion data, generated menu data

[0931] Output: Adjusted menu data

[0932] Step 6:

[0933] Ad selection and delivery

[0934] The server selects relevant advertisements based on the user's input information and emotional data. For example, advertisements for relaxation-related products are selected for a user seeking relaxation. The generated menu and the selected advertisements are sent to the user's device.

[0935] Input: User input, emotion data, adjusted menu data

[0936] Output: Selected ads

[0937] Step 7:

[0938] Displaying the final result

[0939] The user's device receives the response from the server and displays the adjusted menu and related advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[0940] Input: Response data from the server

[0941] Output: Menu and advertisements displayed on the screen

[0942] Step 8:

[0943] Making electronic payments

[0944] If the user wishes to purchase the suggested products (ingredients, cooking utensils, etc.), the purchase process is carried out in cooperation with an electronic payment service (e.g., PayPal, Stripe). After the purchase process is completed, the purchase information is sent to the server.

[0945] Input: User's purchase request, electronic payment platform

[0946] Output: Purchase completion confirmation data

[0947] This allows users to receive personalized menu suggestions and immediately purchase related products.

[0948] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0951] [Third embodiment]

[0952] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0953] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0954] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0956] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0958] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0959] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0960] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0961] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0962] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0963] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0964] This invention is a system that can automatically propose meal menus based on user-specified criteria and display related advertisements. Specifically, it collects menu conditions entered by the user and generates menus based on those conditions using language generation AI. Furthermore, it uses the user's current location information to collect flyers from nearby commercial facilities and analyzes them using image analysis AI to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the necessary ingredients.

[0965] Program processing explanation

[0966] User input of information

[0967] 1. User Action:

[0968] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[0969] Server processing of input information

[0970] 2. Server processing:

[0971] The server receives the menu requirements sent by the user, formats the received information appropriately, and passes it to the language generation AI.

[0972] Menu generation using language generation AI

[0973] 3. Server Operation:

[0974] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the AI ​​will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[0975] Advertisement selection and display

[0976] 4. Server Operation:

[0977] The server selects relevant advertisements based on the user's input information. For example, it selects advertisements for health foods for health-conscious menus, and advertisements for fast food for hearty menus.

[0978] 5. Server Operation:

[0979] The server combines the generated menu with the selected advertisement and transmits it to the user's terminal.

[0980] 6. Device Operation:

[0981] The user's terminal receives the response from the server and displays the menu and advertisements on the screen.

[0982] Implementing the extension

[0983] 7. User Actions:

[0984] When a user wants to use the extended function, the user transmits current location information from the terminal to the server.

[0985] 8. Server Operation:

[0986] The server collects flyer information for nearby commercial facilities based on current location information.

[0987] 9. Server Operation:

[0988] The server uses image analysis AI to analyze flyer information and extract appropriate product information, reflecting the results in the menu and providing recommendations.

[0989] Detailed recipe provided

[0990] 10. Server Operation:

[0991] The server generates a link to the detailed recipe associated with the suggested meal.

[0992] 11. Terminal Operation:

[0993] When the user clicks on the link, the device will transition to the recipe site to display the detailed recipe.

[0994] Specific examples

[0995] Example 1: Healthy Japanese menu

[0996] 1. User input:

[0997] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[0998] 2. Server processing:

[0999] The server sends this information to the language generation AI.

[1000] 3. Results of language generation AI:

[1001] The language generation AI suggests "grilled salted salmon, boiled broccoli, and brown rice."

[1002] 4. Advertisement Selection:

[1003] The server selects advertisements for health foods and beauty salons and sends them to the user's terminal.

[1004] 5. Displaying the results:

[1005] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[1006] 6. Use of extensions:

[1007] When a user submits their current location information, the server collects flyer information from nearby commercial facilities and makes additional suggestions based on the analysis results.

[1008] This allows users to easily plan their daily menu, and by utilizing advertising and flyer information, they can also obtain information on how to shop at bargain prices.

[1009] The processing flow will be explained below.

[1010] Step 1:

[1011] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[1012] Step 2:

[1013] The user clicks the "Submit" button to send the entered menu requirements to the server.

[1014] Step 3:

[1015] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[1016] Step 4:

[1017] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[1018] Step 5:

[1019] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[1020] Step 6:

[1021] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[1022] Step 7:

[1023] The server selects relevant advertisements based on the information entered by the user. For example, advertisements for health foods and beauty salons are selected for health-conscious criteria, and advertisements for fast food are selected for hearty meals.

[1024] Step 8:

[1025] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[1026] Step 9:

[1027] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1028] Step 10:

[1029] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1030] Step 11:

[1031] The server uses the user's current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[1032] Step 12:

[1033] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[1034] Step 13:

[1035] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1036] Step 14:

[1037] The server sends the newly generated recommendations to the user's device.

[1038] Step 15:

[1039] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1040] Step 16:

[1041] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1042] Step 17:

[1043] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1044] By following the above steps, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals.

[1045] Example 1

[1046] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] The present invention aims to provide a system that allows users to efficiently decide on a menu and obtain related information. Specifically, the system automatically generates an appropriate menu based on various menu conditions specified by the user, collects sales information from nearby commercial facilities using the user's current location information, and uses image analysis technology to suggest optimal product information, thereby enabling users to easily plan their daily meals.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1049] In this invention, the server includes means for collecting menu conditions input by the user, means for generating a menu using a generative AI model, means for selecting relevant advertisements based on the information input by the user, means for transmitting the generated menu and selected advertisements to the user's terminal, means for transmitting current location information operated by the user, means for collecting sales information of nearby commercial facilities based on the current location information, means for analyzing the sales information using image analysis technology, and means for recommending product information based on the analyzed sales information. This allows users to obtain menus automatically generated based on a variety of menu conditions, and further enables them to obtain optimal product information by utilizing sales information based on their current location.

[1050] The "means for collecting menu conditions" is a function that collects information such as the cooking genre, theme, type of menu, and ingredients desired to be used that the user inputs within the application.

[1051] "Means for generating menus using a generative AI model" is a function that uses artificial intelligence to automatically generate appropriate menus based on collected menu conditions.

[1052] The "means for selecting advertisements" is a function by which the server searches and selects relevant advertisements based on the information entered by the user and the generated menu.

[1053] "Means for transmitting menu and selected advertisement to user's terminal" is a function in which the server compiles the generated menu information and selected advertisement information and transmits them as data to the user's terminal.

[1054] The "means for transmitting current location information" is a function that allows the user to obtain current location information through an operation and transmit it to the server.

[1055] The "means for collecting sales information" is a function that allows the server to obtain sales information from nearby commercial facilities based on the user's current location information.

[1056] The "means for analyzing sales information using image analysis technology" is a function for analyzing collected sales information using image analysis technology and extracting useful information.

[1057] "Means for recommending product information" is a function that suggests optimal product information to users based on analyzed sales information.

[1058] The present invention is a system that automatically proposes meal plans based on user-specified criteria and displays related advertisements. Specifically, it collects menu plan conditions entered by the user and generates menu plans based on those conditions using a generative AI model. It also utilizes the user's current location information to collect sales information from nearby commercial facilities and analyzes it using image analysis technology to provide optimal product information. A specific embodiment of this system is described below.

[1059] User input of information

[1060] First, the user launches the application and inputs the menu requirements. Specifically, the user enters information into a form, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). This input information is collected on the device and then sent to the server.

[1061] Processing of input information and menu generation by the server

[1062] The server receives the menu requirements sent by the user and converts them into an appropriate format. It then passes the requirements to a generative AI model (such as OpenAI's GPT-4) to generate an appropriate menu. The generative AI model analyzes the input requirements and suggests an appropriate menu. For example, given the requirements of "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and ingredients of "salmon, broccoli, brown rice," it would generate a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[1063] Ad selection and display

[1064] Based on the generated menu, the server searches and selects relevant advertisements. For example, for a health-conscious menu, advertisements for healthy foods are selected. These advertisements are sent to the user's terminal together with the generated menu and displayed on the terminal.

[1065] Utilizing current location information and collecting sales information

[1066] Furthermore, if the user provides their current location information, the device sends that information to a server. Based on the current location information, the server collects sales information from nearby commercial facilities via APIs and other means. The collected sales information is analyzed using image analysis technology (for example, Google Cloud Vision API) to extract useful product information.

[1067] Product recommendations and detailed recipes

[1068] The extracted product information is recommended to the user and reflected in the menu. The server also generates a link to a detailed recipe related to the suggested menu and sends it to the terminal in a format that the user can use. When the user clicks the link, the terminal is redirected to a detailed recipe site where the detailed recipe is displayed.

[1069] Specific examples

[1070] For example, if a user inputs "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients, the generative AI model will suggest "grilled salmon with salt, boiled broccoli, and brown rice." The server will then use image analysis technology to analyze health food advertisements and related product information sold at nearby commercial facilities, and make additional suggestions based on the results. This allows users to easily decide on a menu and obtain the most appropriate information when shopping.

[1071] Prompt Sentence Examples

[1072] "Please suggest a menu for tonight's dinner. I'd like to use salmon, broccoli, and brown rice as healthy ingredients for a Japanese meal."

[1073] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1074] Step 1: The user launches the application and enters menu requirements.

[1075] Specific operation: The user moves from the application's home screen to the menu condition input form. The form has input fields for cooking genre, theme, menu type, and desired ingredients. The user enters the conditions in these fields and presses the submit button.

[1076] Input: cuisine genre, theme, type of menu, ingredients you want to use

[1077] Output: Menu conditions entered by the user

[1078] Step 2: The terminal sends the input data from the user to the server.

[1079] What happens: The device collects the data entered by the user and sends it to the server as an HTTP request. The data is encoded in JSON format.

[1080] Input: Menu conditions entered by the user

[1081] Output: Menu conditions sent to the server

[1082] Step 3: The server receives the menu requirements sent by the user and converts them into an appropriate format.

[1083] What it does: The server parses the received data and formats it for internal processing, in a format that is easy for the generative AI model to understand (e.g., JSON).

[1084] Input: Menu conditions received from the terminal

[1085] Output: Formatted menu items

[1086] Step 4: The server sends the formatted menu requirements to the generative AI model.

[1087] Specific operation: The server sends the formatted data as a request to the API of the generative AI model (e.g., OpenAI's GPT-4).

[1088] Input: Formatted menu requirements

[1089] Output: The request sent to the generative AI model

[1090] Step 5: The generative AI model analyzes the conditions and generates the optimal menu.

[1091] Specific operation: The generative AI model analyzes the received conditions and generates an appropriate menu that meets the conditions. The result is sent back to the server in JSON format.

[1092] Input: The request sent to the generative AI model

[1093] Output: Generated menu

[1094] Step 6: The server receives the generated menu and selects relevant advertisements.

[1095] Specific operation: The server searches the advertisement database based on the generated menu information and selects relevant advertisements. For example, for a health-conscious menu, it selects advertisements for health foods.

[1096] Input: Generated menu

[1097] Output: Selected ads

[1098] Step 7: The server integrates the generated menu with the selected advertisements and sends it to the user's terminal.

[1099] Specific operation: The server combines the generated menu data and the selected advertising data into a single JSON object and sends it to the user's device as an HTTP response.

[1100] Input: Generated menu and selected advertisement

[1101] Output: The response sent to the user's device

[1102] Step 8: The user's device receives the response from the server and displays the menu and advertisements on the screen.

[1103] Specific operation: The device analyzes the received response data and displays menu information and advertising information in the appropriate position on the screen.

[1104] Input: Response from the server

[1105] Output: Menu and advertisements displayed on the screen

[1106] Step 9: If the user provides current location information, the terminal sends the information to the server.

[1107] Specific operation: The user presses the "Send current location" button within the application, and the device obtains GPS data and sends it to the server as an HTTP request.

[1108] Input: User's current location information

[1109] Output: Current location information sent to the server

[1110] Step 10: The server collects sales information from nearby commercial facilities based on the current location information.

[1111] Specific operation: The server uses the collected location information to obtain sales information from the commercial facility's API or database.

[1112] Input: User's current location information

[1113] Output: Retrieved sales information

[1114] Step 11: The server analyzes the collected sales information using image analysis technology.

[1115] Specific operation: The server analyzes the collected sales information using image analysis technology (e.g., Google Cloud Vision API) and extracts useful product information.

[1116] Input: Collected sales information

[1117] Output: Parsed product information

[1118] Step 12: Based on the analyzed product information, the server recommends the most suitable product information to the user.

[1119] Specific operation: Based on the analyzed product information, the server generates recommendations for the user and sends them to the user's device.

[1120] Input: Parsed product information

[1121] Output: Recommendations sent to the user's device

[1122] (Application example 1)

[1123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1124] Conventional food delivery applications require users to select the menu themselves, and they lack the ability to provide menu suggestions and related information. Furthermore, there is a need for applications that utilize the user's current location to collect special offers and menu information from nearby restaurants and efficiently present optimized options.

[1125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1126] In this invention, the server includes means for collecting menu requirements entered by the user, means for generating a menu using language generation AI, means for selecting relevant advertisements based on the information entered by the user, means for acquiring the user's current location information and collecting special offers and menu information from nearby restaurants, and means for transmitting the generated menu and selected advertisements to the user's terminal. This allows the user to efficiently receive optimal menu suggestions and conveniently take advantage of special offer information from nearby restaurants.

[1127] "Menu conditions" refers to information necessary when deciding on a menu, such as the type of cuisine the user desires, the theme, the budget, and the ingredients they wish to use.

[1128] "Language generation AI" is artificial intelligence that uses natural language processing technology to generate text based on specified conditions.

[1129] "Advertisement" means information or messages provided to users for the purpose of promoting a particular product or service.

[1130] "Current location information" refers to location information obtained from a user terminal and is used to identify the geographic location of the user.

[1131] "Restaurant" refers to a facility or store that serves food and drinks.

[1132] "Special offers" refers to special offers such as discounts and campaigns offered by restaurants.

[1133] "Menu Information" refers to the list of food and drinks served at a restaurant and their detailed information.

[1134] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts specific information and features from it.

[1135] "Product information" refers to detailed information about ingredients and related products used in a dish.

[1136] A "menu" refers to the menu, composition, and combination of meals, and is a list of dishes suggested based on the user's conditions.

[1137] A "recipe" is a set of instructions that describes how to cook a dish, the steps involved, and the amounts of ingredients to use.

[1138] "Calorie information" refers to information that indicates the amount of energy contained in food or dishes, and is used for nutritional management and maintaining health.

[1139] This invention is a food delivery system that automatically suggests meal plans based on user-specified criteria and displays related advertisements. The system collects menu options entered by the user and generates menus based on those criteria using language generation AI. It uses the user's current location information to collect special offers and menu information from nearby restaurants, providing efficiently optimized options. It also provides detailed recipes and calorie information for the menus selected by the user.

[1140] Hardware and software used

[1141] Hardware: Smartphone (iOS, Android)

[1142] Software: Food delivery app, cloud server, location information API, image analysis AI, language generation AI

[1143] Program processing explanation

[1144] User operations

[1145] Users open a food delivery app and input their menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.).

[1146] Server Operations

[1147] The server receives the menu requirements sent by the user, formats them appropriately, and sends them to the language generation AI. The language generation AI analyzes these requirements and generates an appropriate menu. For example, if the budget for a Japanese dinner is 2,000 yen and the specified ingredients are "tomatoes and chicken," the language generation AI will suggest a menu such as "teriyaki chicken, tomato salad, and miso soup."

[1148] Advertisement selection and display

[1149] The server selects relevant advertisements based on the user's input information. It obtains the user's current location information and collects special offers and menu information from nearby restaurants. The server analyzes this information using image analysis AI and extracts appropriate product information.

[1150] Sending and displaying results

[1151] The server compiles the generated menu, related advertisements, special offers, and menu information and sends it to the user's device. The user's device receives this information and displays it on the food delivery app. The user can then view the menu suggestions, related advertisements, and special offers.

[1152] Specific examples

[1153] Example: Smartphone food delivery app

[1154] Below is an example of a specific prompt sentence that suggests the optimal menu when the user enters "Japanese food," "dinner," "budget 2,000 yen," and "tomato, chicken."

[1155] Prompt Sentence Examples

[1156] Suggest the best menu for a user when they input "Japanese food," "dinner," "budget 2000 yen," "tomato, chicken." Display the suggested menu with detailed recipes and calorie information.

[1157] The present invention is expected to enable users to efficiently receive optimal menu suggestions and take advantage of special offer information from nearby restaurants, thereby improving their food delivery experience.

[1158] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1159] Step 1:

[1160] The user opens the food delivery app and inputs the menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.). The input information is sent from the device to the server.

[1161] Step 2:

[1162] The server receives the menu requirements sent by the user. It formats the received information into an appropriate format and sends it to the language generation AI. At this time, the input data includes the cuisine genre, theme, budget, and ingredient information. The formatted data is generated as the output.

[1163] Step 3:

[1164] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a Japanese dinner has a budget of 2,000 yen and the specified ingredients are tomatoes and chicken, it will suggest a menu such as "Teriyaki chicken, tomato salad, and miso soup." The proposed menu is generated as the output.

[1165] Step 4:

[1166] The server selects relevant advertisements based on the user's input information. The server takes into consideration the user's cuisine genre, theme, budget, etc., when selecting advertisements. It also obtains the user's current location information and collects special offers and menu information from nearby restaurants. The input data includes the user's current location information, and the output generates relevant advertisements and special offer information from restaurants.

[1167] Step 5:

[1168] The server uses image analysis AI to analyze the collected restaurant special offer information. Through the analysis, restaurant special offer information and new menu details are extracted. The input data includes restaurant flyers and offer images, and the analyzed special offer information is generated as output.

[1169] Step 6:

[1170] The server aggregates the generated menu, related advertisements, special offers, and menu information and sends it to the user's device, allowing the user to view this information in a single interface. The input data includes the generated menu information, related advertisements, and special offer information, and the aggregated information is sent to the user's device as output.

[1171] Step 7:

[1172] The device displays the received information. The user can view and use menu suggestions, advertisements, special offers, and menu information in the application. Output includes information displayed in the application.

[1173] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1174] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining it with an emotion engine that recognizes the user's emotions. Furthermore, it displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the ingredients they need.

[1175] Program processing explanation

[1176] User input of information

[1177] 1. User Action:

[1178] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[1179] Server processing of input information

[1180] 2. Server processing:

[1181] The server receives and formats the menu requirements sent by the user.

[1182] 3. Server processing:

[1183] The server then sends the formatted information to a language generation AI to generate an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the system will suggest a menu such as "grilled salmon, boiled broccoli, and brown rice."

[1184] Using the Emotion Engine

[1185] 4. Device operation:

[1186] The device recognizes the user's current emotion using an emotion engine, which analyzes the user's facial expressions, voice tone, and input text.

[1187] 5. Server Processing:

[1188] The server receives emotional data from the emotion engine and adjusts the menu suggestions accordingly. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1189] Advertisement selection and display

[1190] 6. Server Processing:

[1191] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user desires relaxation, advertisements for products related to relaxation will be displayed.

[1192] 7. Server Processing:

[1193] The server combines the generated menu with the selected advertisements and transmits them to the user's terminal.

[1194] 8. Terminal operation:

[1195] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1196] Implementing the extension

[1197] 9. User Actions:

[1198] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1199] 10. Server Processing:

[1200] The server collects flyer information for nearby commercial facilities based on current location information.

[1201] 11. Server Processing:

[1202] The server uses image analysis AI to analyze the flyer information and extract appropriate product information.

[1203] 12. Server Processing:

[1204] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1205] 13. Server Processing:

[1206] The server sends the newly generated recommendations to the user's device.

[1207] 14. Terminal Operation:

[1208] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1209] Detailed recipe provided

[1210] 15. Server Processing:

[1211] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1212] 16. Terminal Operation:

[1213] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1214] Specific examples

[1215] Example 1: Healthy Japanese menu and use of emotion engine

[1216] 1. User input:

[1217] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[1218] 2. Server processing:

[1219] The server sends this information to a language generation AI to generate a menu, suggesting, for example, "grilled salmon, boiled broccoli, and brown rice."

[1220] 3. Use of Emotion Engine:

[1221] The device recognizes the user's emotions and transmits them to the server. For example, if the device detects that the user is seeking relaxation, the server will select advertisements related to relaxation.

[1222] 4. Advertisement Selection:

[1223] The server sends the selected advertisements and menus to the user's terminal.

[1224] 5. Displaying the results:

[1225] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[1226] By utilizing this system, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals. Furthermore, by combining it with an emotion engine, it becomes possible to make more personalized suggestions based on the user's emotions.

[1227] The processing flow will be explained below.

[1228] Step 1:

[1229] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[1230] Step 2:

[1231] The user clicks the "Submit" button to send the entered menu requirements to the server.

[1232] Step 3:

[1233] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[1234] Step 4:

[1235] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[1236] Step 5:

[1237] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[1238] Step 6:

[1239] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[1240] Step 7:

[1241] The device activates an emotion engine to recognize the user's current emotion, which analyzes emotions from the user's facial expressions, voice tone, and input text.

[1242] Step 8:

[1243] The device sends emotional data analyzed by the emotion engine to the server. For example, the user's emotion is recognized as "stress."

[1244] Step 9:

[1245] The server receives the emotion data and adjusts the menu suggestions accordingly, for example adding ingredients and recipes that have a relaxing effect for a user who is feeling stressed.

[1246] Step 10:

[1247] The server then tailors the selected advertisements based on the emotional data, for example, selecting advertisements for products or services that are expected to have a relaxing effect.

[1248] Step 11:

[1249] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[1250] Step 12:

[1251] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1252] Step 13:

[1253] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1254] Step 14:

[1255] The server uses the current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[1256] Step 15:

[1257] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[1258] Step 16:

[1259] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1260] Step 17:

[1261] The server sends the newly generated recommendations to the user's device.

[1262] Step 18:

[1263] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1264] Step 19:

[1265] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1266] Step 20:

[1267] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1268] By following the above steps, users can easily decide on their daily menu, and by using the emotion engine, it is possible to propose more personalized menus based on the user's emotions.In addition, by utilizing advertisements and flyer information, users can optimally select the ingredients they need and obtain information on deals.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1271] Conventional menu suggestion systems lack personalized suggestions that take into account the user's emotions and information about nearby commercial facilities. Furthermore, they are unable to reflect the user's current emotional state when providing detailed recipe links or selecting related advertisements, resulting in a suboptimal user experience.

[1272] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1273] In this invention, the server includes: means for collecting menu conditions input by the user; means for generating a menu using a generative AI model; means for using an emotion engine to recognize the user's emotions; means for adjusting the menu based on the user's emotion data; means for selecting related advertisements; means for transmitting the generated menu and advertisements to the user's terminal; means for acquiring the user's current location information and collecting information on nearby commercial facilities; and means for generating product information that analyzes the collected information using image analysis AI and reflects it in the menu. This enables more personalized menu suggestions and advertisement selection based on the user's emotions and current location.

[1274] The "means for collecting menu conditions" is an element of the system that collects information about the menu entered by the user, such as the food genre, theme, type of menu, and ingredients desired to be used, through a digital form or the like.

[1275] A "generative AI model" is a type of artificial intelligence that uses natural language processing technology to generate appropriate menus based on conditions provided by the user.

[1276] The "means for using an emotion engine" is a component of the system that has the function of recognizing emotions by analyzing the user's emotional state from facial expressions, voice tone, and input text.

[1277] The "menu adjustment means" is a system element that has the function of adjusting the generated menu based on the user's emotional data and changing it to a form that suits the user's emotions.

[1278] The "means for selecting advertisements" refers to a system element for selecting and displaying highly relevant advertisements based on user input information and emotional data.

[1279] The "transmission means" is a system element that has the function of transmitting the generated menu and selected advertisements in digital form to the user's terminal.

[1280] The "means for collecting information" is an element of the system that has the function of acquiring information about the user's current location and using that information to collect flyer information for nearby commercial facilities, etc.

[1281] The "means for generating product information" is an element of the system that has the function of analyzing collected flyer information using image analysis AI and generating product information related to the menu based on the results.

[1282] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining them with an emotion engine that recognizes the user's emotions. It also displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information.

[1283] System Overview

[1284] This system includes a user device, a server, a generative AI model, an emotion engine, and an image analysis AI. Each component and its function are explained below.

[1285] Collecting user input information

[1286] Users access the application using a device such as a smartphone or PC. The application provides a form for entering the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple dish, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). The device collects this information and sends it to the server.

[1287] Menu generation

[1288] The server receives the information sent by the user and organizes the data. The organized information is then sent to a generative AI model (e.g., GPT-4). The generative AI model creates a prompt based on the information provided and generates an appropriate menu. For example, a prompt might be, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." Based on this prompt, the generative AI model generates a menu such as "grilled salmon, boiled broccoli, and brown rice."

[1289] Using the Emotion Engine

[1290] The user's device uses an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's facial expressions, voice tone, and input text to identify the user's emotions. For example, it may use Microsoft Azure's Emotion API. The analyzed emotion data is sent to the server. The server adjusts the generated menu based on this emotion data. For example, if the user is feeling stressed, it may change the recipe to one that has a relaxing effect.

[1291] Advertisement selection and display

[1292] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for relaxation-related products. The selected advertisements and generated menus are sent from the server to the user's device, where they are displayed.

[1293] Collection and analysis of flyer information

[1294] When a user uses the extension, the device sends the user's current location information to the server. The server uses the current location information to collect flyers from nearby commercial facilities and analyzes the flyer information using image analysis AI (e.g., Google Cloud Vision API). Based on the analysis results, recommendations including information on great deals are generated and reflected in the user's menu suggestions.

[1295] Specific examples

[1296] If a user inputs the conditions "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice," the system sends a prompt to the generative AI model: "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." The generative AI model generates a menu of "grilled salmon with salt, broccoli in ohitashi sauce, and brown rice" and responds to the server. If the emotion engine's analysis determines that the user is seeking relaxation, the server selects relaxation-related advertisements and sends them to the user's device. The user's device displays this information, and the user can view the proposed menu and related advertisements.

[1297] By using this system, users can easily plan their daily menu and receive personalized suggestions that take into account their emotions and special offers.

[1298] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1299] Step 1:

[1300] The user opens the application and enters the cooking genre, theme, type of menu, and ingredients they want to use in the provided form. Specifically, the user enters "Japanese cuisine," "healthy," "staple food / main dish / side dish," and "salmon, broccoli, brown rice." This generates input data on the user's device.

[1301] Step 2:

[1302] When the device receives user input information, it sends it to the server. The server then formats the received data in a way that makes it easier to format. Specifically, the input data is organized by converting each input item into a specific data structure (e.g., JSON format).

[1303] Step 3:

[1304] Based on the prepared information, the server generates a prompt to send to the generative AI model. For example, it generates a prompt such as, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." This generates a prompt.

[1305] Step 4:

[1306] The server sends the generated prompt to the generative AI model to obtain an appropriate menu. The generative AI model (e.g., GPT-4) analyzes the prompt and generates a menu that matches the requested conditions. For example, a menu such as "grilled salted salmon, boiled broccoli, and brown rice" is generated. This generates the menu data.

[1307] Step 5:

[1308] The device recognizes the user's current emotion using an emotion engine (e.g., Microsoft Azure's Emotion API) that analyzes emotions from the user's facial expressions, voice tone, and input text, generating the user's emotion data.

[1309] Step 6:

[1310] The server receives emotional data from the device and adjusts the menu suggestions based on the analysis results. Specifically, it changes ingredients and recipes to have a relaxing effect based on the user's emotional state (e.g., stress). This results in adjusted menu data.

[1311] Step 7:

[1312] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user is looking to relax, the server selects advertisements related to relaxation goods. This generates selected advertisement data.

[1313] Step 8:

[1314] The server combines the generated menu with the selected advertisement and sends it to the user's terminal. The server then compiles the data, generates a response, and sends it to the terminal. This sends the menu and advertisement data to the user's terminal.

[1315] Step 9:

[1316] The device displays the response received from the server. Specifically, it displays menu information and related advertisements on the screen. The user can then check the suggested menu and advertisements. This provides the user with the information they need.

[1317] Step 10:

[1318] When a user uses the extended function, the terminal transmits the current location information to the server, which transmits the current location data to the server.

[1319] Step 11:

[1320] The server collects flyer information from nearby commercial facilities based on the current location information. For example, it obtains supermarket flyer data using an online API. This allows the collected flyer data to be obtained.

[1321] Step 12:

[1322] The server analyzes the flyer information using image analysis AI. Specifically, it uses image analysis AI (e.g., Google Cloud Vision API) to extract product information from the flyer data. This provides the analyzed product information.

[1323] Step 13:

[1324] Based on the analysis results, the server generates recommendations that reflect additional information in the menu suggestions. Specifically, it generates menu suggestions that include information on special sales and deals. This generates recommendation data.

[1325] Step 14:

[1326] The server sends the newly generated recommendations to the user's device, which then sends the recommendation data to the user's device.

[1327] Step 15:

[1328] The device displays new recommendations, and users can check the suggested menu along with the optimal product information, allowing them to efficiently select ingredients by taking advantage of the discount information.

[1329] Step 16:

[1330] The server generates a link to a detailed recipe related to the proposed menu and sends it to the user's terminal, thereby generating a detailed recipe link.

[1331] Step 17:

[1332] When the user clicks on the link, the device will be redirected to a recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1333] (Application example 2)

[1334] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1335] In today's busy society, it is difficult to efficiently plan menus and optimally select the necessary ingredients. Furthermore, there are few systems that can provide personalized suggestions based on the user's emotions and moods. Furthermore, there is a lack of systems for effectively purchasing related products and mechanisms that integrate advertising displays and electronic payments. The purpose of this invention is to solve these problems.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for collecting menu conditions entered by the user, a means for generating menus using a language generation AI, and a means for selecting related advertisements based on the information entered by the user. This enables efficient and personalized menu suggestions and related product purchases.

[1337] The server also includes a means for analyzing the user's facial expressions and voice tone to acquire emotional data, a means for adjusting the menu based on the emotional data, and a means for allowing the user to purchase suggested products in cooperation with an electronic payment service, thereby enabling more personalized suggestions based on the user's emotions and enabling related products to be purchased immediately.

[1338] "Menu conditions" refers to information such as an outline or theme of the dish desired by the user, ingredients that the user wants to use, and the like.

[1339] "Language generation AI" is an artificial intelligence technology that generates sentences and data based on input information.

[1340] "Advertisement" refers to the promotion of related products selected based on the user's interests and input information.

[1341] "Emotion data" is emotional information analyzed from the user's facial expressions and voice tone.

[1342] "Adjusting the menu" means changing the contents of the menu to be suggested based on the acquired emotional data.

[1343] An "electronic payment service" is a service that allows for online monetary transactions using the Internet.

[1344] "Flyer information" is product advertisement information collected from nearby commercial facilities.

[1345] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts information.

[1346] "Product information" refers to detailed information about individual products extracted from flyer information using image analysis AI.

[1347] The present invention is a system that automatically proposes meal menus based on user-entered criteria and further adjusts the proposals by recognizing the user's emotions. Specifically, the system configuration and operation are as follows:

[1348] Overall system configuration

[1349] 1. User operations

[1350] First, users open the smartphone app and input the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[1351] 2. Menu generation

[1352] The server receives the menu requirements sent by the user and formats them. This is then sent to a language generation AI (e.g., GPT-4) to generate an appropriate menu. For example, if a user is health-conscious and prefers Japanese cuisine, and the specified ingredients are salmon, broccoli, and brown rice, the generation AI will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[1353] 3. Emotion recognition

[1354] Using the smartphone's camera and microphone, the user's facial expressions and vocal tone are analyzed by an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to a server.

[1355] 4. Menu adjustment

[1356] The server then adjusts the menu based on the user's emotional data. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1357] 5. Advertisement selection and display

[1358] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for related products. The server then sends the generated menu and the selected advertisements to the user's smartphone for display.

[1359] 6. Electronic payment integration

[1360] The server connects to electronic payment services (e.g., PayPal, Stripe) so that users can instantly purchase the suggested products (ingredients, cooking utensils, etc.) within the app. Users can easily complete the purchase procedure within the app.

[1361] Specific examples

[1362] Example 1: Healthy Japanese menu and use of emotion engine

[1363] 1. The user inputs "Japanese food," "healthy food," "staple food / main dish / side dish," and ingredients "salmon, broccoli, brown rice."

[1364] 2. The server sends this information to a language generation AI to generate a menu. For example, it suggests "grilled salmon, boiled broccoli, and brown rice."

[1365] 3. Emotion recognition: It is analyzed that the user is seeking relaxation.

[1366] 4. Suggest relaxing recipes and display related advertisements (relaxing products, herbal tea, etc.).

[1367] 5. Make ingredients and products available for purchase via electronic payment services.

[1368] Example prompt sentence:

[1369] "The user wants to relax and enjoy healthy Japanese food, with salmon, broccoli, and brown rice. Please suggest a menu based on this."

[1370] This allows users to easily find efficient and personalized menus and even purchase related products directly.

[1371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1372] Step 1:

[1373] User operations

[1374] The user opens the smartphone app and inputs the menu requirements, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), type of menu (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[1375] Input: Menu conditions, theme, ingredients

[1376] Output: The input data is sent to the app's server.

[1377] Step 2:

[1378] Server Processing

[1379] The server receives the input menu requirements and formats this information. The formatted data is sent to a language generation AI model (e.g., GPT-4). For example, it might be formatted as "Japanese food for health-conscious people, with salmon, broccoli, and brown rice."

[1380] Input: User's menu requirements, theme, ingredients

[1381] Output: Formatted prompt text

[1382] Step 3:

[1383] Processing of generated AI

[1384] The server sends prompts to the language generation AI model to generate an appropriate menu. For example, a prompt such as "Healthy Japanese food, with salmon, broccoli, and brown rice" is sent, and the AI ​​generates a menu based on this prompt, such as "Grilled salmon, boiled broccoli, and brown rice."

[1385] Input: Formatted prompt text

[1386] Output: Generated menu data

[1387] Step 4:

[1388] Performing emotion recognition

[1389] The user's device uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to the server.

[1390] Input: User's facial expressions and voice tones

[1391] Output: Emotion data

[1392] Step 5:

[1393] Menu adjustment based on emotions

[1394] The server receives the emotion data and adjusts the generated menu accordingly: for example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1395] Input: Emotion data, generated menu data

[1396] Output: Adjusted menu data

[1397] Step 6:

[1398] Ad selection and delivery

[1399] The server selects relevant advertisements based on the user's input information and emotional data. For example, advertisements for relaxation-related products are selected for a user seeking relaxation. The generated menu and the selected advertisements are sent to the user's device.

[1400] Input: User input, emotion data, adjusted menu data

[1401] Output: Selected ads

[1402] Step 7:

[1403] Displaying the final result

[1404] The user's device receives the response from the server and displays the adjusted menu and related advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1405] Input: Response data from the server

[1406] Output: Menu and advertisements displayed on the screen

[1407] Step 8:

[1408] Making electronic payments

[1409] If the user wishes to purchase the suggested products (ingredients, cooking utensils, etc.), the purchase process is carried out in cooperation with an electronic payment service (e.g., PayPal, Stripe). After the purchase process is completed, the purchase information is sent to the server.

[1410] Input: User's purchase request, electronic payment platform

[1411] Output: Purchase completion confirmation data

[1412] This allows users to receive personalized menu suggestions and immediately purchase related products.

[1413] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1415] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1416] [Fourth embodiment]

[1417] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1418] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1419] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1420] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1421] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1423] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1424] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1425] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1426] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1427] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1428] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1429] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1430] This invention is a system that can automatically propose meal menus based on user-specified criteria and display related advertisements. Specifically, it collects menu conditions entered by the user and generates menus based on those conditions using language generation AI. Furthermore, it uses the user's current location information to collect flyers from nearby commercial facilities and analyzes them using image analysis AI to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the necessary ingredients.

[1431] Program processing explanation

[1432] User input of information

[1433] 1. User Action:

[1434] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[1435] Server processing of input information

[1436] 2. Server processing:

[1437] The server receives the menu requirements sent by the user, formats the received information appropriately, and passes it to the language generation AI.

[1438] Menu generation using language generation AI

[1439] 3. Server Operation:

[1440] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the AI ​​will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[1441] Advertisement selection and display

[1442] 4. Server Operation:

[1443] The server selects relevant advertisements based on the user's input information. For example, it selects advertisements for health foods for health-conscious menus, and advertisements for fast food for hearty menus.

[1444] 5. Server Operation:

[1445] The server combines the generated menu with the selected advertisement and transmits it to the user's terminal.

[1446] 6. Device Operation:

[1447] The user's terminal receives the response from the server and displays the menu and advertisements on the screen.

[1448] Implementing the extension

[1449] 7. User Actions:

[1450] When a user wants to use the extended function, the user transmits current location information from the terminal to the server.

[1451] 8. Server Operation:

[1452] The server collects flyer information for nearby commercial facilities based on current location information.

[1453] 9. Server Operation:

[1454] The server uses image analysis AI to analyze flyer information and extract appropriate product information, reflecting the results in the menu and providing recommendations.

[1455] Detailed recipe provided

[1456] 10. Server Operation:

[1457] The server generates a link to the detailed recipe associated with the suggested meal.

[1458] 11. Terminal Operation:

[1459] When the user clicks on the link, the device will transition to the recipe site to display the detailed recipe.

[1460] Specific examples

[1461] Example 1: Healthy Japanese menu

[1462] 1. User input:

[1463] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[1464] 2. Server processing:

[1465] The server sends this information to the language generation AI.

[1466] 3. Results of language generation AI:

[1467] The language generation AI suggests "grilled salted salmon, boiled broccoli, and brown rice."

[1468] 4. Advertisement Selection:

[1469] The server selects advertisements for health foods and beauty salons and sends them to the user's terminal.

[1470] 5. Displaying the results:

[1471] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[1472] 6. Use of extensions:

[1473] When a user submits their current location information, the server collects flyer information from nearby commercial facilities and makes additional suggestions based on the analysis results.

[1474] This allows users to easily plan their daily menu, and by utilizing advertising and flyer information, they can also obtain information on how to shop at bargain prices.

[1475] The processing flow will be explained below.

[1476] Step 1:

[1477] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[1478] Step 2:

[1479] The user clicks the "Submit" button to send the entered menu requirements to the server.

[1480] Step 3:

[1481] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[1482] Step 4:

[1483] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[1484] Step 5:

[1485] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[1486] Step 6:

[1487] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[1488] Step 7:

[1489] The server selects relevant advertisements based on the information entered by the user. For example, advertisements for health foods and beauty salons are selected for health-conscious criteria, and advertisements for fast food are selected for hearty meals.

[1490] Step 8:

[1491] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[1492] Step 9:

[1493] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1494] Step 10:

[1495] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1496] Step 11:

[1497] The server uses the user's current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[1498] Step 12:

[1499] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[1500] Step 13:

[1501] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1502] Step 14:

[1503] The server sends the newly generated recommendations to the user's device.

[1504] Step 15:

[1505] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1506] Step 16:

[1507] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1508] Step 17:

[1509] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1510] By following the above steps, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals.

[1511] Example 1

[1512] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1513] The present invention aims to provide a system that allows users to efficiently decide on a menu and obtain related information. Specifically, the system automatically generates an appropriate menu based on various menu conditions specified by the user, collects sales information from nearby commercial facilities using the user's current location information, and uses image analysis technology to suggest optimal product information, thereby enabling users to easily plan their daily meals.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1515] In this invention, the server includes means for collecting menu conditions input by the user, means for generating a menu using a generative AI model, means for selecting relevant advertisements based on the information input by the user, means for transmitting the generated menu and selected advertisements to the user's terminal, means for transmitting current location information operated by the user, means for collecting sales information of nearby commercial facilities based on the current location information, means for analyzing the sales information using image analysis technology, and means for recommending product information based on the analyzed sales information. This allows users to obtain menus automatically generated based on a variety of menu conditions, and further enables them to obtain optimal product information by utilizing sales information based on their current location.

[1516] The "means for collecting menu conditions" is a function that collects information such as the cooking genre, theme, type of menu, and ingredients desired to be used that the user inputs within the application.

[1517] "Means for generating menus using a generative AI model" is a function that uses artificial intelligence to automatically generate appropriate menus based on collected menu conditions.

[1518] The "means for selecting advertisements" is a function by which the server searches and selects relevant advertisements based on the information entered by the user and the generated menu.

[1519] "Means for transmitting menu and selected advertisement to user's terminal" is a function in which the server compiles the generated menu information and selected advertisement information and transmits them as data to the user's terminal.

[1520] The "means for transmitting current location information" is a function that allows the user to obtain current location information through an operation and transmit it to the server.

[1521] The "means for collecting sales information" is a function that allows the server to obtain sales information from nearby commercial facilities based on the user's current location information.

[1522] The "means for analyzing sales information using image analysis technology" is a function for analyzing collected sales information using image analysis technology and extracting useful information.

[1523] "Means for recommending product information" is a function that suggests optimal product information to users based on analyzed sales information.

[1524] The present invention is a system that automatically proposes meal plans based on user-specified criteria and displays related advertisements. Specifically, it collects menu plan conditions entered by the user and generates menu plans based on those conditions using a generative AI model. It also utilizes the user's current location information to collect sales information from nearby commercial facilities and analyzes it using image analysis technology to provide optimal product information. A specific embodiment of this system is described below.

[1525] User input of information

[1526] First, the user launches the application and inputs the menu requirements. Specifically, the user enters information into a form, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). This input information is collected on the device and then sent to the server.

[1527] Processing of input information and menu generation by the server

[1528] The server receives the menu requirements sent by the user and converts them into an appropriate format. It then passes the requirements to a generative AI model (such as OpenAI's GPT-4) to generate an appropriate menu. The generative AI model analyzes the input requirements and suggests an appropriate menu. For example, given the requirements of "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and ingredients of "salmon, broccoli, brown rice," it would generate a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[1529] Ad selection and display

[1530] Based on the generated menu, the server searches and selects relevant advertisements. For example, for a health-conscious menu, advertisements for healthy foods are selected. These advertisements are sent to the user's terminal together with the generated menu and displayed on the terminal.

[1531] Utilizing current location information and collecting sales information

[1532] Furthermore, if the user provides their current location information, the device sends that information to a server. Based on the current location information, the server collects sales information from nearby commercial facilities via APIs and other means. The collected sales information is analyzed using image analysis technology (for example, Google Cloud Vision API) to extract useful product information.

[1533] Product recommendations and detailed recipes

[1534] The extracted product information is recommended to the user and reflected in the menu. The server also generates a link to a detailed recipe related to the suggested menu and sends it to the terminal in a format that the user can use. When the user clicks the link, the terminal is redirected to a detailed recipe site where the detailed recipe is displayed.

[1535] Specific examples

[1536] For example, if a user inputs "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients, the generative AI model will suggest "grilled salmon with salt, boiled broccoli, and brown rice." The server will then use image analysis technology to analyze health food advertisements and related product information sold at nearby commercial facilities, and make additional suggestions based on the results. This allows users to easily decide on a menu and obtain the most appropriate information when shopping.

[1537] Prompt Sentence Examples

[1538] "Please suggest a menu for tonight's dinner. I'd like to use salmon, broccoli, and brown rice as healthy ingredients for a Japanese meal."

[1539] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1540] Step 1: The user launches the application and enters menu requirements.

[1541] Specific operation: The user moves from the application's home screen to the menu condition input form. The form has input fields for cooking genre, theme, menu type, and desired ingredients. The user enters the conditions in these fields and presses the submit button.

[1542] Input: cuisine genre, theme, type of menu, ingredients you want to use

[1543] Output: Menu conditions entered by the user

[1544] Step 2: The terminal sends the input data from the user to the server.

[1545] What happens: The device collects the data entered by the user and sends it to the server as an HTTP request. The data is encoded in JSON format.

[1546] Input: Menu conditions entered by the user

[1547] Output: Menu conditions sent to the server

[1548] Step 3: The server receives the menu requirements sent by the user and converts them into an appropriate format.

[1549] What it does: The server parses the received data and formats it for internal processing, in a format that is easy for the generative AI model to understand (e.g., JSON).

[1550] Input: Menu conditions received from the terminal

[1551] Output: Formatted menu items

[1552] Step 4: The server sends the formatted menu requirements to the generative AI model.

[1553] Specific operation: The server sends the formatted data as a request to the API of the generative AI model (e.g., OpenAI's GPT-4).

[1554] Input: Formatted menu requirements

[1555] Output: The request sent to the generative AI model

[1556] Step 5: The generative AI model analyzes the conditions and generates the optimal menu.

[1557] Specific operation: The generative AI model analyzes the received conditions and generates an appropriate menu that meets the conditions. The result is sent back to the server in JSON format.

[1558] Input: The request sent to the generative AI model

[1559] Output: Generated menu

[1560] Step 6: The server receives the generated menu and selects relevant advertisements.

[1561] Specific operation: The server searches the advertisement database based on the generated menu information and selects relevant advertisements. For example, for a health-conscious menu, it selects advertisements for health foods.

[1562] Input: Generated menu

[1563] Output: Selected ads

[1564] Step 7: The server integrates the generated menu with the selected advertisements and sends it to the user's terminal.

[1565] Specific operation: The server combines the generated menu data and the selected advertising data into a single JSON object and sends it to the user's device as an HTTP response.

[1566] Input: Generated menu and selected advertisement

[1567] Output: The response sent to the user's device

[1568] Step 8: The user's device receives the response from the server and displays the menu and advertisements on the screen.

[1569] Specific operation: The device analyzes the received response data and displays menu information and advertising information in the appropriate position on the screen.

[1570] Input: Response from the server

[1571] Output: Menu and advertisements displayed on the screen

[1572] Step 9: If the user provides current location information, the terminal sends the information to the server.

[1573] Specific operation: The user presses the "Send current location" button within the application, and the device obtains GPS data and sends it to the server as an HTTP request.

[1574] Input: User's current location information

[1575] Output: Current location information sent to the server

[1576] Step 10: The server collects sales information from nearby commercial facilities based on the current location information.

[1577] Specific operation: The server uses the collected location information to obtain sales information from the commercial facility's API or database.

[1578] Input: User's current location information

[1579] Output: Retrieved sales information

[1580] Step 11: The server analyzes the collected sales information using image analysis technology.

[1581] Specific operation: The server analyzes the collected sales information using image analysis technology (e.g., Google Cloud Vision API) and extracts useful product information.

[1582] Input: Collected sales information

[1583] Output: Parsed product information

[1584] Step 12: Based on the analyzed product information, the server recommends the most suitable product information to the user.

[1585] Specific operation: Based on the analyzed product information, the server generates recommendations for the user and sends them to the user's device.

[1586] Input: Parsed product information

[1587] Output: Recommendations sent to the user's device

[1588] (Application example 1)

[1589] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1590] Conventional food delivery applications require users to select the menu themselves, and they lack the ability to provide menu suggestions and related information. Furthermore, there is a need for applications that utilize the user's current location to collect special offers and menu information from nearby restaurants and efficiently present optimized options.

[1591] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1592] In this invention, the server includes means for collecting menu requirements entered by the user, means for generating a menu using language generation AI, means for selecting relevant advertisements based on the information entered by the user, means for acquiring the user's current location information and collecting special offers and menu information from nearby restaurants, and means for transmitting the generated menu and selected advertisements to the user's terminal. This allows the user to efficiently receive optimal menu suggestions and conveniently take advantage of special offer information from nearby restaurants.

[1593] "Menu conditions" refers to information necessary when deciding on a menu, such as the type of cuisine the user desires, the theme, the budget, and the ingredients they wish to use.

[1594] "Language generation AI" is artificial intelligence that uses natural language processing technology to generate text based on specified conditions.

[1595] "Advertisement" means information or messages provided to users for the purpose of promoting a particular product or service.

[1596] "Current location information" refers to location information obtained from a user terminal and is used to identify the geographic location of the user.

[1597] "Restaurant" refers to a facility or store that serves food and drinks.

[1598] "Special offers" refers to special offers such as discounts and campaigns offered by restaurants.

[1599] "Menu Information" refers to the list of food and drinks served at a restaurant and their detailed information.

[1600] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts specific information and features from it.

[1601] "Product information" refers to detailed information about ingredients and related products used in a dish.

[1602] A "menu" refers to the menu, composition, and combination of meals, and is a list of dishes suggested based on the user's conditions.

[1603] A "recipe" is a set of instructions that describes how to cook a dish, the steps involved, and the amounts of ingredients to use.

[1604] "Calorie information" refers to information that indicates the amount of energy contained in food or dishes, and is used for nutritional management and maintaining health.

[1605] This invention is a food delivery system that automatically suggests meal plans based on user-specified criteria and displays related advertisements. The system collects menu options entered by the user and generates menus based on those criteria using language generation AI. It uses the user's current location information to collect special offers and menu information from nearby restaurants, providing efficiently optimized options. It also provides detailed recipes and calorie information for the menus selected by the user.

[1606] Hardware and software used

[1607] Hardware: Smartphone (iOS, Android)

[1608] Software: Food delivery app, cloud server, location information API, image analysis AI, language generation AI

[1609] Program processing explanation

[1610] User operations

[1611] Users open a food delivery app and input their menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.).

[1612] Server Operations

[1613] The server receives the menu requirements sent by the user, formats them appropriately, and sends them to the language generation AI. The language generation AI analyzes these requirements and generates an appropriate menu. For example, if the budget for a Japanese dinner is 2,000 yen and the specified ingredients are "tomatoes and chicken," the language generation AI will suggest a menu such as "teriyaki chicken, tomato salad, and miso soup."

[1614] Advertisement selection and display

[1615] The server selects relevant advertisements based on the user's input information. It obtains the user's current location information and collects special offers and menu information from nearby restaurants. The server analyzes this information using image analysis AI and extracts appropriate product information.

[1616] Sending and displaying results

[1617] The server compiles the generated menu, related advertisements, special offers, and menu information and sends it to the user's device. The user's device receives this information and displays it on the food delivery app. The user can then view the menu suggestions, related advertisements, and special offers.

[1618] Specific examples

[1619] Example: Smartphone food delivery app

[1620] Below is an example of a specific prompt sentence that suggests the optimal menu when the user enters "Japanese food," "dinner," "budget 2,000 yen," and "tomato, chicken."

[1621] Prompt Sentence Examples

[1622] Suggest the best menu for a user when they input "Japanese food," "dinner," "budget 2000 yen," "tomato, chicken." Display the suggested menu with detailed recipes and calorie information.

[1623] The present invention is expected to enable users to efficiently receive optimal menu suggestions and take advantage of special offer information from nearby restaurants, thereby improving their food delivery experience.

[1624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1625] Step 1:

[1626] The user opens the food delivery app and inputs the menu requirements, including the cuisine type (Japanese, Western, Italian, etc.), theme (light meal, dinner), budget, and desired ingredients (e.g., tomatoes, chicken, etc.). The input information is sent from the device to the server.

[1627] Step 2:

[1628] The server receives the menu requirements sent by the user. It formats the received information into an appropriate format and sends it to the language generation AI. At this time, the input data includes the cuisine genre, theme, budget, and ingredient information. The formatted data is generated as the output.

[1629] Step 3:

[1630] The server sends the formatted menu conditions to the language generation AI, which analyzes these conditions and generates an appropriate menu. For example, if a Japanese dinner has a budget of 2,000 yen and the specified ingredients are tomatoes and chicken, it will suggest a menu such as "Teriyaki chicken, tomato salad, and miso soup." The proposed menu is generated as the output.

[1631] Step 4:

[1632] The server selects relevant advertisements based on the user's input information. The server takes into consideration the user's cuisine genre, theme, budget, etc., when selecting advertisements. It also obtains the user's current location information and collects special offers and menu information from nearby restaurants. The input data includes the user's current location information, and the output generates relevant advertisements and special offer information from restaurants.

[1633] Step 5:

[1634] The server uses image analysis AI to analyze the collected restaurant special offer information. Through the analysis, restaurant special offer information and new menu details are extracted. The input data includes restaurant flyers and offer images, and the analyzed special offer information is generated as output.

[1635] Step 6:

[1636] The server aggregates the generated menu, related advertisements, special offers, and menu information and sends it to the user's device, allowing the user to view this information in a single interface. The input data includes the generated menu information, related advertisements, and special offer information, and the aggregated information is sent to the user's device as output.

[1637] Step 7:

[1638] The device displays the received information. The user can view and use menu suggestions, advertisements, special offers, and menu information in the application. Output includes information displayed in the application.

[1639] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1640] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining it with an emotion engine that recognizes the user's emotions. Furthermore, it displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information. This allows users to efficiently decide on a menu and optimally select the ingredients they need.

[1641] Program processing explanation

[1642] User input of information

[1643] 1. User Action:

[1644] The user opens the application and enters the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice) into a form.

[1645] Server processing of input information

[1646] 2. Server processing:

[1647] The server receives and formats the menu requirements sent by the user.

[1648] 3. Server processing:

[1649] The server then sends the formatted information to a language generation AI to generate an appropriate menu. For example, if a customer is conscious about healthy Japanese cuisine and the specified ingredients are "salmon, broccoli, and brown rice," the system will suggest a menu such as "grilled salmon, boiled broccoli, and brown rice."

[1650] Using the Emotion Engine

[1651] 4. Device operation:

[1652] The device recognizes the user's current emotion using an emotion engine, which analyzes the user's facial expressions, voice tone, and input text.

[1653] 5. Server Processing:

[1654] The server receives emotional data from the emotion engine and adjusts the menu suggestions accordingly. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1655] Advertisement selection and display

[1656] 6. Server Processing:

[1657] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user desires relaxation, advertisements for products related to relaxation will be displayed.

[1658] 7. Server Processing:

[1659] The server combines the generated menu with the selected advertisements and transmits them to the user's terminal.

[1660] 8. Terminal operation:

[1661] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1662] Implementing the extension

[1663] 9. User Actions:

[1664] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1665] 10. Server Processing:

[1666] The server collects flyer information for nearby commercial facilities based on current location information.

[1667] 11. Server Processing:

[1668] The server uses image analysis AI to analyze the flyer information and extract appropriate product information.

[1669] 12. Server Processing:

[1670] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1671] 13. Server Processing:

[1672] The server sends the newly generated recommendations to the user's device.

[1673] 14. Terminal Operation:

[1674] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1675] Detailed recipe provided

[1676] 15. Server Processing:

[1677] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1678] 16. Terminal Operation:

[1679] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1680] Specific examples

[1681] Example 1: Healthy Japanese menu and use of emotion engine

[1682] 1. User input:

[1683] The user inputs "Japanese food," "health-oriented," "staple food / main dish / side dish," and "salmon, broccoli, brown rice" as ingredients.

[1684] 2. Server processing:

[1685] The server sends this information to a language generation AI to generate a menu, suggesting, for example, "grilled salmon, boiled broccoli, and brown rice."

[1686] 3. Use of Emotion Engine:

[1687] The device recognizes the user's emotions and transmits them to the server. For example, if the device detects that the user is seeking relaxation, the server will select advertisements related to relaxation.

[1688] 4. Advertisement Selection:

[1689] The server sends the selected advertisements and menus to the user's terminal.

[1690] 5. Displaying the results:

[1691] The user's device displays these, allowing the user to view the suggested menu and related advertisements.

[1692] By utilizing this system, users can easily plan their daily menu, and by utilizing advertisements and flyer information, they can optimally select the ingredients they need and obtain information on great deals. Furthermore, by combining it with an emotion engine, it becomes possible to make more personalized suggestions based on the user's emotions.

[1693] The processing flow will be explained below.

[1694] Step 1:

[1695] The user opens the application and enters the menu requirements. The user fills in a form with the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and ingredients they want to use (e.g., salmon, broccoli, brown rice).

[1696] Step 2:

[1697] The user clicks the "Submit" button to send the entered menu requirements to the server.

[1698] Step 3:

[1699] The server receives the menu conditions sent by the user, which include the cooking genre, theme, menu type, and ingredient information.

[1700] Step 4:

[1701] The server converts the received information into an appropriate format and prepares it for transmission to the language generation AI.

[1702] Step 5:

[1703] The server sends the formatted menu requirements to a language generation AI, which analyzes this information and generates the corresponding menu.

[1704] Step 6:

[1705] The language generation AI sends the generated menu back to the server. For example, it might generate a menu such as "grilled salted salmon, boiled broccoli, and brown rice."

[1706] Step 7:

[1707] The device activates an emotion engine to recognize the user's current emotion, which analyzes emotions from the user's facial expressions, voice tone, and input text.

[1708] Step 8:

[1709] The device sends emotional data analyzed by the emotion engine to the server. For example, the user's emotion is recognized as "stress."

[1710] Step 9:

[1711] The server receives the emotion data and adjusts the menu suggestions accordingly, for example adding ingredients and recipes that have a relaxing effect for a user who is feeling stressed.

[1712] Step 10:

[1713] The server then tailors the selected advertisements based on the emotional data, for example, selecting advertisements for products or services that are expected to have a relaxing effect.

[1714] Step 11:

[1715] The server combines the generated menu with the selected advertisements and generates a response to be sent to the user's terminal.

[1716] Step 12:

[1717] The user's device receives the response from the server and displays the menu and advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1718] Step 13:

[1719] When a user wants to use the extended function, the user transmits current location information from the device to the server.

[1720] Step 14:

[1721] The server uses the current location information to collect flyers from nearby commercial establishments, including supermarkets and grocery stores.

[1722] Step 15:

[1723] The server uses image analysis AI to analyze the collected flyer information and extract appropriate product information.

[1724] Step 16:

[1725] Based on the analysis results, the server generates recommendations that reflect additional information in the newly proposed menu, including information on bargain items.

[1726] Step 17:

[1727] The server sends the newly generated recommendations to the user's device.

[1728] Step 18:

[1729] The user's device will display new recommendations, and the user can check the suggested menu along with the optimal product information.

[1730] Step 19:

[1731] The server generates a link to the detailed recipe associated with the proposed menu and sends it to the user's terminal.

[1732] Step 20:

[1733] When the user clicks on the link, the device will be redirected to the recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1734] By following the above steps, users can easily decide on their daily menu, and by using the emotion engine, it is possible to propose more personalized menus based on the user's emotions.In addition, by utilizing advertisements and flyer information, users can optimally select the ingredients they need and obtain information on deals.

[1735] Example 2

[1736] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1737] Conventional menu suggestion systems lack personalized suggestions that take into account the user's emotions and information about nearby commercial facilities. Furthermore, they are unable to reflect the user's current emotional state when providing detailed recipe links or selecting related advertisements, resulting in a suboptimal user experience.

[1738] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1739] In this invention, the server includes: means for collecting menu conditions input by the user; means for generating a menu using a generative AI model; means for using an emotion engine to recognize the user's emotions; means for adjusting the menu based on the user's emotion data; means for selecting related advertisements; means for transmitting the generated menu and advertisements to the user's terminal; means for acquiring the user's current location information and collecting information on nearby commercial facilities; and means for generating product information that analyzes the collected information using image analysis AI and reflects it in the menu. This enables more personalized menu suggestions and advertisement selection based on the user's emotions and current location.

[1740] The "means for collecting menu conditions" is an element of the system that collects information about the menu entered by the user, such as the food genre, theme, type of menu, and ingredients desired to be used, through a digital form or the like.

[1741] A "generative AI model" is a type of artificial intelligence that uses natural language processing technology to generate appropriate menus based on conditions provided by the user.

[1742] The "means for using an emotion engine" is a component of the system that has the function of recognizing emotions by analyzing the user's emotional state from facial expressions, voice tone, and input text.

[1743] The "menu adjustment means" is a system element that has the function of adjusting the generated menu based on the user's emotional data and changing it to a form that suits the user's emotions.

[1744] The "means for selecting advertisements" refers to a system element for selecting and displaying highly relevant advertisements based on user input information and emotional data.

[1745] The "transmission means" is a system element that has the function of transmitting the generated menu and selected advertisements in digital form to the user's terminal.

[1746] The "means for collecting information" is an element of the system that has the function of acquiring information about the user's current location and using that information to collect flyer information for nearby commercial facilities, etc.

[1747] The "means for generating product information" is an element of the system that has the function of analyzing collected flyer information using image analysis AI and generating product information related to the menu based on the results.

[1748] This system automatically proposes meal plans based on user-entered criteria and adjusts the proposals by combining them with an emotion engine that recognizes the user's emotions. It also displays relevant advertisements and analyzes flyers from nearby commercial facilities to provide optimal product information.

[1749] System Overview

[1750] This system includes a user device, a server, a generative AI model, an emotion engine, and an image analysis AI. Each component and its function are explained below.

[1751] Collecting user input information

[1752] Users access the application using a device such as a smartphone or PC. The application provides a form for entering the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple dish, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice). The device collects this information and sends it to the server.

[1753] Menu generation

[1754] The server receives the information sent by the user and organizes the data. The organized information is then sent to a generative AI model (e.g., GPT-4). The generative AI model creates a prompt based on the information provided and generates an appropriate menu. For example, a prompt might be, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." Based on this prompt, the generative AI model generates a menu such as "grilled salmon, boiled broccoli, and brown rice."

[1755] Using the Emotion Engine

[1756] The user's device uses an emotion engine to recognize the user's emotions. This emotion engine analyzes the user's facial expressions, voice tone, and input text to identify the user's emotions. For example, it may use Microsoft Azure's Emotion API. The analyzed emotion data is sent to the server. The server adjusts the generated menu based on this emotion data. For example, if the user is feeling stressed, it may change the recipe to one that has a relaxing effect.

[1757] Advertisement selection and display

[1758] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for relaxation-related products. The selected advertisements and generated menus are sent from the server to the user's device, where they are displayed.

[1759] Collection and analysis of flyer information

[1760] When a user uses the extension, the device sends the user's current location information to the server. The server uses the current location information to collect flyers from nearby commercial facilities and analyzes the flyer information using image analysis AI (e.g., Google Cloud Vision API). Based on the analysis results, recommendations including information on great deals are generated and reflected in the user's menu suggestions.

[1761] Specific examples

[1762] If a user inputs the conditions "Japanese cuisine," "health-conscious," "staple food / main dish / side dish," and "salmon, broccoli, brown rice," the system sends a prompt to the generative AI model: "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." The generative AI model generates a menu of "grilled salmon with salt, broccoli in ohitashi sauce, and brown rice" and responds to the server. If the emotion engine's analysis determines that the user is seeking relaxation, the server selects relaxation-related advertisements and sends them to the user's device. The user's device displays this information, and the user can view the proposed menu and related advertisements.

[1763] By using this system, users can easily plan their daily menu and receive personalized suggestions that take into account their emotions and special offers.

[1764] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1765] Step 1:

[1766] The user opens the application and enters the cooking genre, theme, type of menu, and ingredients they want to use in the provided form. Specifically, the user enters "Japanese cuisine," "healthy," "staple food / main dish / side dish," and "salmon, broccoli, brown rice." This generates input data on the user's device.

[1767] Step 2:

[1768] When the device receives user input information, it sends it to the server. The server then formats the received data in a way that makes it easier to format. Specifically, the input data is organized by converting each input item into a specific data structure (e.g., JSON format).

[1769] Step 3:

[1770] Based on the prepared information, the server generates a prompt to send to the generative AI model. For example, it generates a prompt such as, "Please suggest a healthy Japanese menu using salmon, broccoli, and brown rice." This generates a prompt.

[1771] Step 4:

[1772] The server sends the generated prompt to the generative AI model to obtain an appropriate menu. The generative AI model (e.g., GPT-4) analyzes the prompt and generates a menu that matches the requested conditions. For example, a menu such as "grilled salted salmon, boiled broccoli, and brown rice" is generated. This generates the menu data.

[1773] Step 5:

[1774] The device recognizes the user's current emotion using an emotion engine (e.g., Microsoft Azure's Emotion API) that analyzes emotions from the user's facial expressions, voice tone, and input text, generating the user's emotion data.

[1775] Step 6:

[1776] The server receives emotional data from the device and adjusts the menu suggestions based on the analysis results. Specifically, it changes ingredients and recipes to have a relaxing effect based on the user's emotional state (e.g., stress). This results in adjusted menu data.

[1777] Step 7:

[1778] The server selects relevant advertisements based on the user's input information and emotion data. For example, if the user is looking to relax, the server selects advertisements related to relaxation goods. This generates selected advertisement data.

[1779] Step 8:

[1780] The server combines the generated menu with the selected advertisement and sends it to the user's terminal. The server then compiles the data, generates a response, and sends it to the terminal. This sends the menu and advertisement data to the user's terminal.

[1781] Step 9:

[1782] The device displays the response received from the server. Specifically, it displays menu information and related advertisements on the screen. The user can then check the suggested menu and advertisements. This provides the user with the information they need.

[1783] Step 10:

[1784] When a user uses the extended function, the terminal transmits the current location information to the server, which transmits the current location data to the server.

[1785] Step 11:

[1786] The server collects flyer information from nearby commercial facilities based on the current location information. For example, it obtains supermarket flyer data using an online API. This allows the collected flyer data to be obtained.

[1787] Step 12:

[1788] The server analyzes the flyer information using image analysis AI. Specifically, it uses image analysis AI (e.g., Google Cloud Vision API) to extract product information from the flyer data. This provides the analyzed product information.

[1789] Step 13:

[1790] Based on the analysis results, the server generates recommendations that reflect additional information in the menu suggestions. Specifically, it generates menu suggestions that include information on special sales and deals. This generates recommendation data.

[1791] Step 14:

[1792] The server sends the newly generated recommendations to the user's device, which then sends the recommendation data to the user's device.

[1793] Step 15:

[1794] The device displays new recommendations, and users can check the suggested menu along with the optimal product information, allowing them to efficiently select ingredients by taking advantage of the discount information.

[1795] Step 16:

[1796] The server generates a link to a detailed recipe related to the proposed menu and sends it to the user's terminal, thereby generating a detailed recipe link.

[1797] Step 17:

[1798] When the user clicks on the link, the device will be redirected to a recipe site to display the detailed recipe, allowing the user to check the detailed cooking instructions.

[1799] (Application example 2)

[1800] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1801] In today's busy society, it is difficult to efficiently plan menus and optimally select the necessary ingredients. Furthermore, there are few systems that can provide personalized suggestions based on the user's emotions and moods. Furthermore, there is a lack of systems for effectively purchasing related products and mechanisms that integrate advertising displays and electronic payments. The purpose of this invention is to solve these problems.

[1802] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for collecting menu conditions entered by the user, a means for generating menus using a language generation AI, and a means for selecting related advertisements based on the information entered by the user. This enables efficient and personalized menu suggestions and related product purchases.

[1803] The server also includes a means for analyzing the user's facial expressions and voice tone to acquire emotional data, a means for adjusting the menu based on the emotional data, and a means for allowing the user to purchase suggested products in cooperation with an electronic payment service, thereby enabling more personalized suggestions based on the user's emotions and enabling related products to be purchased immediately.

[1804] "Menu conditions" refers to information such as an outline or theme of the dish desired by the user, ingredients that the user wants to use, and the like.

[1805] "Language generation AI" is an artificial intelligence technology that generates sentences and data based on input information.

[1806] "Advertisement" refers to the promotion of related products selected based on the user's interests and input information.

[1807] "Emotion data" is emotional information analyzed from the user's facial expressions and voice tone.

[1808] "Adjusting the menu" means changing the contents of the menu to be suggested based on the acquired emotional data.

[1809] An "electronic payment service" is a service that allows for online monetary transactions using the Internet.

[1810] "Flyer information" is product advertisement information collected from nearby commercial facilities.

[1811] "Image analysis AI" is an artificial intelligence technology that analyzes image data and extracts information.

[1812] "Product information" refers to detailed information about individual products extracted from flyer information using image analysis AI.

[1813] The present invention is a system that automatically proposes meal menus based on user-entered criteria and further adjusts the proposals by recognizing the user's emotions. Specifically, the system configuration and operation are as follows:

[1814] Overall system configuration

[1815] 1. User operations

[1816] First, users open the smartphone app and input the menu criteria, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), menu type (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[1817] 2. Menu generation

[1818] The server receives the menu requirements sent by the user and formats them. This is then sent to a language generation AI (e.g., GPT-4) to generate an appropriate menu. For example, if a user is health-conscious and prefers Japanese cuisine, and the specified ingredients are salmon, broccoli, and brown rice, the generation AI will suggest a menu such as "grilled salmon with salt, boiled broccoli, and brown rice."

[1819] 3. Emotion recognition

[1820] Using the smartphone's camera and microphone, the user's facial expressions and vocal tone are analyzed by an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to a server.

[1821] 4. Menu adjustment

[1822] The server then adjusts the menu based on the user's emotional data. For example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1823] 5. Advertisement selection and display

[1824] The server selects relevant advertisements based on the user's input information and emotional data. For example, if the user wants to relax, it selects advertisements for related products. The server then sends the generated menu and the selected advertisements to the user's smartphone for display.

[1825] 6. Electronic payment integration

[1826] The server connects to electronic payment services (e.g., PayPal, Stripe) so that users can instantly purchase the suggested products (ingredients, cooking utensils, etc.) within the app. Users can easily complete the purchase procedure within the app.

[1827] Specific examples

[1828] Example 1: Healthy Japanese menu and use of emotion engine

[1829] 1. The user inputs "Japanese food," "healthy food," "staple food / main dish / side dish," and ingredients "salmon, broccoli, brown rice."

[1830] 2. The server sends this information to a language generation AI to generate a menu. For example, it suggests "grilled salmon, boiled broccoli, and brown rice."

[1831] 3. Emotion recognition: It is analyzed that the user is seeking relaxation.

[1832] 4. Suggest relaxing recipes and display related advertisements (relaxing products, herbal tea, etc.).

[1833] 5. Make ingredients and products available for purchase via electronic payment services.

[1834] Example prompt sentence:

[1835] "The user wants to relax and enjoy healthy Japanese food, with salmon, broccoli, and brown rice. Please suggest a menu based on this."

[1836] This allows users to easily find efficient and personalized menus and even purchase related products directly.

[1837] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1838] Step 1:

[1839] User operations

[1840] The user opens the smartphone app and inputs the menu requirements, such as the cuisine genre (Japanese, Western, Chinese, etc.), theme (hearty, stylish, health-conscious, etc.), type of menu (staple food, main dish, side dish), and desired ingredients (e.g., salmon, broccoli, brown rice).

[1841] Input: Menu conditions, theme, ingredients

[1842] Output: The input data is sent to the app's server.

[1843] Step 2:

[1844] Server Processing

[1845] The server receives the input menu requirements and formats this information. The formatted data is sent to a language generation AI model (e.g., GPT-4). For example, it might be formatted as "Japanese food for health-conscious people, with salmon, broccoli, and brown rice."

[1846] Input: User's menu requirements, theme, ingredients

[1847] Output: Formatted prompt text

[1848] Step 3:

[1849] Processing of generated AI

[1850] The server sends prompts to the language generation AI model to generate an appropriate menu. For example, a prompt such as "Healthy Japanese food, with salmon, broccoli, and brown rice" is sent, and the AI ​​generates a menu based on this prompt, such as "Grilled salmon, boiled broccoli, and brown rice."

[1851] Input: Formatted prompt text

[1852] Output: Generated menu data

[1853] Step 4:

[1854] Performing emotion recognition

[1855] The user's device uses the smartphone's camera and microphone to analyze the user's facial expressions and voice tone using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to obtain emotional data, which is then sent to the server.

[1856] Input: User's facial expressions and voice tones

[1857] Output: Emotion data

[1858] Step 5:

[1859] Menu adjustment based on emotions

[1860] The server receives the emotion data and adjusts the generated menu accordingly: for example, if the user is feeling stressed, it will suggest ingredients and recipes that have a relaxing effect.

[1861] Input: Emotion data, generated menu data

[1862] Output: Adjusted menu data

[1863] Step 6:

[1864] Ad selection and delivery

[1865] The server selects relevant advertisements based on the user's input information and emotional data. For example, advertisements for relaxation-related products are selected for a user seeking relaxation. The generated menu and the selected advertisements are sent to the user's device.

[1866] Input: User input, emotion data, adjusted menu data

[1867] Output: Selected ads

[1868] Step 7:

[1869] Displaying the final result

[1870] The user's device receives the response from the server and displays the adjusted menu and related advertisements on the screen, allowing the user to check the suggested menu and related advertisements.

[1871] Input: Response data from the server

[1872] Output: Menu and advertisements displayed on the screen

[1873] Step 8:

[1874] Making electronic payments

[1875] If the user wishes to purchase the suggested products (ingredients, cooking utensils, etc.), the purchase process is carried out in cooperation with an electronic payment service (e.g., PayPal, Stripe). After the purchase process is completed, the purchase information is sent to the server.

[1876] Input: User's purchase request, electronic payment platform

[1877] Output: Purchase completion confirmation data

[1878] This allows users to receive personalized menu suggestions and immediately purchase related products.

[1879] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1881] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1882] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1883] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1884] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1885] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1886] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1887] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1888] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1889] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1890] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1891] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1892] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1893] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1894] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1895] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1896] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1897] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1898] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1899] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1900] The following is further disclosed regarding the above embodiment.

[1901] (Claim 1)

[1902] A means for collecting menu conditions input by a user;

[1903] A means for generating a menu by a language generation AI based on the menu conditions;

[1904] A means for selecting relevant advertisements based on the information entered by the user;

[1905] A means for transmitting the generated menu and the selected advertisement to a user's terminal;

[1906] A system including:

[1907] (Claim 2)

[1908] The system according to claim 1, wherein the system acquires the user's current location information and collects flyer information of nearby commercial facilities.

[1909] (Claim 3)

[1910] The system of claim 1 analyzes the flyer information using image analysis AI and generates product information that is reflected in the menu.

[1911] "Example 1"

[1912] (Claim 1)

[1913] A means for collecting menu conditions input by a user;

[1914] A means for generating a menu by a generation AI model based on the menu conditions;

[1915] A means for selecting relevant advertisements based on the information entered by the user;

[1916] A means for transmitting the generated menu and the selected advertisement to a user's terminal;

[1917] A means for transmitting current location information by user operation;

[1918] A means for collecting sales information of nearby commercial facilities based on the current location information;

[1919] A means for analyzing the sales information using image analysis technology;

[1920] A means for recommending product information based on the analyzed sales information;

[1921] A system including:

[1922] (Claim 2)

[1923] The system of claim 1, further comprising: collecting current location information of the user.

[1924] (Claim 3)

[1925] The system according to claim 1 analyzes the sales information using image analysis technology and generates product information that is reflected in the menu.

[1926] "Application Example 1"

[1927] (Claim 1)

[1928] A means for collecting menu conditions input by a user;

[1929] A means for generating a menu by a language generation AI based on the menu conditions;

[1930] A means for selecting relevant advertisements based on the information entered by the user;

[1931] A means for obtaining the user's current location information and collecting special offers and menu information from nearby restaurants;

[1932] A means for transmitting the generated menu and the selected advertisement to a user's terminal;

[1933] A system including:

[1934] (Claim 2)

[1935] The system according to claim 1 generates product information that analyzes the restaurant's offer information using image analysis AI and reflects it in the menu.

[1936] (Claim 3)

[1937] 2. The system according to claim 1, further comprising means for providing detailed recipes and calorie information for dishes based on the menu suggestions and product information.

[1938] "Example 2: Combining Emotion Engines"

[1939] (Claim 1)

[1940] A means for collecting menu conditions input by a user;

[1941] A means for generating a menu by a generation AI model based on the menu conditions;

[1942] Using an emotion engine to recognize user emotions;

[1943] a menu adjustment means for adjusting the menu based on the user's emotion data;

[1944] A means for selecting relevant advertisements based on the information and emotional data input by the user;

[1945] A means for transmitting the generated menu and the selected advertisement to a user's terminal;

[1946] A means for acquiring the user's current location information and collecting information on nearby commercial facilities;

[1947] The collected information is analyzed by image analysis AI and reflected in the menu [means for generating product information;

[1948] A system including:

[1949] (Claim 2)

[1950] The system according to claim 1, wherein a link to a detailed recipe related to the generated menu is generated and transmitted to the user's terminal.

[1951] (Claim 3)

[1952] The system of claim 1, wherein the generated menu and selected advertisement are displayed on a user's terminal.

[1953] "Application example 2 when combining emotion engines"

[1954] (Claim 1)

[1955] A means for collecting menu conditions input by a user;

[1956] A means for generating a menu by a language generation AI based on the menu conditions;

[1957] A means for selecting relevant advertisements based on the information entered by the user;

[1958] A means for transmitting the generated menu and the selected advertisement to a user's terminal;

[1959] A means of acquiring emotional data by analyzing the user's facial expressions and voice tone;

[1960] A means for adjusting a menu based on the emotion data;

[1961] In cooperation with electronic payment services, users can purchase suggested products,

[1962] A system including:

[1963] (Claim 2)

[1964] The system according to claim 1, wherein the system acquires the user's current location information and collects flyer information of nearby commercial facilities.

[1965] (Claim 3)

[1966] The system of claim 1 analyzes the flyer information using image analysis AI and generates product information that is reflected in the menu. [Explanation of symbols]

[1967] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting menu conditions input by a user; A means for generating a menu using a language generation AI based on the menu conditions; means for selecting relevant advertisements based on information input by a user; means for transmitting the generated menu and the selected advertisement to a user's terminal; A system including:

2. The system according to claim 1, further comprising: acquiring information on the user's current location; and collecting flyer information on nearby commercial facilities.

3. The system according to claim 1, wherein the flyer information is analyzed using image analysis AI and product information to be reflected in the menu is generated.

Images