System

Abstract

A system is provided.SOLUTION: A system includes food management means for managing food in a refrigerator, recipe acquisition means for acquiring a recipe of a dish from an external resource, recipe adjustment means for automatically adjusting an amount of the acquired recipe based on a family structure and preference, shopping list generation means for generating a shopping list for purchasing necessary food, advertisement presentation means for presenting an advertisement to a user, and receipt analysis means for analyzing a receipt photographed by the user and automatically updating information of the purchased food.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern life demands efficient integration of multiple elements, such as managing ingredients in the refrigerator, planning meals, adjusting recipes, creating shopping lists, and displaying advertisements. However, current methods manage each element individually, making centralized management difficult and requiring time and effort. Furthermore, the cumbersome task of manually inputting receipts makes this impractical. The present invention aims to provide a system that solves these problems and streamlines users' daily lives. [Means for solving the problem]

[0005] The present invention is a system that includes an ingredient management means for managing ingredients in a refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing necessary ingredients, an advertisement presentation means for presenting advertisements to the user, and a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on purchased ingredients.This system allows the user to centrally manage ingredients in the refrigerator and efficiently plan menus, and further allows operations from generating the shopping list to presenting advertisements to the user to be performed using a single application.

[0006] The "food management means" is a function for listing the food ingredients in the refrigerator and managing the inventory status in a unified manner.

[0007] The "recipe acquisition means" is a function for acquiring cooking recipes from external resources or databases and incorporating that information into the system.

[0008] The "recipe adjustment means" is a function that automatically adjusts the quantities and steps of the acquired recipe based on profile information such as family composition, preferences, and cooking skills.

[0009] The "shopping list generating means" is a function that makes it possible for the user to easily go shopping by listing the ingredients needed.

[0010] "Advertising presentation means" is a function for selecting and presenting appropriate advertisements based on the user's profile and browsing information.

[0011] The "receipt analysis means" is a function that analyzes the image of the receipt taken by the user and automatically updates the system with information about the ingredients purchased. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] MODE FOR CARRYING OUT THE INVENTION

[0034] The system of the present invention has the functions of managing ingredients in a refrigerator, proposing optimal menus based on the user's family structure and preferences, generating a shopping list for purchasing necessary ingredients, and providing appropriate advertisements.

[0035] Food management function

[0036] It provides a means to automatically capture purchased ingredient information using a receipt photographed by the user. The user photographs the receipt with their smartphone camera and uploads the image to the app. At this time, the device sends the photographed image to a server. The server analyzes the receipt image, extracts purchased ingredient information, and stores that information in a database. This allows the user to always check the latest ingredient list.

[0037] AI menu suggestion function

[0038] When a user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate an optimal menu using an AI model. It generates multiple menu options and creates a list of ingredients required for each. This allows the user to choose from multiple menu options.

[0039] Automatic recipe generation function

[0040] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities can be adjusted to suit the number of family members and preferences. The generated recipe is sent to the device and can be viewed by the user.

[0041] Advertisement display function

[0042] As a user browses recipes, their device sends that information to a server, which uses the user's profile and recent browsing information to select relevant ads, such as special offers on ingredients or convenient cooking tools. Once a suitable ad is selected, it is sent to the device and presented to the user.

[0043] Shopping list generator

[0044] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a link to purchase the ingredient. The server then sends the link to the device, allowing the user to easily click the link to make a purchase on the relevant site.

[0045] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the ingredients they need. Furthermore, by providing appropriate advertisements, it is possible to provide new value to users.

[0046] The processing flow will be explained below.

[0047] Food management function

[0048] Step 1:

[0049] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[0050] Step 2:

[0051] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user confirms it, the image is sent to the server by pressing the send button.

[0052] Step 3:

[0053] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[0054] Step 4:

[0055] Server: The OCR engine extracts text information from the receipt image and generates data in the format of "3 apples" or "1 liter of milk."

[0056] Step 5:

[0057] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[0058] Step 6:

[0059] Server: Updates the user's ingredient list and sends the latest list information to the device.

[0060] Step 7:

[0061] Device: Display the latest ingredient list on the screen.

[0062] Step 8:

[0063] User: View updated ingredient list in the app.

[0064] ---

[0065] AI menu suggestion function

[0066] Step 1:

[0067] User: Tap the "Menu Suggestion" button on the app.

[0068] Step 2:

[0069] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[0070] Step 3:

[0071] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[0072] Step 4:

[0073] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[0074] Step 5:

[0075] Server: Sends the generated menu options to the terminal.

[0076] Step 6:

[0077] Terminal: Presents multiple menu options to the user.

[0078] Step 7:

[0079] User: Select the desired menu from the displayed menu.

[0080] ---

[0081] Automatic recipe generation function

[0082] Step 1:

[0083] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[0084] Step 2:

[0085] Terminal: Sends the selected menu information and the user's family composition information to the server.

[0086] Step 3:

[0087] Server: Based on the selected menu, launches the recipe generation engine.

[0088] Step 4:

[0089] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members (e.g., "300g chicken, 1 / 2 teaspoon salt...").

[0090] Step 5:

[0091] Server: Sends the generated recipe to the device.

[0092] Step 6:

[0093] Terminal: Displays the generated recipe and cooking instructions to the user.

[0094] Step 7:

[0095] User: Cooks using the displayed recipe as a reference.

[0096] ---

[0097] Advertisement display function

[0098] Step 1:

[0099] Users: See ads while browsing recipes.

[0100] Step 2:

[0101] Device: Sends browsing information to the server.

[0102] Step 3:

[0103] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[0104] Step 4:

[0105] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[0106] Step 5:

[0107] Server: Sends the selected advertisement to the device.

[0108] Step 6:

[0109] Device: Display the received advertisement on the recipe viewing screen.

[0110] Step 7:

[0111] User: View the ad displayed and click to view more information if necessary.

[0112] ---

[0113] Shopping list generator

[0114] Step 1:

[0115] User: Requests to purchase desired ingredients.

[0116] Step 2:

[0117] Terminal: Sends a purchase request to the server.

[0118] Step 3:

[0119] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[0120] Step 4:

[0121] Server: Sends the generated purchase link to the device.

[0122] Step 5:

[0123] Device: Show the user a purchase link.

[0124] Step 6:

[0125] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Conventional refrigerator ingredient management and menu suggestion systems required a lot of manual input work, which was time-consuming for users. They also lacked the functionality to suggest menus suited to the user's family structure and preferences, or to easily manage the purchasing procedures for necessary ingredients. Furthermore, there was a need for a way to provide new value by providing appropriate advertisements to users.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes a means for analyzing receipts photographed by the user and automatically capturing purchased ingredient information, a means for proposing optimal menus based on the user's family structure and preferences, and a means for generating menus using an AI model based on the ingredient list and profile information stored on the server. This allows the user to efficiently manage ingredients in the refrigerator, receive suggestions for optimal menus suited to their family structure and preferences, and smoothly purchase the necessary ingredients.

[0131] "Means of analyzing receipts photographed by users and automatically importing information about purchased ingredients" refers to a function that analyzes images of receipts photographed by users on devices such as smartphones, extracts information about purchased ingredients from the images, and automatically imports the information into a database.

[0132] "Means for suggesting optimal menus based on family structure and preferences" refers to a function that automatically generates and suggests optimal menus based on the user's family structure (for example, number of people and age range) and preferences (preferred dishes and ingredients).

[0133] "Means for generating menus using an AI model based on ingredient lists and profile information stored on a server" refers to a function that stores ingredient lists and user profile information entered by the user or automatically obtained on a server, and automatically generates optimal menus using an AI model based on that information.

[0134] The "recipe generation means for automatically generating the quantities of ingredients required and cooking procedures" is a function for automatically generating and providing the specific quantities of ingredients required and cooking procedures based on the selected menu.

[0135] "Means for selecting relevant advertisements based on the generated recipe and presenting them to the user" refers to a function for selecting highly relevant advertisements (for example, information on special sales of ingredients or advertisements for cooking utensils) based on automatically generated recipe information and displaying them to the user.

[0136] "Means for generating a purchase link using the API of an affiliated online shopping site" refers to a function that uses the API of an affiliated online shopping site to generate a purchase link and provide that link to the user when the user purchases the ingredients they desire.

[0137] MODE FOR CARRYING OUT THE INVENTION

[0138] The system of the present invention utilizes data processing and AI technology to seamlessly perform a series of tasks, from managing ingredients in the refrigerator to suggesting menus, creating recipes, generating shopping lists, and providing advertisements.

[0139] Food management function

[0140] First, the user takes a photo of the receipt for the purchased ingredients using the smartphone camera. Then, they upload the image of the receipt to the app. The device sends the uploaded receipt image to the server. The server uses image analysis software such as Google (registered trademark) Cloud Vision API to extract information about the purchased ingredients from the receipt image. The extracted information about the ingredients is saved in a database. This allows the user to always check the latest list of ingredients on the app. As a specific example, when a user takes a photo of a receipt from a supermarket purchase and uploads it to the app, the server automatically extracts and saves information such as "1L of milk, 10 eggs, 4 tomatoes."

[0141] AI menu suggestion function

[0142] When a user requests a "menu suggestion" within the app, the device sends the latest ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. Based on this information, the server uses a generative AI model such as OpenAI's (registered trademark) GPT-3 (registered trademark) to generate the optimal menu. A specific prompt might be something like, "My family consists of three people, I like Japanese food, and I have chicken and potatoes. Please suggest a menu for me." The server then generates the optimal menu from multiple candidates and provides it to the user. This function allows users to easily obtain the optimal menu.

[0143] Automatic recipe generation function

[0144] When a user selects a desired menu from the menu options provided, the device sends that information to the server. The server uses IBM Watson® recipe generation engine to automatically generate specific amounts of ingredients and cooking instructions based on the selected menu. The generated recipe is sent to the device and can be viewed by the user on the app. For example, a recipe for "Teriyaki Chicken" may contain details such as "200g of chicken, 2 tablespoons of soy sauce, and 1 tablespoon of sugar."

[0145] Advertisement display function

[0146] When a user is browsing a recipe generated by the app, the device sends browsing information to the server. The server selects relevant advertisements based on the user's profile and recent browsing information. These advertisements may include, for example, special sale ingredients or convenient cooking utensils. The selected advertisements are sent to the device in a timely manner and presented to the user. For example, if a user is browsing a recipe for "Mapo Tofu," advertisements for special sale tofu or new cooking utensils may be displayed in the app.

[0147] Shopping list generator

[0148] When a user sends a request to purchase necessary ingredients, the device sends that information to the server. The server uses the API of an affiliated online shopping site (e.g., shopping site API) to generate a purchase link for the necessary ingredients. The generated link is sent to the device, and the user can click on it to easily make a purchase on the related site. For example, in response to a request for "100g of tofu and 1 green onion," the generated purchase link is displayed in the app, and the user can click on it to smoothly make a purchase on the online shopping site.

[0149] The system of the present invention is designed to allow users to efficiently manage ingredients in their refrigerators, suggest optimal menus, and easily purchase the ingredients they need. Furthermore, by providing relevant advertisements, the system can provide new value to users.

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1: Take a photo of your receipt

[0152] Users take a photo of the receipt for the food they purchased using their smartphone camera, and this image becomes input data for the system.

[0153] Step 2: Upload a receipt image

[0154] The user opens the app and uploads a photo of the receipt, which becomes the input data sent to the server.

[0155] Step 3: Send the receipt image to the server

[0156] The device sends the uploaded receipt image to the server. The server uses image analysis software (e.g., image analysis API) to process the data and extract ingredient information from the receipt image. The input in this step is the receipt image, and the output is ingredient information.

[0157] Step 4: Store purchased ingredients in a database

[0158] The server stores the extracted ingredient information in a database. This process allows users to access the latest ingredient list. The input is the extracted ingredient information, and the output is the ingredient information stored in the database.

[0159] Step 5: Request menu suggestions

[0160] Users can request a "menu suggestion" within the app, which becomes new input data for the system.

[0161] Step 6: Submit user information

[0162] The device sends the latest ingredient list and user profile information (family composition, preferences, etc.) to the server. The ingredient list and profile information are input, and are used in the AI ​​model.

[0163] Step 7: Menu generation using AI models

[0164] The server uses a generative AI model to generate an optimal menu based on the input ingredient list and profile information. The specific input prompt is "Family size is 3 people, preference is Japanese food, ingredients available are chicken and potatoes, please suggest a menu." The output of this step is the generated menu.

[0165] Step 8: Offer menu options

[0166] The server sends the generated menu options to the terminal. The user selects the desired menu from these options. The input is the generated menu options, and the output is the menu options provided to the user.

[0167] Step 9: Send menu selection to server

[0168] When the user selects the desired menu, the terminal sends the selection information to the server. The input is the selected menu, which becomes the basis for recipe generation.

[0169] Step 10: Auto-generating recipes

[0170] The server uses a recipe generation engine (e.g., a recipe generation API) to automatically generate the required ingredient quantities and cooking instructions based on the selected menu. The input is the selected menu, and the output is the specific recipe information.

[0171] Step 11: User Provides Recipe Information

[0172] The server sends the generated recipe information to the device so that the user can view it in the app. The input is the automatically generated recipe information, and the output is the recipe information provided to the user.

[0173] Step 12: Send recipe viewing information to the server

[0174] When a user browses recipes on the app, the device sends browsing information to the server. The input is the user's browsing information, which is used as the basis for selecting ads.

[0175] Step 13: Select relevant ads

[0176] The server selects relevant advertisements based on browsing information and user profile. The input is browsing information and profile information, and the output is the selected advertisement.

[0177] Step 14: User Provides Ad Presentation

[0178] The server sends the selected advertisement to the terminal for presentation to the user. The input is the selected advertisement, and the output is the advertisement provided to the user.

[0179] Step 15: Send purchase request to server

[0180] When a user requests to purchase a desired ingredient, the device sends the information to the server. The input is the user's purchase request, which becomes the basis for generating a purchase link.

[0181] Step 16: Generate a Purchase Link

[0182] The server uses the API of an online shopping site to generate a purchase link for the required ingredients. The input is the purchase request, and the output is the generated purchase link.

[0183] Step 17: User Provided Purchase Link

[0184] The server sends the generated purchase link to the terminal, which the user can click to easily make a purchase on the related site. The input is the generated purchase link, and the output is the purchase link provided to the user.

[0185] (Application example 1)

[0186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0187] In modern households, managing the ingredients in the refrigerator, proposing optimal menus, and purchasing ingredients are time-consuming and labor-intensive. This makes preparing home-cooked meals complicated, leading to unnecessary food waste and incorrect purchases. Furthermore, users have limited means of obtaining the information they need without receiving advertisements based on individual ingredients. Furthermore, the inability to smoothly integrate with food delivery services reduces the efficiency of home cooking. A new system is needed to solve these issues.

[0188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0189] In this invention, the server includes an ingredient management unit for managing ingredients in the refrigerator, a recipe acquisition unit for acquiring recipes from external resources, a recipe adjustment unit for automatically adjusting recipe portion sizes based on family composition and preferences, a shopping list generation unit for generating a shopping list for purchasing necessary ingredients, an advertisement presentation unit for presenting advertisements to the user, a receipt analysis unit for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, a menu suggestion unit for generating an optimal menu based on the user's profile information using an AI model, and a delivery collaboration unit for ordering necessary ingredients based on the menu in collaboration with a food delivery service. This allows users to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase and deliver the necessary ingredients. Furthermore, providing appropriate advertisements to users encourages purchasing behavior and improves the efficiency of home cooking.

[0190] The "foodstuff management means" is a system for listing the foodstuffs in the refrigerator and constantly updating and managing the inventory status.

[0191] The "recipe acquisition means" is a mechanism for acquiring cooking recipe information from an external database, the Internet, etc.

[0192] The "recipe adjustment means" is a mechanism for automatically adjusting the amounts of ingredients in the acquired recipe according to the household composition and individual preferences.

[0193] The "shopping list generating means" is a mechanism for automatically generating a list for purchasing necessary ingredients.

[0194] An "advertising presentation means" is a mechanism for displaying highly relevant advertisements to users.

[0195] The "receipt analysis means" is a mechanism that analyzes receipt images taken by the user and automatically updates the list of purchased ingredients based on that information.

[0196] The "menu suggestion method" is a system that uses an AI model to calculate and suggest the optimal menu based on the user's profile information.

[0197] "Delivery collaboration means" is a system that links with a food delivery service to order and request delivery of the necessary ingredients based on the proposed menu.

[0198] MODE FOR CARRYING OUT THE INVENTION

[0199] The present invention is a system that efficiently manages ingredients in a refrigerator, proposes optimal menus, and allows necessary ingredients to be easily purchased and delivered. An embodiment of the present invention will be described below.

[0200] First, to manage the information on ingredients purchased by the user, the user takes a photo of the receipt with the smartphone camera and uploads the image to the app. At this time, the device sends the captured image to the server. The server performs image analysis (using OCR technology) to extract ingredient information from the receipt and saves it in a database. This function allows the user to always manage an up-to-date ingredient list.

[0201] Next, when the user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information to the server. Based on this information, the server uses a generative AI model to generate an optimal menu based on family composition and preferences. At this time, multiple menu options are generated, and the user can choose from these menu options.

[0202] After that, when the user selects the desired menu, the device sends that information to the server, which then uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions based on the selected menu, allowing users to easily obtain recipes that suit the number of family members and their preferences.

[0203] Additionally, when a user is browsing recipes, appropriate advertisements are displayed. The device sends recipe browsing information and the user's profile to the server, and the server selects and presents highly relevant advertisements (such as ingredients on sale or convenient cooking utensils) to the user. This encourages users to make purchases and provides them with even more useful information.

[0204] Furthermore, if the user requests to purchase ingredients, the device sends that information to the server, which then uses the API of a partner food delivery service to generate a link to purchase the ingredients, allowing the user to request delivery with just one click.

[0205] The hardware and software used to build this system includes:

[0206] Hardware: Smartphones (iPhone(R), ANDROID(R) devices)

[0207] software:

[0208] Python: Backend processing for the application

[0209] OpenCV: Receipt image processing and analysis

[0210] Requests: Communication with the server

[0211] For example, a user can purchase new ingredients at the supermarket to prepare dinner and upload the receipt to the app. The app then uses AI to suggest the optimal dinner menu and places an order for the ingredients needed for the user's selected menu with a food delivery service. A detailed recipe is then displayed to help prepare dinner.

[0212] Example prompt sentence:

[0213] 1. "Take a photo of your receipt and upload it."

[0214] 2. "I'll suggest the perfect menu for tonight's dinner."

[0215] 3. "I can quickly order the ingredients I need for this menu."

[0216] 4. "Check out the cookware on sale."

[0217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0218] Step 1:

[0219] The user takes a photo of the receipt with the smartphone camera. Then, they upload the image to the app. The input is the image of the receipt, and the output is its image data. The image data is sent from the device to the server.

[0220] Step 2:

[0221] The server analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the image data of the receipt, and the output is text data of the extracted ingredient information. This text data is stored in a database.

[0222] Step 3:

[0223] The user requests a "menu suggestion" within the app. The input is the user's request, profile information (family composition, preferences, etc.), and saved ingredient list. The output is this information sent to the server.

[0224] Step 4:

[0225] The server uses the received user profile information and ingredient list to generate an optimal meal plan using a generative AI model (e.g., GPT-3). The input is the profile information and ingredient list, and the output is multiple suggested meal options. These meal options are sent to the device.

[0226] Step 5:

[0227] The user selects the desired menu. The input is the user's menu selection information, and the output is the selection information sent to the server.

[0228] Step 6:

[0229] The server uses a recipe generation engine (e.g., Recipe API) to automatically generate the quantities of ingredients and cooking instructions based on the selected menu. The input is the selected menu information, and the output is the generated recipe information. This recipe information is sent to the device.

[0230] Step 7:

[0231] When a user browses a recipe, the device sends the browsing information to the server. The input is the browsing information and the user's profile information, and the output is the information sent to the server.

[0232] Step 8:

[0233] The server uses the profile information and browsing information to search a database for relevant advertisements (e.g., special offers on ingredients or cooking utensils). The input is the profile information and browsing information, and the output is appropriate advertising information. This advertising information is sent to the terminal and displayed to the user.

[0234] Step 9:

[0235] A user submits an order request to purchase the ingredients they need. The input is the purchase request, and the output is the request information sent to the server.

[0236] Step 10:

[0237] The server uses the API of a partner food delivery service to generate a purchase link for the required ingredients. The input is the purchase request information, and the output is the generated purchase link. This link is sent to the terminal, and the user can click on it to use the delivery service.

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] MODE FOR CARRYING OUT THE INVENTION

[0240] The system of the present invention manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion engine can further improve the user experience.

[0241] Food management function

[0242] Users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This allows users to always check the latest ingredient list.

[0243] AI menu suggestion function

[0244] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate menu suggestions using an AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0245] Automatic recipe generation function

[0246] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[0247] Advertisement display function

[0248] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools. Once the appropriate advertisement is selected, it is sent to the device and presented to the user.

[0249] Shopping list generator

[0250] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[0251] Use of emotion engine

[0252] Users can input their emotions using the app's camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[0253] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[0254] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[0255] The processing flow will be explained below.

[0256] A system that combines emotion engines

[0257] Food management function

[0258] Step 1:

[0259] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[0260] Step 2:

[0261] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user has checked it, the image is sent to the server when the user presses the send button.

[0262] Step 3:

[0263] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[0264] Step 4:

[0265] Server: The OCR engine extracts text information from the receipt image and generates data such as "3 apples" and "1 liter of milk."

[0266] Step 5:

[0267] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[0268] Step 6:

[0269] Server: Updates the user's ingredient list and sends the latest list information to the device.

[0270] Step 7:

[0271] Device: Display the latest ingredient list on the screen.

[0272] Step 8:

[0273] User: View updated ingredient list in the app.

[0274] ---

[0275] AI menu suggestion function

[0276] Step 1:

[0277] User: Tap the "Menu Suggestion" button on the app.

[0278] Step 2:

[0279] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[0280] Step 3:

[0281] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[0282] Step 4:

[0283] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[0284] Step 5:

[0285] Server: Sends the generated menu options to the terminal.

[0286] Step 6:

[0287] Terminal: Presents multiple menu options to the user.

[0288] Step 7:

[0289] User: Select the desired menu from the displayed menu.

[0290] ---

[0291] Automatic recipe generation function

[0292] Step 1:

[0293] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[0294] Step 2:

[0295] Terminal: Sends the selected menu information and the user's family composition information to the server.

[0296] Step 3:

[0297] Server: Based on the selected menu, launches the recipe generation engine.

[0298] Step 4:

[0299] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members and their preferences (e.g., "300g chicken, 1 / 2 teaspoon salt").

[0300] Step 5:

[0301] Server: Sends the generated recipe to the device.

[0302] Step 6:

[0303] Terminal: Displays the generated recipe and cooking instructions to the user.

[0304] Step 7:

[0305] User: Cooks using the displayed recipe as a reference.

[0306] ---

[0307] Advertisement display function

[0308] Step 1:

[0309] Users: See ads while browsing recipes.

[0310] Step 2:

[0311] Device: Sends browsing information to the server.

[0312] Step 3:

[0313] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[0314] Step 4:

[0315] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[0316] Step 5:

[0317] Server: Sends the selected advertisement to the device.

[0318] Step 6:

[0319] Device: Display the received advertisement on the recipe viewing screen.

[0320] Step 7:

[0321] User: View the ad displayed and click to view more information if necessary.

[0322] ---

[0323] Shopping list generator

[0324] Step 1:

[0325] User: Requests to purchase desired ingredients.

[0326] Step 2:

[0327] Terminal: Sends a purchase request to the server.

[0328] Step 3:

[0329] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[0330] Step 4:

[0331] Server: Sends the generated purchase link to the device.

[0332] Step 5:

[0333] Device: Show the user a purchase link.

[0334] Step 6:

[0335] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[0336] ---

[0337] Use of emotion engine

[0338] Step 1:

[0339] User: Use the camera and microphone in the app to input their own emotions.

[0340] Step 2:

[0341] Terminal: Sends the input emotion data to the server.

[0342] Step 3:

[0343] Server: Utilizes the emotion engine to analyze the user's emotional state.

[0344] Step 4:

[0345] Server: Optimize menu suggestions and advertisement displays based on analyzed emotional information.

[0346] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[0347] Through these steps, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[0348] Example 2

[0349] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] Conventional food management systems have struggled to efficiently manage ingredients in the refrigerator, suggest optimal meals based on the user's family composition and preferences, and smoothly purchase the necessary ingredients. Another issue is that they were unable to optimize suggestions or advertisements based on the user's emotional state, and there was a lack of means to improve the user experience.

[0351] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisements, and a menu suggestion means for generating menu candidates using an AI model based on the acquired ingredient list and user profile information. This allows the user to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing the emotion engine, the user experience can be further improved.

[0352] "Food in the refrigerator" refers to food and beverages stored in the refrigerator or freezer.

[0353] The "food management means" is a means for efficiently listing the food ingredients stored in the refrigerator and managing that information.

[0354] "External resources" are information sources that exist outside the system, such as databases and websites on the Internet.

[0355] The "recipe acquisition means" is a means for acquiring cooking recipe information from an external resource.

[0356] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the user's family composition and preferences.

[0357] The "shopping list generating means" is a means for automatically generating a list for purchasing necessary ingredients.

[0358] "Advertising presentation means" refers to a means for selecting and presenting appropriate advertisements to users.

[0359] The "receipt analysis means" is a means for analyzing the receipt image taken by the user and automatically updating the information on the purchased ingredients.

[0360] "Emotion analysis means" refers to means for analyzing a user's emotional state and optimizing suggestions and advertisements based on that.

[0361] The "menu suggestion means" is a means of generating optimal menu candidates using an AI model based on the acquired ingredient list and user profile information.

[0362] "Profile information" refers to information such as the user's family structure, preferences, cooking skills, etc.

[0363] An "AI model" is a collection of algorithms and data that uses machine learning techniques to perform a specific task.

[0364] This invention is a system that manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion analysis engine can further improve the user experience.

[0365] Hardware and Software

[0366] This system uses the following hardware and software:

[0367] Smartphone: Take and upload receipt images, operate the app.

[0368] Server: Data processing and storage.

[0369] OCR engine: Receipt image analysis (e.g., Tesseract or Google Cloud Vision API).

[0370] Recipe generation engine: Automatically generate the required ingredient quantities and cooking instructions (e.g., custom algorithms using Python libraries).

[0371] AI Model: A generative AI model for menu suggestions (e.g., TENSORFLOW® or PyTorch).

[0372] Sentiment analysis engine: Analyzes the user's emotional state (e.g., Microsoft® Azure® Emotion API).

[0373] System Operation

[0374] Food management function

[0375] Users take a photo of their supermarket receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This ensures that users always have the most up-to-date ingredient list.

[0376] Menu suggestion function

[0377] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, preferences, cooking skills, etc.) to the server. The server uses this information to use an AI model to generate menu suggestions that suit the user. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0378] Automatic recipe generation function

[0379] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[0380] Advertisement display function

[0381] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it might select ingredients on sale or convenient cooking tools. Appropriate advertisements are selected, sent to the device, and then presented to the user.

[0382] Shopping list generator

[0383] When a user makes a request to purchase desired ingredients, the device sends the information to the server, which then generates a purchase link using the API of a partner online shopping site. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[0384] Use of emotion engine

[0385] Users can input their emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust portion sizes based on the user's profile information.

[0386] Examples and prompts

[0387] Example: When a user takes a photo of a receipt for groceries purchased at the supermarket and uploads it to the app, the system registers the information and displays the latest list of ingredients.

[0388] Example prompt: "I just uploaded the receipt for the groceries I bought at the supermarket."

[0389] Example: When a user requests a menu suggestion, the system will suggest the following dishes based on the ingredients in the refrigerator and the user's preferences: "curry rice," "Chinese rice bowl," and "pasta."

[0390] Example prompt: "Can you suggest some good dishes for dinner tonight?"

[0391] Example: When a user selects the recipe for "curry rice," the system generates ingredient quantities and cooking instructions for four people and sends them to the user.

[0392] Example prompt: "Generate a recipe for curry rice."

[0393] This system allows users to check the latest ingredient list, efficiently manage ingredients, suggest optimal meals, and smoothly purchase the ingredients they need.The system also further improves the user experience by customizing the system based on the user's emotional state.

[0394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0395] System program processing flow

[0396] Step 1: Upload a receipt image

[0397] What happens: The user takes a photo of the receipt with their smartphone camera.

[0398] Input: Receipt image.

[0399] What happens: A user taps the "Upload Receipt" button in the app.

[0400] Output: The receipt image is sent from the device to the server.

[0401] Step 2: OCR analysis and updating the ingredients list

[0402] Specific operation: The server passes the received receipt image to the OCR engine.

[0403] Input: Receipt image.

[0404] How it works: The server uses an OCR engine (e.g., Tesseract or Google Cloud Vision API) to analyze the receipt image and extract information about the ingredients purchased.

[0405] Output: The extracted ingredient information is saved in the database. An updated ingredient list is generated.

[0406] Step 3: Menu suggestions

[0407] Specific behavior: The user taps the "Menu Suggestion" button.

[0408] Input: Ingredient list, profile information (family composition, preferences, cooking skills, etc.).

[0409] How it works: The device sends the ingredient list and profile information to the server, which uses the AI ​​model to generate meal suggestions.

[0410] Output: Multiple menu options (e.g., "curry rice," "Chinese rice bowl," and "pasta") are generated and sent to the terminal.

[0411] Step 4: Auto-generate and adjust recipes

[0412] Specific behavior: The user selects the desired menu.

[0413] Input: Selected menu information.

[0414] How it works: The device sends the selected menu information to the server, which uses a recipe generation engine to automatically generate the required ingredient quantities and cooking instructions. The quantities are adjusted to suit the number of family members and their preferences.

[0415] Output: The generated recipe (ingredients and cooking instructions) is sent to the device.

[0416] Step 5: Ad Presentation

[0417] Specific operation: When a user is browsing a recipe, the device sends that information to the server.

[0418] Input: User profile, recipe viewing information.

[0419] How it works: The server selects relevant ads based on the user's profile and browsing information.

[0420] Output: Appropriate advertisements (e.g., special offers on ingredients or convenient cooking tools) are sent to the device and displayed.

[0421] Step 6: Generate a shopping list

[0422] Specific behavior: The user makes a request to purchase the desired ingredients.

[0423] Input: Shopping request information.

[0424] How it works: The device sends the request information to the server, which then generates a purchase link using the API of a partner online shopping site.

[0425] Output: The generated purchase link is sent to the device, allowing the user to make a purchase on the associated site.

[0426] Step 7: Use the Emotion Engine

[0427] What it does: Users input their emotions using the in-app camera or microphone.

[0428] Input: Emotion data (images, audio).

[0429] How it works: The device sends this emotion data to a server, which uses an emotion analysis engine (e.g., Microsoft Azure Emotion API) to analyze the emotional state.

[0430] Output: Optimal meal suggestions and advertisements based on the emotional state are generated and sent to the device.

[0431] (Application example 2)

[0432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0433] Conventional food management and recipe suggestion systems were able to properly manage ingredients in the refrigerator and suggest recipes tailored to the user's preferences and family composition, but they were unable to suggest menus based on the user's emotional state or arrange for delivery of missing ingredients immediately. Furthermore, they lacked functionality that responded to the user's emotional state, such as cooking suggestions and shopping list generation that took into account the user's stress and fatigue, and therefore did not adequately improve the user experience. To solve these issues, there is a need for the development of a system that incorporates emotion analysis to suggest appropriate menus, display advertisements, and arrange for delivery based on the user's emotional state.

[0434] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0435] In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing the necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on the purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on the emotion, and a delivery arrangement means for arranging delivery of missing ingredients based on the ingredient list and the proposed menu. This allows for efficient management of ingredients in the user's refrigerator and enables optimal menu suggestions based on the emotional state and immediate delivery arrangements for missing ingredients.

[0436] The "food management means" is a means for listing the food ingredients in the refrigerator and efficiently managing the food ingredients that the user has.

[0437] The "recipe acquisition means" is a means for acquiring a cooking recipe from an external resource and incorporating the recipe into the system.

[0438] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the family structure and the user's preferences.

[0439] The "shopping list generating means" is a means for automatically generating a shopping list for purchasing necessary ingredients.

[0440] "Advertisement presentation means" refers to a means for appropriately presenting relevant advertisements to users.

[0441] The "receipt analysis means" is a means for analyzing an image of a receipt photographed by a user and automatically updating information about purchased ingredients.

[0442] The "emotion analysis means" is a means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on that emotion.

[0443] The "delivery arrangement means" is a means for arranging for missing ingredients to be delivered by a delivery service based on the ingredient list and the proposed menu.

[0444] The system that realizes this application example is configured as follows: First, the user takes a photo of the receipt with their smartphone camera and taps the "Upload Receipt" button to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image using an OCR engine (such as Tesseract OCR), extracts information about the ingredients purchased, and stores that information in a database. This allows the user to always check the latest ingredient list.

[0445] Next, when the user requests a "menu suggestion," the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) and emotional state to the server. The server uses this information to generate menu suggestions using a generative AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0446] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user uses it as a reference when cooking.

[0447] Furthermore, when a user browses a recipe, the device sends that information to the server, which then selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools, and send the appropriate advertisements to the device to present to the user.

[0448] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and sends the generated link to the device, allowing the user to easily make a purchase on the related site.

[0449] Users can also input their own emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion analysis engine (such as AWS® Comprehend or Google Cloud Natural Language) to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[0450] Furthermore, the system also has a function to arrange for delivery of missing ingredients based on the list of ingredients in the user's refrigerator and the proposed menu.The server identifies missing ingredients based on the ingredient list and the selected menu, and sends a request to the delivery service via API.

[0451] For example, if the emotion analysis engine detects that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time, which is used to improve the system.

[0452] Prompt Sentence Examples

[0453] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[0454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0455] Step 1:

[0456] To manage the ingredients in their refrigerator, users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image.

[0457] Input: A captured image of the receipt

[0458] What happens: The user takes a photo of the receipt with their smartphone camera and taps a button in the app.

[0459] Step 2:

[0460] The terminal transmits the receipt image to the server.

[0461] Input: Receipt image

[0462] Specific operation: The receipt image is sent from the device to the server.

[0463] Step 3:

[0464] The server analyzes the received receipt image using an OCR engine (e.g., Tesseract OCR), extracts information about the purchased ingredients, and stores that information in a database.

[0465] Input: Receipt image

[0466] Data processing and calculation: The receipt image is analyzed using an OCR engine to generate text data.

[0467] Output: Extracted ingredient information

[0468] Specific operation: The server uses an OCR engine to analyze the receipt image and saves the ingredient information in a database.

[0469] Step 4:

[0470] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, and emotional state) to the server.

[0471] Input: Ingredient list, user profile information, emotional state

[0472] Specific operation: The user requests a "menu suggestion" from within the app, and the device sends the necessary information to the server.

[0473] Step 5:

[0474] The server uses a generative AI model to generate menu suggestions based on the submitted ingredient list, profile information, and emotional state, and sends them to the device.

[0475] Input: Ingredient list, profile information, emotional state

[0476] Data processing and calculation: The generative AI model generates optimal menu candidates based on the input data.

[0477] Output: Menu candidate list

[0478] Specific operation: The server runs the generative AI model and sends menu suggestions to the device.

[0479] Step 6:

[0480] When the user selects the desired menu, the terminal transmits the information to the server.

[0481] Input: Selected menu information

[0482] Specific operation: The user selects a menu and the device sends that information to the server.

[0483] Step 7:

[0484] Based on the selected menu, the server uses a recipe generation engine to automatically generate the quantities of ingredients and cooking instructions required, and sends them to the terminal.

[0485] Input: Selected menu information

[0486] Data processing and calculation: The recipe generation engine generates quantities and cooking instructions based on the menu.

[0487] Output: Recipe with portion adjustments

[0488] Specific operation: The server executes the recipe generation engine and sends the generated recipe to the device.

[0489] Step 8:

[0490] When a user is browsing recipes, the device sends that information to the server, which then selects appropriate advertisements and sends them to the device.

[0491] Input: Recipe viewing information

[0492] Data processing and calculation: The server selects appropriate advertisements based on the advertisement prediction model.

[0493] Output: Selected ads

[0494] Specific operation: The device sends recipe viewing information, and the server selects appropriate advertisements and sends them to the device.

[0495] Step 9:

[0496] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and send it to the device.

[0497] Input: Desired ingredient information

[0498] Data processing and calculation: The server generates a purchase link using the API of the online shopping site.

[0499] Output: Purchase link

[0500] Specific operation: The device sends a request to purchase ingredients, and the server generates a purchase link and sends it to the device.

[0501] Step 10:

[0502] Users can input their emotions using the in-app camera or microphone, and the device sends this information to a server, which then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state.

[0503] Input: Emotion data

[0504] Data processing and calculation: The server analyzes the emotion data using an emotion analysis engine.

[0505] Output: Emotion analysis results

[0506] Specific operation: The user inputs their emotions, the device sends them to the server, and the server analyzes them.

[0507] Step 11:

[0508] The server optimizes menu suggestions and advertisement displays based on the emotional state.

[0509] Input: Sentiment analysis results

[0510] Data processing and calculation: Propose optimal menus and select advertisements based on emotions.

[0511] Output: Optimized menu suggestions and advertisements

[0512] Specific operation: The server optimizes menu suggestions and advertisement display based on the results of sentiment analysis.

[0513] Step 12:

[0514] The server sends an API request to arrange for delivery of any missing ingredients based on the ingredient list and the proposed menu.

[0515] Input: Ingredient list, suggested meal

[0516] Data processing and calculation: Generates API requests for delivery arrangements.

[0517] Output: Delivery arrangement completion notification

[0518] Specific operation: The server sends an API request to the delivery service to complete the arrangement.

[0519] Prompt Sentence Examples

[0520] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[0521] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0522] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0523] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0524] [Second embodiment]

[0525] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0526] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0527] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0528] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0529] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0530] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0531] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0532] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0533] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0534] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0535] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0536] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0537] MODE FOR CARRYING OUT THE INVENTION

[0538] The system of the present invention has the functions of managing ingredients in a refrigerator, proposing optimal menus based on the user's family structure and preferences, generating a shopping list for purchasing necessary ingredients, and providing appropriate advertisements.

[0539] Food management function

[0540] It provides a means to automatically capture purchased ingredient information using a receipt photographed by the user. The user photographs the receipt with their smartphone camera and uploads the image to the app. At this time, the device sends the photographed image to a server. The server analyzes the receipt image, extracts purchased ingredient information, and stores that information in a database. This allows the user to always check the latest ingredient list.

[0541] AI menu suggestion function

[0542] When a user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate an optimal menu using an AI model. It generates multiple menu options and creates a list of ingredients required for each. This allows the user to choose from multiple menu options.

[0543] Automatic recipe generation function

[0544] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities can be adjusted to suit the number of family members and preferences. The generated recipe is sent to the device and can be viewed by the user.

[0545] Advertisement display function

[0546] As a user browses recipes, their device sends that information to a server, which uses the user's profile and recent browsing information to select relevant ads, such as special offers on ingredients or convenient cooking tools. Once a suitable ad is selected, it is sent to the device and presented to the user.

[0547] Shopping list generator

[0548] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a link to purchase the ingredient. The server then sends the link to the device, allowing the user to easily click the link to make a purchase on the relevant site.

[0549] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the ingredients they need. Furthermore, by providing appropriate advertisements, it is possible to provide new value to users.

[0550] The processing flow will be explained below.

[0551] Food management function

[0552] Step 1:

[0553] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[0554] Step 2:

[0555] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user confirms it, the image is sent to the server by pressing the send button.

[0556] Step 3:

[0557] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[0558] Step 4:

[0559] Server: The OCR engine extracts text information from the receipt image and generates data in the format of "3 apples" or "1 liter of milk."

[0560] Step 5:

[0561] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[0562] Step 6:

[0563] Server: Updates the user's ingredient list and sends the latest list information to the device.

[0564] Step 7:

[0565] Device: Display the latest ingredient list on the screen.

[0566] Step 8:

[0567] User: View updated ingredient list in the app.

[0568] ---

[0569] AI menu suggestion function

[0570] Step 1:

[0571] User: Tap the "Menu Suggestion" button on the app.

[0572] Step 2:

[0573] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[0574] Step 3:

[0575] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[0576] Step 4:

[0577] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[0578] Step 5:

[0579] Server: Sends the generated menu options to the terminal.

[0580] Step 6:

[0581] Terminal: Presents multiple menu options to the user.

[0582] Step 7:

[0583] User: Select the desired menu from the displayed menu.

[0584] ---

[0585] Automatic recipe generation function

[0586] Step 1:

[0587] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[0588] Step 2:

[0589] Terminal: Sends the selected menu information and the user's family composition information to the server.

[0590] Step 3:

[0591] Server: Based on the selected menu, launches the recipe generation engine.

[0592] Step 4:

[0593] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members (e.g., "300g chicken, 1 / 2 teaspoon salt...").

[0594] Step 5:

[0595] Server: Sends the generated recipe to the device.

[0596] Step 6:

[0597] Terminal: Displays the generated recipe and cooking instructions to the user.

[0598] Step 7:

[0599] User: Cooks using the displayed recipe as a reference.

[0600] ---

[0601] Advertisement display function

[0602] Step 1:

[0603] Users: See ads while browsing recipes.

[0604] Step 2:

[0605] Device: Sends browsing information to the server.

[0606] Step 3:

[0607] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[0608] Step 4:

[0609] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[0610] Step 5:

[0611] Server: Sends the selected advertisement to the device.

[0612] Step 6:

[0613] Device: Display the received advertisement on the recipe viewing screen.

[0614] Step 7:

[0615] User: View the ad displayed and click to view more information if necessary.

[0616] ---

[0617] Shopping list generator

[0618] Step 1:

[0619] User: Requests to purchase desired ingredients.

[0620] Step 2:

[0621] Terminal: Sends a purchase request to the server.

[0622] Step 3:

[0623] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[0624] Step 4:

[0625] Server: Sends the generated purchase link to the device.

[0626] Step 5:

[0627] Device: Show the user a purchase link.

[0628] Step 6:

[0629] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[0630] Example 1

[0631] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] Conventional refrigerator ingredient management and menu suggestion systems required a lot of manual input work, which was time-consuming for users. They also lacked the functionality to suggest menus suited to the user's family structure and preferences, or to easily manage the purchasing procedures for necessary ingredients. Furthermore, there was a need for a way to provide new value by providing appropriate advertisements to users.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0634] In this invention, the server includes a means for analyzing receipts photographed by the user and automatically capturing purchased ingredient information, a means for proposing optimal menus based on the user's family structure and preferences, and a means for generating menus using an AI model based on the ingredient list and profile information stored on the server. This allows the user to efficiently manage ingredients in the refrigerator, receive suggestions for optimal menus suited to their family structure and preferences, and smoothly purchase the necessary ingredients.

[0635] "Means of analyzing receipts photographed by users and automatically importing information about purchased ingredients" refers to a function that analyzes images of receipts photographed by users on devices such as smartphones, extracts information about purchased ingredients from the images, and automatically imports the information into a database.

[0636] "Means for suggesting optimal menus based on family structure and preferences" refers to a function that automatically generates and suggests optimal menus based on the user's family structure (for example, number of people and age range) and preferences (preferred dishes and ingredients).

[0637] "Means for generating menus using an AI model based on ingredient lists and profile information stored on a server" refers to a function that stores ingredient lists and user profile information entered by the user or automatically obtained on a server, and automatically generates optimal menus using an AI model based on that information.

[0638] The "recipe generation means for automatically generating the quantities of ingredients required and cooking procedures" is a function for automatically generating and providing the specific quantities of ingredients required and cooking procedures based on the selected menu.

[0639] "Means for selecting relevant advertisements based on the generated recipe and presenting them to the user" refers to a function for selecting highly relevant advertisements (for example, information on special sales of ingredients or advertisements for cooking utensils) based on automatically generated recipe information and displaying them to the user.

[0640] "Means for generating a purchase link using the API of an affiliated online shopping site" refers to a function that uses the API of an affiliated online shopping site to generate a purchase link and provide that link to the user when the user purchases the ingredients they desire.

[0641] MODE FOR CARRYING OUT THE INVENTION

[0642] The system of the present invention utilizes data processing and AI technology to seamlessly perform a series of tasks, from managing ingredients in the refrigerator to suggesting menus, creating recipes, generating shopping lists, and providing advertisements.

[0643] Food management function

[0644] First, the user takes a photo of the receipt for the ingredients they purchased using their smartphone camera. They then upload the image of the receipt to the app. The device then sends the uploaded receipt image to the server. The server uses image analysis software such as Google Cloud Vision API to extract information about the ingredients purchased from the receipt image. The extracted information about the ingredients is saved in a database. This allows the user to always check the latest list of ingredients on the app. As a specific example, when a user takes a photo of a receipt from a supermarket purchase and uploads it to the app, the server automatically extracts and saves information such as "1L of milk, 10 eggs, 4 tomatoes."

[0645] AI menu suggestion function

[0646] When a user requests a "menu suggestion" within the app, the device sends the latest ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. Based on this information, the server uses a generative AI model such as OpenAI's GPT-3 to generate the optimal menu. A specific prompt might be something like, "My family consists of three people, I like Japanese food, and I have chicken and potatoes. Please suggest a menu for me." The server then generates the optimal menu from multiple candidates and provides it to the user. This function allows users to easily obtain the optimal menu.

[0647] Automatic recipe generation function

[0648] When a user selects a desired menu from the menu options provided, the device sends that information to the server. The server then uses IBM Watson's recipe generation engine to automatically generate the specific amounts of ingredients needed and cooking instructions based on the selected menu. The generated recipe is sent to the device and can be viewed by the user on the app. For example, a recipe for "Teriyaki Chicken" may include details such as "200g of chicken, 2 tablespoons of soy sauce, and 1 tablespoon of sugar."

[0649] Advertisement display function

[0650] When a user is browsing a recipe generated by the app, the device sends browsing information to the server. The server selects relevant advertisements based on the user's profile and recent browsing information. These advertisements may include, for example, special sale ingredients or convenient cooking utensils. The selected advertisements are sent to the device in a timely manner and presented to the user. For example, if a user is browsing a recipe for "Mapo Tofu," advertisements for special sale tofu or new cooking utensils may be displayed in the app.

[0651] Shopping list generator

[0652] When a user sends a request to purchase necessary ingredients, the device sends that information to the server. The server uses the API of an affiliated online shopping site (e.g., shopping site API) to generate a purchase link for the necessary ingredients. The generated link is sent to the device, and the user can click on it to easily make a purchase on the related site. For example, in response to a request for "100g of tofu and 1 green onion," the generated purchase link is displayed in the app, and the user can click on it to smoothly make a purchase on the online shopping site.

[0653] The system of the present invention is designed to allow users to efficiently manage ingredients in their refrigerators, suggest optimal menus, and easily purchase the ingredients they need. Furthermore, by providing relevant advertisements, the system can provide new value to users.

[0654] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0655] Step 1: Take a photo of your receipt

[0656] Users take a photo of the receipt for the food they purchased using their smartphone camera, and this image becomes input data for the system.

[0657] Step 2: Upload a receipt image

[0658] The user opens the app and uploads a photo of the receipt, which becomes the input data sent to the server.

[0659] Step 3: Send the receipt image to the server

[0660] The device sends the uploaded receipt image to the server. The server uses image analysis software (e.g., image analysis API) to process the data and extract ingredient information from the receipt image. The input in this step is the receipt image, and the output is ingredient information.

[0661] Step 4: Store purchased ingredients in a database

[0662] The server stores the extracted ingredient information in a database. This process allows users to access the latest ingredient list. The input is the extracted ingredient information, and the output is the ingredient information stored in the database.

[0663] Step 5: Request menu suggestions

[0664] Users can request a "menu suggestion" within the app, which becomes new input data for the system.

[0665] Step 6: Submit user information

[0666] The device sends the latest ingredient list and user profile information (family composition, preferences, etc.) to the server. The ingredient list and profile information are input, and are used in the AI ​​model.

[0667] Step 7: Menu generation using AI models

[0668] The server uses a generative AI model to generate an optimal menu based on the input ingredient list and profile information. The specific input prompt is "Family size is 3 people, preference is Japanese food, ingredients available are chicken and potatoes, please suggest a menu." The output of this step is the generated menu.

[0669] Step 8: Offer menu options

[0670] The server sends the generated menu options to the terminal. The user selects the desired menu from these options. The input is the generated menu options, and the output is the menu options provided to the user.

[0671] Step 9: Send menu selection to server

[0672] When the user selects the desired menu, the terminal sends the selection information to the server. The input is the selected menu, which becomes the basis for recipe generation.

[0673] Step 10: Auto-generating recipes

[0674] The server uses a recipe generation engine (e.g., a recipe generation API) to automatically generate the required ingredient quantities and cooking instructions based on the selected menu. The input is the selected menu, and the output is the specific recipe information.

[0675] Step 11: User Provides Recipe Information

[0676] The server sends the generated recipe information to the device so that the user can view it in the app. The input is the automatically generated recipe information, and the output is the recipe information provided to the user.

[0677] Step 12: Send recipe viewing information to the server

[0678] When a user browses recipes on the app, the device sends browsing information to the server. The input is the user's browsing information, which is used as the basis for selecting ads.

[0679] Step 13: Select relevant ads

[0680] The server selects relevant advertisements based on browsing information and user profile. The input is browsing information and profile information, and the output is the selected advertisement.

[0681] Step 14: User Provides Ad Presentation

[0682] The server sends the selected advertisement to the terminal for presentation to the user. The input is the selected advertisement, and the output is the advertisement provided to the user.

[0683] Step 15: Send purchase request to server

[0684] When a user requests to purchase a desired ingredient, the device sends the information to the server. The input is the user's purchase request, which becomes the basis for generating a purchase link.

[0685] Step 16: Generate a Purchase Link

[0686] The server uses the API of an online shopping site to generate a purchase link for the required ingredients. The input is the purchase request, and the output is the generated purchase link.

[0687] Step 17: User Provided Purchase Link

[0688] The server sends the generated purchase link to the terminal, which the user can click to easily make a purchase on the related site. The input is the generated purchase link, and the output is the purchase link provided to the user.

[0689] (Application example 1)

[0690] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0691] In modern households, managing the ingredients in the refrigerator, proposing optimal menus, and purchasing ingredients are time-consuming and labor-intensive. This makes preparing home-cooked meals complicated, leading to unnecessary food waste and incorrect purchases. Furthermore, users have limited means of obtaining the information they need without receiving advertisements based on individual ingredients. Furthermore, the inability to smoothly integrate with food delivery services reduces the efficiency of home cooking. A new system is needed to solve these issues.

[0692] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0693] In this invention, the server includes an ingredient management unit for managing ingredients in the refrigerator, a recipe acquisition unit for acquiring recipes from external resources, a recipe adjustment unit for automatically adjusting recipe portion sizes based on family composition and preferences, a shopping list generation unit for generating a shopping list for purchasing necessary ingredients, an advertisement presentation unit for presenting advertisements to the user, a receipt analysis unit for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, a menu suggestion unit for generating an optimal menu based on the user's profile information using an AI model, and a delivery collaboration unit for ordering necessary ingredients based on the menu in collaboration with a food delivery service. This allows users to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase and deliver the necessary ingredients. Furthermore, providing appropriate advertisements to users encourages purchasing behavior and improves the efficiency of home cooking.

[0694] The "foodstuff management means" is a system for listing the foodstuffs in the refrigerator and constantly updating and managing the inventory status.

[0695] The "recipe acquisition means" is a mechanism for acquiring cooking recipe information from an external database, the Internet, etc.

[0696] The "recipe adjustment means" is a mechanism for automatically adjusting the amounts of ingredients in the acquired recipe according to the household composition and individual preferences.

[0697] The "shopping list generating means" is a mechanism for automatically generating a list for purchasing necessary ingredients.

[0698] An "advertising presentation means" is a mechanism for displaying highly relevant advertisements to users.

[0699] The "receipt analysis means" is a mechanism that analyzes receipt images taken by the user and automatically updates the list of purchased ingredients based on that information.

[0700] The "menu suggestion method" is a system that uses an AI model to calculate and suggest the optimal menu based on the user's profile information.

[0701] "Delivery collaboration means" is a system that links with a food delivery service to order and request delivery of the necessary ingredients based on the proposed menu.

[0702] MODE FOR CARRYING OUT THE INVENTION

[0703] The present invention is a system that efficiently manages ingredients in a refrigerator, proposes optimal menus, and allows necessary ingredients to be easily purchased and delivered. An embodiment of the present invention will be described below.

[0704] First, to manage the information on ingredients purchased by the user, the user takes a photo of the receipt with the smartphone camera and uploads the image to the app. At this time, the device sends the captured image to the server. The server performs image analysis (using OCR technology) to extract ingredient information from the receipt and saves it in a database. This function allows the user to always manage an up-to-date ingredient list.

[0705] Next, when the user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information to the server. Based on this information, the server uses a generative AI model to generate an optimal menu based on family composition and preferences. At this time, multiple menu options are generated, and the user can choose from these menu options.

[0706] After that, when the user selects the desired menu, the device sends that information to the server, which then uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions based on the selected menu, allowing users to easily obtain recipes that suit the number of family members and their preferences.

[0707] Additionally, when a user is browsing recipes, appropriate advertisements are displayed. The device sends recipe browsing information and the user's profile to the server, and the server selects and presents highly relevant advertisements (such as ingredients on sale or convenient cooking utensils) to the user. This encourages users to make purchases and provides them with even more useful information.

[0708] Furthermore, if the user requests to purchase ingredients, the device sends that information to the server, which then uses the API of a partner food delivery service to generate a link to purchase the ingredients, allowing the user to request delivery with just one click.

[0709] The hardware and software used to build this system includes:

[0710] Hardware: Smartphone (iPhone, Android device)

[0711] software:

[0712] Python: Backend processing for the application

[0713] OpenCV: Receipt image processing and analysis

[0714] Requests: Communication with the server

[0715] For example, a user can purchase new ingredients at the supermarket to prepare dinner and upload the receipt to the app. The app then uses AI to suggest the optimal dinner menu and places an order for the ingredients needed for the user's selected menu with a food delivery service. A detailed recipe is then displayed to help prepare dinner.

[0716] Example prompt sentence:

[0717] 1. "Take a photo of your receipt and upload it."

[0718] 2. "I'll suggest the perfect menu for tonight's dinner."

[0719] 3. "I can quickly order the ingredients I need for this menu."

[0720] 4. "Check out the cookware on sale."

[0721] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0722] Step 1:

[0723] The user takes a photo of the receipt with the smartphone camera. Then, they upload the image to the app. The input is the image of the receipt, and the output is its image data. The image data is sent from the device to the server.

[0724] Step 2:

[0725] The server analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the image data of the receipt, and the output is text data of the extracted ingredient information. This text data is stored in a database.

[0726] Step 3:

[0727] The user requests a "menu suggestion" within the app. The input is the user's request, profile information (family composition, preferences, etc.), and saved ingredient list. The output is this information sent to the server.

[0728] Step 4:

[0729] The server uses the received user profile information and ingredient list to generate an optimal meal plan using a generative AI model (e.g., GPT-3). The input is the profile information and ingredient list, and the output is multiple suggested meal options. These meal options are sent to the device.

[0730] Step 5:

[0731] The user selects the desired menu. The input is the user's menu selection information, and the output is the selection information sent to the server.

[0732] Step 6:

[0733] The server uses a recipe generation engine (e.g., Recipe API) to automatically generate the quantities of ingredients and cooking instructions based on the selected menu. The input is the selected menu information, and the output is the generated recipe information. This recipe information is sent to the device.

[0734] Step 7:

[0735] When a user browses a recipe, the device sends the browsing information to the server. The input is the browsing information and the user's profile information, and the output is the information sent to the server.

[0736] Step 8:

[0737] The server uses the profile information and browsing information to search a database for relevant advertisements (e.g., special offers on ingredients or cooking utensils). The input is the profile information and browsing information, and the output is appropriate advertising information. This advertising information is sent to the terminal and displayed to the user.

[0738] Step 9:

[0739] A user submits an order request to purchase the ingredients they need. The input is the purchase request, and the output is the request information sent to the server.

[0740] Step 10:

[0741] The server uses the API of a partner food delivery service to generate a purchase link for the required ingredients. The input is the purchase request information, and the output is the generated purchase link. This link is sent to the terminal, and the user can click on it to use the delivery service.

[0742] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0743] MODE FOR CARRYING OUT THE INVENTION

[0744] The system of the present invention manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion engine can further improve the user experience.

[0745] Food management function

[0746] Users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This allows users to always check the latest ingredient list.

[0747] AI menu suggestion function

[0748] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate menu suggestions using an AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0749] Automatic recipe generation function

[0750] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[0751] Advertisement display function

[0752] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools. Once the appropriate advertisement is selected, it is sent to the device and presented to the user.

[0753] Shopping list generator

[0754] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[0755] Use of emotion engine

[0756] Users can input their emotions using the app's camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[0757] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[0758] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[0759] The processing flow will be explained below.

[0760] A system that combines emotion engines

[0761] Food management function

[0762] Step 1:

[0763] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[0764] Step 2:

[0765] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user has checked it, the image is sent to the server when the user presses the send button.

[0766] Step 3:

[0767] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[0768] Step 4:

[0769] Server: The OCR engine extracts text information from the receipt image and generates data such as "3 apples" and "1 liter of milk."

[0770] Step 5:

[0771] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[0772] Step 6:

[0773] Server: Updates the user's ingredient list and sends the latest list information to the device.

[0774] Step 7:

[0775] Device: Display the latest ingredient list on the screen.

[0776] Step 8:

[0777] User: View updated ingredient list in the app.

[0778] ---

[0779] AI menu suggestion function

[0780] Step 1:

[0781] User: Tap the "Menu Suggestion" button on the app.

[0782] Step 2:

[0783] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[0784] Step 3:

[0785] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[0786] Step 4:

[0787] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[0788] Step 5:

[0789] Server: Sends the generated menu options to the terminal.

[0790] Step 6:

[0791] Terminal: Presents multiple menu options to the user.

[0792] Step 7:

[0793] User: Select the desired menu from the displayed menu.

[0794] ---

[0795] Automatic recipe generation function

[0796] Step 1:

[0797] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[0798] Step 2:

[0799] Terminal: Sends the selected menu information and the user's family composition information to the server.

[0800] Step 3:

[0801] Server: Based on the selected menu, launches the recipe generation engine.

[0802] Step 4:

[0803] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members and their preferences (e.g., "300g chicken, 1 / 2 teaspoon salt").

[0804] Step 5:

[0805] Server: Sends the generated recipe to the device.

[0806] Step 6:

[0807] Terminal: Displays the generated recipe and cooking instructions to the user.

[0808] Step 7:

[0809] User: Cooks using the displayed recipe as a reference.

[0810] ---

[0811] Advertisement display function

[0812] Step 1:

[0813] Users: See ads while browsing recipes.

[0814] Step 2:

[0815] Device: Sends browsing information to the server.

[0816] Step 3:

[0817] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[0818] Step 4:

[0819] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[0820] Step 5:

[0821] Server: Sends the selected advertisement to the device.

[0822] Step 6:

[0823] Device: Display the received advertisement on the recipe viewing screen.

[0824] Step 7:

[0825] User: View the ad displayed and click to view more information if necessary.

[0826] ---

[0827] Shopping list generator

[0828] Step 1:

[0829] User: Requests to purchase desired ingredients.

[0830] Step 2:

[0831] Terminal: Sends a purchase request to the server.

[0832] Step 3:

[0833] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[0834] Step 4:

[0835] Server: Sends the generated purchase link to the device.

[0836] Step 5:

[0837] Device: Show the user a purchase link.

[0838] Step 6:

[0839] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[0840] ---

[0841] Use of emotion engine

[0842] Step 1:

[0843] User: Use the camera and microphone in the app to input their own emotions.

[0844] Step 2:

[0845] Terminal: Sends the input emotion data to the server.

[0846] Step 3:

[0847] Server: Utilizes the emotion engine to analyze the user's emotional state.

[0848] Step 4:

[0849] Server: Optimize menu suggestions and advertisement displays based on analyzed emotional information.

[0850] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[0851] Through these steps, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[0852] Example 2

[0853] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0854] Conventional food management systems have struggled to efficiently manage ingredients in the refrigerator, suggest optimal meals based on the user's family composition and preferences, and smoothly purchase the necessary ingredients. Another issue is that they were unable to optimize suggestions or advertisements based on the user's emotional state, and there was a lack of means to improve the user experience.

[0855] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisements, and a menu suggestion means for generating menu candidates using an AI model based on the acquired ingredient list and user profile information. This allows the user to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing the emotion engine, the user experience can be further improved.

[0856] "Food in the refrigerator" refers to food and beverages stored in the refrigerator or freezer.

[0857] The "food management means" is a means for efficiently listing the food ingredients stored in the refrigerator and managing that information.

[0858] "External resources" are information sources that exist outside the system, such as databases and websites on the Internet.

[0859] The "recipe acquisition means" is a means for acquiring cooking recipe information from an external resource.

[0860] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the user's family composition and preferences.

[0861] The "shopping list generating means" is a means for automatically generating a list for purchasing necessary ingredients.

[0862] "Advertising presentation means" refers to a means for selecting and presenting appropriate advertisements to users.

[0863] The "receipt analysis means" is a means for analyzing the receipt image taken by the user and automatically updating the information on the purchased ingredients.

[0864] "Emotion analysis means" refers to means for analyzing a user's emotional state and optimizing suggestions and advertisements based on that.

[0865] The "menu suggestion means" is a means of generating optimal menu candidates using an AI model based on the acquired ingredient list and user profile information.

[0866] "Profile information" refers to information such as the user's family structure, preferences, cooking skills, etc.

[0867] An "AI model" is a collection of algorithms and data that uses machine learning techniques to perform a specific task.

[0868] This invention is a system that manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion analysis engine can further improve the user experience.

[0869] Hardware and Software

[0870] This system uses the following hardware and software:

[0871] Smartphone: Take and upload receipt images, operate the app.

[0872] Server: Data processing and storage.

[0873] OCR engine: Receipt image analysis (e.g., Tesseract or Google Cloud Vision API).

[0874] Recipe generation engine: Automatically generate the required ingredient quantities and cooking instructions (e.g., custom algorithms using Python libraries).

[0875] AI Model: A generative AI model (e.g., TensorFlow or PyTorch) for menu suggestions.

[0876] Sentiment analysis engine: Analyzes the user's emotional state (e.g., Microsoft Azure Emotion API).

[0877] System Operation

[0878] Food management function

[0879] Users take a photo of their supermarket receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This ensures that users always have the most up-to-date ingredient list.

[0880] Menu suggestion function

[0881] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, preferences, cooking skills, etc.) to the server. The server uses this information to use an AI model to generate menu suggestions that suit the user. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0882] Automatic recipe generation function

[0883] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[0884] Advertisement display function

[0885] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it might select ingredients on sale or convenient cooking tools. Appropriate advertisements are selected, sent to the device, and then presented to the user.

[0886] Shopping list generator

[0887] When a user makes a request to purchase desired ingredients, the device sends the information to the server, which then generates a purchase link using the API of a partner online shopping site. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[0888] Use of emotion engine

[0889] Users can input their emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust portion sizes based on the user's profile information.

[0890] Examples and prompts

[0891] Example: When a user takes a photo of a receipt for groceries purchased at the supermarket and uploads it to the app, the system registers the information and displays the latest list of ingredients.

[0892] Example prompt: "I just uploaded the receipt for the groceries I bought at the supermarket."

[0893] Example: When a user requests a menu suggestion, the system will suggest the following dishes based on the ingredients in the refrigerator and the user's preferences: "curry rice," "Chinese rice bowl," and "pasta."

[0894] Example prompt: "Can you suggest some good dishes for dinner tonight?"

[0895] Example: When a user selects the recipe for "curry rice," the system generates ingredient quantities and cooking instructions for four people and sends them to the user.

[0896] Example prompt: "Generate a recipe for curry rice."

[0897] This system allows users to check the latest ingredient list, efficiently manage ingredients, suggest optimal meals, and smoothly purchase the ingredients they need.The system also further improves the user experience by customizing the system based on the user's emotional state.

[0898] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0899] System program processing flow

[0900] Step 1: Upload a receipt image

[0901] What happens: The user takes a photo of the receipt with their smartphone camera.

[0902] Input: Receipt image.

[0903] What happens: A user taps the "Upload Receipt" button in the app.

[0904] Output: The receipt image is sent from the device to the server.

[0905] Step 2: OCR analysis and updating the ingredients list

[0906] Specific operation: The server passes the received receipt image to the OCR engine.

[0907] Input: Receipt image.

[0908] How it works: The server uses an OCR engine (e.g., Tesseract or Google Cloud Vision API) to analyze the receipt image and extract information about the ingredients purchased.

[0909] Output: The extracted ingredient information is saved in the database. An updated ingredient list is generated.

[0910] Step 3: Menu suggestions

[0911] Specific behavior: The user taps the "Menu Suggestion" button.

[0912] Input: Ingredient list, profile information (family composition, preferences, cooking skills, etc.).

[0913] How it works: The device sends the ingredient list and profile information to the server, which uses the AI ​​model to generate meal suggestions.

[0914] Output: Multiple menu options (e.g., "curry rice," "Chinese rice bowl," and "pasta") are generated and sent to the terminal.

[0915] Step 4: Auto-generate and adjust recipes

[0916] Specific behavior: The user selects the desired menu.

[0917] Input: Selected menu information.

[0918] How it works: The device sends the selected menu information to the server, which uses a recipe generation engine to automatically generate the required ingredient quantities and cooking instructions. The quantities are adjusted to suit the number of family members and their preferences.

[0919] Output: The generated recipe (ingredients and cooking instructions) is sent to the device.

[0920] Step 5: Ad Presentation

[0921] Specific operation: When a user is browsing a recipe, the device sends that information to the server.

[0922] Input: User profile, recipe viewing information.

[0923] How it works: The server selects relevant ads based on the user's profile and browsing information.

[0924] Output: Appropriate advertisements (e.g., special offers on ingredients or convenient cooking tools) are sent to the device and displayed.

[0925] Step 6: Generate a shopping list

[0926] Specific behavior: The user makes a request to purchase the desired ingredients.

[0927] Input: Shopping request information.

[0928] How it works: The device sends the request information to the server, which then generates a purchase link using the API of a partner online shopping site.

[0929] Output: The generated purchase link is sent to the device, allowing the user to make a purchase on the associated site.

[0930] Step 7: Use the Emotion Engine

[0931] What it does: Users input their emotions using the in-app camera or microphone.

[0932] Input: Emotion data (images, audio).

[0933] How it works: The device sends this emotion data to a server, which uses an emotion analysis engine (e.g., Microsoft Azure Emotion API) to analyze the emotional state.

[0934] Output: Optimal meal suggestions and advertisements based on the emotional state are generated and sent to the device.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0937] Conventional food management and recipe suggestion systems were able to properly manage ingredients in the refrigerator and suggest recipes tailored to the user's preferences and family composition, but they were unable to suggest menus based on the user's emotional state or arrange for delivery of missing ingredients immediately. Furthermore, they lacked functionality that responded to the user's emotional state, such as cooking suggestions and shopping list generation that took into account the user's stress and fatigue, and therefore did not adequately improve the user experience. To solve these issues, there is a need for the development of a system that incorporates emotion analysis to suggest appropriate menus, display advertisements, and arrange for delivery based on the user's emotional state.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0939] In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing the necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on the purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on the emotion, and a delivery arrangement means for arranging delivery of missing ingredients based on the ingredient list and the proposed menu. This allows for efficient management of ingredients in the user's refrigerator and enables optimal menu suggestions based on the emotional state and immediate delivery arrangements for missing ingredients.

[0940] The "food management means" is a means for listing the food ingredients in the refrigerator and efficiently managing the food ingredients that the user has.

[0941] The "recipe acquisition means" is a means for acquiring a cooking recipe from an external resource and incorporating the recipe into the system.

[0942] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the family structure and the user's preferences.

[0943] The "shopping list generating means" is a means for automatically generating a shopping list for purchasing necessary ingredients.

[0944] "Advertisement presentation means" refers to a means for appropriately presenting relevant advertisements to users.

[0945] The "receipt analysis means" is a means for analyzing an image of a receipt photographed by a user and automatically updating information about purchased ingredients.

[0946] The "emotion analysis means" is a means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on that emotion.

[0947] The "delivery arrangement means" is a means for arranging for missing ingredients to be delivered by a delivery service based on the ingredient list and the proposed menu.

[0948] The system that realizes this application example is configured as follows: First, the user takes a photo of the receipt with their smartphone camera and taps the "Upload Receipt" button to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image using an OCR engine (such as Tesseract OCR), extracts information about the ingredients purchased, and stores that information in a database. This allows the user to always check the latest ingredient list.

[0949] Next, when the user requests a "menu suggestion," the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) and emotional state to the server. The server uses this information to generate menu suggestions using a generative AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[0950] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user uses it as a reference when cooking.

[0951] Furthermore, when a user browses a recipe, the device sends that information to the server, which then selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools, and send the appropriate advertisements to the device to present to the user.

[0952] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and sends the generated link to the device, allowing the user to easily make a purchase on the related site.

[0953] Users can also input their own emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[0954] Furthermore, the system also has a function to arrange for delivery of missing ingredients based on the list of ingredients in the user's refrigerator and the proposed menu.The server identifies missing ingredients based on the ingredient list and the selected menu, and sends a request to the delivery service via API.

[0955] For example, if the emotion analysis engine detects that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time, which is used to improve the system.

[0956] Prompt Sentence Examples

[0957] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[0958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0959] Step 1:

[0960] To manage the ingredients in their refrigerator, users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image.

[0961] Input: A captured image of the receipt

[0962] What happens: The user takes a photo of the receipt with their smartphone camera and taps a button in the app.

[0963] Step 2:

[0964] The terminal transmits the receipt image to the server.

[0965] Input: Receipt image

[0966] Specific operation: The receipt image is sent from the device to the server.

[0967] Step 3:

[0968] The server analyzes the received receipt image using an OCR engine (e.g., Tesseract OCR), extracts information about the purchased ingredients, and stores that information in a database.

[0969] Input: Receipt image

[0970] Data processing and calculation: The receipt image is analyzed using an OCR engine to generate text data.

[0971] Output: Extracted ingredient information

[0972] Specific operation: The server uses an OCR engine to analyze the receipt image and saves the ingredient information in a database.

[0973] Step 4:

[0974] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, and emotional state) to the server.

[0975] Input: Ingredient list, user profile information, emotional state

[0976] Specific operation: The user requests a "menu suggestion" from within the app, and the device sends the necessary information to the server.

[0977] Step 5:

[0978] The server uses a generative AI model to generate menu suggestions based on the submitted ingredient list, profile information, and emotional state, and sends them to the device.

[0979] Input: Ingredient list, profile information, emotional state

[0980] Data processing and calculation: The generative AI model generates optimal menu candidates based on the input data.

[0981] Output: Menu candidate list

[0982] Specific operation: The server runs the generative AI model and sends menu suggestions to the device.

[0983] Step 6:

[0984] When the user selects the desired menu, the terminal transmits the information to the server.

[0985] Input: Selected menu information

[0986] Specific operation: The user selects a menu and the device sends that information to the server.

[0987] Step 7:

[0988] Based on the selected menu, the server uses a recipe generation engine to automatically generate the quantities of ingredients and cooking instructions required, and sends them to the terminal.

[0989] Input: Selected menu information

[0990] Data processing and calculation: The recipe generation engine generates quantities and cooking instructions based on the menu.

[0991] Output: Recipe with portion adjustments

[0992] Specific operation: The server executes the recipe generation engine and sends the generated recipe to the device.

[0993] Step 8:

[0994] When a user is browsing recipes, the device sends that information to the server, which then selects appropriate advertisements and sends them to the device.

[0995] Input: Recipe viewing information

[0996] Data processing and calculation: The server selects appropriate advertisements based on the advertisement prediction model.

[0997] Output: Selected ads

[0998] Specific operation: The device sends recipe viewing information, and the server selects appropriate advertisements and sends them to the device.

[0999] Step 9:

[1000] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and send it to the device.

[1001] Input: Desired ingredient information

[1002] Data processing and calculation: The server generates a purchase link using the API of the online shopping site.

[1003] Output: Purchase link

[1004] Specific operation: The device sends a request to purchase ingredients, and the server generates a purchase link and sends it to the device.

[1005] Step 10:

[1006] Users can input their emotions using the in-app camera or microphone, and the device sends this information to a server, which then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state.

[1007] Input: Emotion data

[1008] Data processing and calculation: The server analyzes the emotion data using an emotion analysis engine.

[1009] Output: Emotion analysis results

[1010] Specific operation: The user inputs their emotions, the device sends them to the server, and the server analyzes them.

[1011] Step 11:

[1012] The server optimizes menu suggestions and advertisement displays based on the emotional state.

[1013] Input: Sentiment analysis results

[1014] Data processing and calculation: Propose optimal menus and select advertisements based on emotions.

[1015] Output: Optimized menu suggestions and advertisements

[1016] Specific operation: The server optimizes menu suggestions and advertisement display based on the results of sentiment analysis.

[1017] Step 12:

[1018] The server sends an API request to arrange for delivery of any missing ingredients based on the ingredient list and the proposed menu.

[1019] Input: Ingredient list, suggested meal

[1020] Data processing and calculation: Generates API requests for delivery arrangements.

[1021] Output: Delivery arrangement completion notification

[1022] Specific operation: The server sends an API request to the delivery service to complete the arrangement.

[1023] Prompt Sentence Examples

[1024] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[1025] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1026] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1027] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1028] [Third embodiment]

[1029] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1030] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1032] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1033] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1034] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1036] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1037] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1039] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1040] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1041] MODE FOR CARRYING OUT THE INVENTION

[1042] The system of the present invention has the functions of managing ingredients in a refrigerator, proposing optimal menus based on the user's family structure and preferences, generating a shopping list for purchasing necessary ingredients, and providing appropriate advertisements.

[1043] Food management function

[1044] It provides a means to automatically capture purchased ingredient information using a receipt photographed by the user. The user photographs the receipt with their smartphone camera and uploads the image to the app. At this time, the device sends the photographed image to a server. The server analyzes the receipt image, extracts purchased ingredient information, and stores that information in a database. This allows the user to always check the latest ingredient list.

[1045] AI menu suggestion function

[1046] When a user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate an optimal menu using an AI model. It generates multiple menu options and creates a list of ingredients required for each. This allows the user to choose from multiple menu options.

[1047] Automatic recipe generation function

[1048] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities can be adjusted to suit the number of family members and preferences. The generated recipe is sent to the device and can be viewed by the user.

[1049] Advertisement display function

[1050] As a user browses recipes, their device sends that information to a server, which uses the user's profile and recent browsing information to select relevant ads, such as special offers on ingredients or convenient cooking tools. Once a suitable ad is selected, it is sent to the device and presented to the user.

[1051] Shopping list generator

[1052] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a link to purchase the ingredient. The server then sends the link to the device, allowing the user to easily click the link to make a purchase on the relevant site.

[1053] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the ingredients they need. Furthermore, by providing appropriate advertisements, it is possible to provide new value to users.

[1054] The processing flow will be explained below.

[1055] Food management function

[1056] Step 1:

[1057] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[1058] Step 2:

[1059] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user confirms it, the image is sent to the server by pressing the send button.

[1060] Step 3:

[1061] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[1062] Step 4:

[1063] Server: The OCR engine extracts text information from the receipt image and generates data in the format of "3 apples" or "1 liter of milk."

[1064] Step 5:

[1065] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[1066] Step 6:

[1067] Server: Updates the user's ingredient list and sends the latest list information to the device.

[1068] Step 7:

[1069] Device: Display the latest ingredient list on the screen.

[1070] Step 8:

[1071] User: View updated ingredient list in the app.

[1072] ---

[1073] AI menu suggestion function

[1074] Step 1:

[1075] User: Tap the "Menu Suggestion" button on the app.

[1076] Step 2:

[1077] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[1078] Step 3:

[1079] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[1080] Step 4:

[1081] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[1082] Step 5:

[1083] Server: Sends the generated menu options to the terminal.

[1084] Step 6:

[1085] Terminal: Presents multiple menu options to the user.

[1086] Step 7:

[1087] User: Select the desired menu from the displayed menu.

[1088] ---

[1089] Automatic recipe generation function

[1090] Step 1:

[1091] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[1092] Step 2:

[1093] Terminal: Sends the selected menu information and the user's family composition information to the server.

[1094] Step 3:

[1095] Server: Based on the selected menu, launches the recipe generation engine.

[1096] Step 4:

[1097] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members (e.g., "300g chicken, 1 / 2 teaspoon salt...").

[1098] Step 5:

[1099] Server: Sends the generated recipe to the device.

[1100] Step 6:

[1101] Terminal: Displays the generated recipe and cooking instructions to the user.

[1102] Step 7:

[1103] User: Cooks using the displayed recipe as a reference.

[1104] ---

[1105] Advertisement display function

[1106] Step 1:

[1107] Users: See ads while browsing recipes.

[1108] Step 2:

[1109] Device: Sends browsing information to the server.

[1110] Step 3:

[1111] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[1112] Step 4:

[1113] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[1114] Step 5:

[1115] Server: Sends the selected advertisement to the device.

[1116] Step 6:

[1117] Device: Display the received advertisement on the recipe viewing screen.

[1118] Step 7:

[1119] User: View the ad displayed and click to view more information if necessary.

[1120] ---

[1121] Shopping list generator

[1122] Step 1:

[1123] User: Requests to purchase desired ingredients.

[1124] Step 2:

[1125] Terminal: Sends a purchase request to the server.

[1126] Step 3:

[1127] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[1128] Step 4:

[1129] Server: Sends the generated purchase link to the device.

[1130] Step 5:

[1131] Device: Show the user a purchase link.

[1132] Step 6:

[1133] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[1134] Example 1

[1135] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1136] Conventional refrigerator ingredient management and menu suggestion systems required a lot of manual input work, which was time-consuming for users. They also lacked the functionality to suggest menus suited to the user's family structure and preferences, or to easily manage the purchasing procedures for necessary ingredients. Furthermore, there was a need for a way to provide new value by providing appropriate advertisements to users.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1138] In this invention, the server includes a means for analyzing receipts photographed by the user and automatically capturing purchased ingredient information, a means for proposing optimal menus based on the user's family structure and preferences, and a means for generating menus using an AI model based on the ingredient list and profile information stored on the server. This allows the user to efficiently manage ingredients in the refrigerator, receive suggestions for optimal menus suited to their family structure and preferences, and smoothly purchase the necessary ingredients.

[1139] "Means of analyzing receipts photographed by users and automatically importing information about purchased ingredients" refers to a function that analyzes images of receipts photographed by users on devices such as smartphones, extracts information about purchased ingredients from the images, and automatically imports the information into a database.

[1140] "Means for suggesting optimal menus based on family structure and preferences" refers to a function that automatically generates and suggests optimal menus based on the user's family structure (for example, number of people and age range) and preferences (preferred dishes and ingredients).

[1141] "Means for generating menus using an AI model based on ingredient lists and profile information stored on a server" refers to a function that stores ingredient lists and user profile information entered by the user or automatically obtained on a server, and automatically generates optimal menus using an AI model based on that information.

[1142] The "recipe generation means for automatically generating the quantities of ingredients required and cooking procedures" is a function for automatically generating and providing the specific quantities of ingredients required and cooking procedures based on the selected menu.

[1143] "Means for selecting relevant advertisements based on the generated recipe and presenting them to the user" refers to a function for selecting highly relevant advertisements (for example, information on special sales of ingredients or advertisements for cooking utensils) based on automatically generated recipe information and displaying them to the user.

[1144] "Means for generating a purchase link using the API of an affiliated online shopping site" refers to a function that uses the API of an affiliated online shopping site to generate a purchase link and provide that link to the user when the user purchases the ingredients they desire.

[1145] MODE FOR CARRYING OUT THE INVENTION

[1146] The system of the present invention utilizes data processing and AI technology to seamlessly perform a series of tasks, from managing ingredients in the refrigerator to suggesting menus, creating recipes, generating shopping lists, and providing advertisements.

[1147] Food management function

[1148] First, the user takes a photo of the receipt for the ingredients they purchased using their smartphone camera. They then upload the image of the receipt to the app. The device then sends the uploaded receipt image to the server. The server uses image analysis software such as Google Cloud Vision API to extract information about the ingredients purchased from the receipt image. The extracted information about the ingredients is saved in a database. This allows the user to always check the latest list of ingredients on the app. As a specific example, when a user takes a photo of a receipt from a supermarket purchase and uploads it to the app, the server automatically extracts and saves information such as "1L of milk, 10 eggs, 4 tomatoes."

[1149] AI menu suggestion function

[1150] When a user requests a "menu suggestion" within the app, the device sends the latest ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. Based on this information, the server uses a generative AI model such as OpenAI's GPT-3 to generate the optimal menu. A specific prompt might be something like, "My family consists of three people, I like Japanese food, and I have chicken and potatoes. Please suggest a menu for me." The server then generates the optimal menu from multiple candidates and provides it to the user. This function allows users to easily obtain the optimal menu.

[1151] Automatic recipe generation function

[1152] When a user selects a desired menu from the menu options provided, the device sends that information to the server. The server then uses IBM Watson's recipe generation engine to automatically generate the specific amounts of ingredients needed and cooking instructions based on the selected menu. The generated recipe is sent to the device and can be viewed by the user on the app. For example, a recipe for "Teriyaki Chicken" may include details such as "200g of chicken, 2 tablespoons of soy sauce, and 1 tablespoon of sugar."

[1153] Advertisement display function

[1154] When a user is browsing a recipe generated by the app, the device sends browsing information to the server. The server selects relevant advertisements based on the user's profile and recent browsing information. These advertisements may include, for example, special sale ingredients or convenient cooking utensils. The selected advertisements are sent to the device in a timely manner and presented to the user. For example, if a user is browsing a recipe for "Mapo Tofu," advertisements for special sale tofu or new cooking utensils may be displayed in the app.

[1155] Shopping list generator

[1156] When a user sends a request to purchase necessary ingredients, the device sends that information to the server. The server uses the API of an affiliated online shopping site (e.g., shopping site API) to generate a purchase link for the necessary ingredients. The generated link is sent to the device, and the user can click on it to easily make a purchase on the related site. For example, in response to a request for "100g of tofu and 1 green onion," the generated purchase link is displayed in the app, and the user can click on it to smoothly make a purchase on the online shopping site.

[1157] The system of the present invention is designed to allow users to efficiently manage ingredients in their refrigerators, suggest optimal menus, and easily purchase the ingredients they need. Furthermore, by providing relevant advertisements, the system can provide new value to users.

[1158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1159] Step 1: Take a photo of your receipt

[1160] Users take a photo of the receipt for the food they purchased using their smartphone camera, and this image becomes input data for the system.

[1161] Step 2: Upload a receipt image

[1162] The user opens the app and uploads a photo of the receipt, which becomes the input data sent to the server.

[1163] Step 3: Send the receipt image to the server

[1164] The device sends the uploaded receipt image to the server. The server uses image analysis software (e.g., image analysis API) to process the data and extract ingredient information from the receipt image. The input in this step is the receipt image, and the output is ingredient information.

[1165] Step 4: Store purchased ingredients in a database

[1166] The server stores the extracted ingredient information in a database. This process allows users to access the latest ingredient list. The input is the extracted ingredient information, and the output is the ingredient information stored in the database.

[1167] Step 5: Request menu suggestions

[1168] Users can request a "menu suggestion" within the app, which becomes new input data for the system.

[1169] Step 6: Submit user information

[1170] The device sends the latest ingredient list and user profile information (family composition, preferences, etc.) to the server. The ingredient list and profile information are input, and are used in the AI ​​model.

[1171] Step 7: Menu generation using AI models

[1172] The server uses a generative AI model to generate an optimal menu based on the input ingredient list and profile information. The specific input prompt is "Family size is 3 people, preference is Japanese food, ingredients available are chicken and potatoes, please suggest a menu." The output of this step is the generated menu.

[1173] Step 8: Offer menu options

[1174] The server sends the generated menu options to the terminal. The user selects the desired menu from these options. The input is the generated menu options, and the output is the menu options provided to the user.

[1175] Step 9: Send menu selection to server

[1176] When the user selects the desired menu, the terminal sends the selection information to the server. The input is the selected menu, which becomes the basis for recipe generation.

[1177] Step 10: Auto-generating recipes

[1178] The server uses a recipe generation engine (e.g., a recipe generation API) to automatically generate the required ingredient quantities and cooking instructions based on the selected menu. The input is the selected menu, and the output is the specific recipe information.

[1179] Step 11: User Provides Recipe Information

[1180] The server sends the generated recipe information to the device so that the user can view it in the app. The input is the automatically generated recipe information, and the output is the recipe information provided to the user.

[1181] Step 12: Send recipe viewing information to the server

[1182] When a user browses recipes on the app, the device sends browsing information to the server. The input is the user's browsing information, which is used as the basis for selecting ads.

[1183] Step 13: Select relevant ads

[1184] The server selects relevant advertisements based on browsing information and user profile. The input is browsing information and profile information, and the output is the selected advertisement.

[1185] Step 14: User Provides Ad Presentation

[1186] The server sends the selected advertisement to the terminal for presentation to the user. The input is the selected advertisement, and the output is the advertisement provided to the user.

[1187] Step 15: Send purchase request to server

[1188] When a user requests to purchase a desired ingredient, the device sends the information to the server. The input is the user's purchase request, which becomes the basis for generating a purchase link.

[1189] Step 16: Generate a Purchase Link

[1190] The server uses the API of an online shopping site to generate a purchase link for the required ingredients. The input is the purchase request, and the output is the generated purchase link.

[1191] Step 17: User Provided Purchase Link

[1192] The server sends the generated purchase link to the terminal, which the user can click to easily make a purchase on the related site. The input is the generated purchase link, and the output is the purchase link provided to the user.

[1193] (Application example 1)

[1194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] In modern households, managing the ingredients in the refrigerator, proposing optimal menus, and purchasing ingredients are time-consuming and labor-intensive. This makes preparing home-cooked meals complicated, leading to unnecessary food waste and incorrect purchases. Furthermore, users have limited means of obtaining the information they need without receiving advertisements based on individual ingredients. Furthermore, the inability to smoothly integrate with food delivery services reduces the efficiency of home cooking. A new system is needed to solve these issues.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1197] In this invention, the server includes an ingredient management unit for managing ingredients in the refrigerator, a recipe acquisition unit for acquiring recipes from external resources, a recipe adjustment unit for automatically adjusting recipe portion sizes based on family composition and preferences, a shopping list generation unit for generating a shopping list for purchasing necessary ingredients, an advertisement presentation unit for presenting advertisements to the user, a receipt analysis unit for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, a menu suggestion unit for generating an optimal menu based on the user's profile information using an AI model, and a delivery collaboration unit for ordering necessary ingredients based on the menu in collaboration with a food delivery service. This allows users to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase and deliver the necessary ingredients. Furthermore, providing appropriate advertisements to users encourages purchasing behavior and improves the efficiency of home cooking.

[1198] The "foodstuff management means" is a system for listing the foodstuffs in the refrigerator and constantly updating and managing the inventory status.

[1199] The "recipe acquisition means" is a mechanism for acquiring cooking recipe information from an external database, the Internet, etc.

[1200] The "recipe adjustment means" is a mechanism for automatically adjusting the amounts of ingredients in the acquired recipe according to the household composition and individual preferences.

[1201] The "shopping list generating means" is a mechanism for automatically generating a list for purchasing necessary ingredients.

[1202] An "advertising presentation means" is a mechanism for displaying highly relevant advertisements to users.

[1203] The "receipt analysis means" is a mechanism that analyzes receipt images taken by the user and automatically updates the list of purchased ingredients based on that information.

[1204] The "menu suggestion method" is a system that uses an AI model to calculate and suggest the optimal menu based on the user's profile information.

[1205] "Delivery collaboration means" is a system that links with a food delivery service to order and request delivery of the necessary ingredients based on the proposed menu.

[1206] MODE FOR CARRYING OUT THE INVENTION

[1207] The present invention is a system that efficiently manages ingredients in a refrigerator, proposes optimal menus, and allows necessary ingredients to be easily purchased and delivered. An embodiment of the present invention will be described below.

[1208] First, to manage the information on ingredients purchased by the user, the user takes a photo of the receipt with the smartphone camera and uploads the image to the app. At this time, the device sends the captured image to the server. The server performs image analysis (using OCR technology) to extract ingredient information from the receipt and saves it in a database. This function allows the user to always manage an up-to-date ingredient list.

[1209] Next, when the user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information to the server. Based on this information, the server uses a generative AI model to generate an optimal menu based on family composition and preferences. At this time, multiple menu options are generated, and the user can choose from these menu options.

[1210] After that, when the user selects the desired menu, the device sends that information to the server, which then uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions based on the selected menu, allowing users to easily obtain recipes that suit the number of family members and their preferences.

[1211] Additionally, when a user is browsing recipes, appropriate advertisements are displayed. The device sends recipe browsing information and the user's profile to the server, and the server selects and presents highly relevant advertisements (such as ingredients on sale or convenient cooking utensils) to the user. This encourages users to make purchases and provides them with even more useful information.

[1212] Furthermore, if the user requests to purchase ingredients, the device sends that information to the server, which then uses the API of a partner food delivery service to generate a link to purchase the ingredients, allowing the user to request delivery with just one click.

[1213] The hardware and software used to build this system includes:

[1214] Hardware: Smartphone (iPhone, Android device)

[1215] software:

[1216] Python: Backend processing for the application

[1217] OpenCV: Receipt image processing and analysis

[1218] Requests: Communication with the server

[1219] For example, a user can purchase new ingredients at the supermarket to prepare dinner and upload the receipt to the app. The app then uses AI to suggest the optimal dinner menu and places an order for the ingredients needed for the user's selected menu with a food delivery service. A detailed recipe is then displayed to help prepare dinner.

[1220] Example prompt sentence:

[1221] 1. "Take a photo of your receipt and upload it."

[1222] 2. "I'll suggest the perfect menu for tonight's dinner."

[1223] 3. "I can quickly order the ingredients I need for this menu."

[1224] 4. "Check out the cookware on sale."

[1225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1226] Step 1:

[1227] The user takes a photo of the receipt with the smartphone camera. Then, they upload the image to the app. The input is the image of the receipt, and the output is its image data. The image data is sent from the device to the server.

[1228] Step 2:

[1229] The server analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the image data of the receipt, and the output is text data of the extracted ingredient information. This text data is stored in a database.

[1230] Step 3:

[1231] The user requests a "menu suggestion" within the app. The input is the user's request, profile information (family composition, preferences, etc.), and saved ingredient list. The output is this information sent to the server.

[1232] Step 4:

[1233] The server uses the received user profile information and ingredient list to generate an optimal meal plan using a generative AI model (e.g., GPT-3). The input is the profile information and ingredient list, and the output is multiple suggested meal options. These meal options are sent to the device.

[1234] Step 5:

[1235] The user selects the desired menu. The input is the user's menu selection information, and the output is the selection information sent to the server.

[1236] Step 6:

[1237] The server uses a recipe generation engine (e.g., Recipe API) to automatically generate the quantities of ingredients and cooking instructions based on the selected menu. The input is the selected menu information, and the output is the generated recipe information. This recipe information is sent to the device.

[1238] Step 7:

[1239] When a user browses a recipe, the device sends the browsing information to the server. The input is the browsing information and the user's profile information, and the output is the information sent to the server.

[1240] Step 8:

[1241] The server uses the profile information and browsing information to search a database for relevant advertisements (e.g., special offers on ingredients or cooking utensils). The input is the profile information and browsing information, and the output is appropriate advertising information. This advertising information is sent to the terminal and displayed to the user.

[1242] Step 9:

[1243] A user submits an order request to purchase the ingredients they need. The input is the purchase request, and the output is the request information sent to the server.

[1244] Step 10:

[1245] The server uses the API of a partner food delivery service to generate a purchase link for the required ingredients. The input is the purchase request information, and the output is the generated purchase link. This link is sent to the terminal, and the user can click on it to use the delivery service.

[1246] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1247] MODE FOR CARRYING OUT THE INVENTION

[1248] The system of the present invention manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion engine can further improve the user experience.

[1249] Food management function

[1250] Users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This allows users to always check the latest ingredient list.

[1251] AI menu suggestion function

[1252] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate menu suggestions using an AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1253] Automatic recipe generation function

[1254] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[1255] Advertisement display function

[1256] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools. Once the appropriate advertisement is selected, it is sent to the device and presented to the user.

[1257] Shopping list generator

[1258] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[1259] Use of emotion engine

[1260] Users can input their emotions using the app's camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[1261] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[1262] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[1263] The processing flow will be explained below.

[1264] A system that combines emotion engines

[1265] Food management function

[1266] Step 1:

[1267] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[1268] Step 2:

[1269] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user has checked it, the image is sent to the server when the user presses the send button.

[1270] Step 3:

[1271] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[1272] Step 4:

[1273] Server: The OCR engine extracts text information from the receipt image and generates data such as "3 apples" and "1 liter of milk."

[1274] Step 5:

[1275] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[1276] Step 6:

[1277] Server: Updates the user's ingredient list and sends the latest list information to the device.

[1278] Step 7:

[1279] Device: Display the latest ingredient list on the screen.

[1280] Step 8:

[1281] User: View updated ingredient list in the app.

[1282] ---

[1283] AI menu suggestion function

[1284] Step 1:

[1285] User: Tap the "Menu Suggestion" button on the app.

[1286] Step 2:

[1287] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[1288] Step 3:

[1289] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[1290] Step 4:

[1291] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[1292] Step 5:

[1293] Server: Sends the generated menu options to the terminal.

[1294] Step 6:

[1295] Terminal: Presents multiple menu options to the user.

[1296] Step 7:

[1297] User: Select the desired menu from the displayed menu.

[1298] ---

[1299] Automatic recipe generation function

[1300] Step 1:

[1301] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[1302] Step 2:

[1303] Terminal: Sends the selected menu information and the user's family composition information to the server.

[1304] Step 3:

[1305] Server: Based on the selected menu, launches the recipe generation engine.

[1306] Step 4:

[1307] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members and their preferences (e.g., "300g chicken, 1 / 2 teaspoon salt").

[1308] Step 5:

[1309] Server: Sends the generated recipe to the device.

[1310] Step 6:

[1311] Terminal: Displays the generated recipe and cooking instructions to the user.

[1312] Step 7:

[1313] User: Cooks using the displayed recipe as a reference.

[1314] ---

[1315] Advertisement display function

[1316] Step 1:

[1317] Users: See ads while browsing recipes.

[1318] Step 2:

[1319] Device: Sends browsing information to the server.

[1320] Step 3:

[1321] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[1322] Step 4:

[1323] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[1324] Step 5:

[1325] Server: Sends the selected advertisement to the device.

[1326] Step 6:

[1327] Device: Display the received advertisement on the recipe viewing screen.

[1328] Step 7:

[1329] User: View the ad displayed and click to view more information if necessary.

[1330] ---

[1331] Shopping list generator

[1332] Step 1:

[1333] User: Requests to purchase desired ingredients.

[1334] Step 2:

[1335] Terminal: Sends a purchase request to the server.

[1336] Step 3:

[1337] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[1338] Step 4:

[1339] Server: Sends the generated purchase link to the device.

[1340] Step 5:

[1341] Device: Show the user a purchase link.

[1342] Step 6:

[1343] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[1344] ---

[1345] Use of emotion engine

[1346] Step 1:

[1347] User: Use the camera and microphone in the app to input their own emotions.

[1348] Step 2:

[1349] Terminal: Sends the input emotion data to the server.

[1350] Step 3:

[1351] Server: Utilizes the emotion engine to analyze the user's emotional state.

[1352] Step 4:

[1353] Server: Optimize menu suggestions and advertisement displays based on analyzed emotional information.

[1354] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[1355] Through these steps, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[1356] Example 2

[1357] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1358] Conventional food management systems have struggled to efficiently manage ingredients in the refrigerator, suggest optimal meals based on the user's family composition and preferences, and smoothly purchase the necessary ingredients. Another issue is that they were unable to optimize suggestions or advertisements based on the user's emotional state, and there was a lack of means to improve the user experience.

[1359] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisements, and a menu suggestion means for generating menu candidates using an AI model based on the acquired ingredient list and user profile information. This allows the user to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing the emotion engine, the user experience can be further improved.

[1360] "Food in the refrigerator" refers to food and beverages stored in the refrigerator or freezer.

[1361] The "food management means" is a means for efficiently listing the food ingredients stored in the refrigerator and managing that information.

[1362] "External resources" are information sources that exist outside the system, such as databases and websites on the Internet.

[1363] The "recipe acquisition means" is a means for acquiring cooking recipe information from an external resource.

[1364] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the user's family composition and preferences.

[1365] The "shopping list generating means" is a means for automatically generating a list for purchasing necessary ingredients.

[1366] "Advertising presentation means" refers to a means for selecting and presenting appropriate advertisements to users.

[1367] The "receipt analysis means" is a means for analyzing the receipt image taken by the user and automatically updating the information on the purchased ingredients.

[1368] "Emotion analysis means" refers to means for analyzing a user's emotional state and optimizing suggestions and advertisements based on that.

[1369] The "menu suggestion means" is a means of generating optimal menu candidates using an AI model based on the acquired ingredient list and user profile information.

[1370] "Profile information" refers to information such as the user's family structure, preferences, cooking skills, etc.

[1371] An "AI model" is a collection of algorithms and data that uses machine learning techniques to perform a specific task.

[1372] This invention is a system that manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion analysis engine can further improve the user experience.

[1373] Hardware and Software

[1374] This system uses the following hardware and software:

[1375] Smartphone: Take and upload receipt images, operate the app.

[1376] Server: Data processing and storage.

[1377] OCR engine: Receipt image analysis (e.g., Tesseract or Google Cloud Vision API).

[1378] Recipe generation engine: Automatically generate the required ingredient quantities and cooking instructions (e.g., custom algorithms using Python libraries).

[1379] AI Model: A generative AI model (e.g., TensorFlow or PyTorch) for menu suggestions.

[1380] Sentiment analysis engine: Analyzes the user's emotional state (e.g., Microsoft Azure Emotion API).

[1381] System Operation

[1382] Food management function

[1383] Users take a photo of their supermarket receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This ensures that users always have the most up-to-date ingredient list.

[1384] Menu suggestion function

[1385] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, preferences, cooking skills, etc.) to the server. The server uses this information to use an AI model to generate menu suggestions that suit the user. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1386] Automatic recipe generation function

[1387] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[1388] Advertisement display function

[1389] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it might select ingredients on sale or convenient cooking tools. Appropriate advertisements are selected, sent to the device, and then presented to the user.

[1390] Shopping list generator

[1391] When a user makes a request to purchase desired ingredients, the device sends the information to the server, which then generates a purchase link using the API of a partner online shopping site. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[1392] Use of emotion engine

[1393] Users can input their emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust portion sizes based on the user's profile information.

[1394] Examples and prompts

[1395] Example: When a user takes a photo of a receipt for groceries purchased at the supermarket and uploads it to the app, the system registers the information and displays the latest list of ingredients.

[1396] Example prompt: "I just uploaded the receipt for the groceries I bought at the supermarket."

[1397] Example: When a user requests a menu suggestion, the system will suggest the following dishes based on the ingredients in the refrigerator and the user's preferences: "curry rice," "Chinese rice bowl," and "pasta."

[1398] Example prompt: "Can you suggest some good dishes for dinner tonight?"

[1399] Example: When a user selects the recipe for "curry rice," the system generates ingredient quantities and cooking instructions for four people and sends them to the user.

[1400] Example prompt: "Generate a recipe for curry rice."

[1401] This system allows users to check the latest ingredient list, efficiently manage ingredients, suggest optimal meals, and smoothly purchase the ingredients they need.The system also further improves the user experience by customizing the system based on the user's emotional state.

[1402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1403] System program processing flow

[1404] Step 1: Upload a receipt image

[1405] What happens: The user takes a photo of the receipt with their smartphone camera.

[1406] Input: Receipt image.

[1407] What happens: A user taps the "Upload Receipt" button in the app.

[1408] Output: The receipt image is sent from the device to the server.

[1409] Step 2: OCR analysis and updating the ingredients list

[1410] Specific operation: The server passes the received receipt image to the OCR engine.

[1411] Input: Receipt image.

[1412] How it works: The server uses an OCR engine (e.g., Tesseract or Google Cloud Vision API) to analyze the receipt image and extract information about the ingredients purchased.

[1413] Output: The extracted ingredient information is saved in the database. An updated ingredient list is generated.

[1414] Step 3: Menu suggestions

[1415] Specific behavior: The user taps the "Menu Suggestion" button.

[1416] Input: Ingredient list, profile information (family composition, preferences, cooking skills, etc.).

[1417] How it works: The device sends the ingredient list and profile information to the server, which uses the AI ​​model to generate meal suggestions.

[1418] Output: Multiple menu options (e.g., "curry rice," "Chinese rice bowl," and "pasta") are generated and sent to the terminal.

[1419] Step 4: Auto-generate and adjust recipes

[1420] Specific behavior: The user selects the desired menu.

[1421] Input: Selected menu information.

[1422] How it works: The device sends the selected menu information to the server, which uses a recipe generation engine to automatically generate the required ingredient quantities and cooking instructions. The quantities are adjusted to suit the number of family members and their preferences.

[1423] Output: The generated recipe (ingredients and cooking instructions) is sent to the device.

[1424] Step 5: Ad Presentation

[1425] Specific operation: When a user is browsing a recipe, the device sends that information to the server.

[1426] Input: User profile, recipe viewing information.

[1427] How it works: The server selects relevant ads based on the user's profile and browsing information.

[1428] Output: Appropriate advertisements (e.g., special offers on ingredients or convenient cooking tools) are sent to the device and displayed.

[1429] Step 6: Generate a shopping list

[1430] Specific behavior: The user makes a request to purchase the desired ingredients.

[1431] Input: Shopping request information.

[1432] How it works: The device sends the request information to the server, which then generates a purchase link using the API of a partner online shopping site.

[1433] Output: The generated purchase link is sent to the device, allowing the user to make a purchase on the associated site.

[1434] Step 7: Use the Emotion Engine

[1435] What it does: Users input their emotions using the in-app camera or microphone.

[1436] Input: Emotion data (images, audio).

[1437] How it works: The device sends this emotion data to a server, which uses an emotion analysis engine (e.g., Microsoft Azure Emotion API) to analyze the emotional state.

[1438] Output: Optimal meal suggestions and advertisements based on the emotional state are generated and sent to the device.

[1439] (Application example 2)

[1440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] Conventional food management and recipe suggestion systems were able to properly manage ingredients in the refrigerator and suggest recipes tailored to the user's preferences and family composition, but they were unable to suggest menus based on the user's emotional state or arrange for delivery of missing ingredients immediately. Furthermore, they lacked functionality that responded to the user's emotional state, such as cooking suggestions and shopping list generation that took into account the user's stress and fatigue, and therefore did not adequately improve the user experience. To solve these issues, there is a need for the development of a system that incorporates emotion analysis to suggest appropriate menus, display advertisements, and arrange for delivery based on the user's emotional state.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1443] In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing the necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on the purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on the emotion, and a delivery arrangement means for arranging delivery of missing ingredients based on the ingredient list and the proposed menu. This allows for efficient management of ingredients in the user's refrigerator and enables optimal menu suggestions based on the emotional state and immediate delivery arrangements for missing ingredients.

[1444] The "food management means" is a means for listing the food ingredients in the refrigerator and efficiently managing the food ingredients that the user has.

[1445] The "recipe acquisition means" is a means for acquiring a cooking recipe from an external resource and incorporating the recipe into the system.

[1446] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the family structure and the user's preferences.

[1447] The "shopping list generating means" is a means for automatically generating a shopping list for purchasing necessary ingredients.

[1448] "Advertisement presentation means" refers to a means for appropriately presenting relevant advertisements to users.

[1449] The "receipt analysis means" is a means for analyzing an image of a receipt photographed by a user and automatically updating information about purchased ingredients.

[1450] The "emotion analysis means" is a means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on that emotion.

[1451] The "delivery arrangement means" is a means for arranging for missing ingredients to be delivered by a delivery service based on the ingredient list and the proposed menu.

[1452] The system that realizes this application example is configured as follows: First, the user takes a photo of the receipt with their smartphone camera and taps the "Upload Receipt" button to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image using an OCR engine (such as Tesseract OCR), extracts information about the ingredients purchased, and stores that information in a database. This allows the user to always check the latest ingredient list.

[1453] Next, when the user requests a "menu suggestion," the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) and emotional state to the server. The server uses this information to generate menu suggestions using a generative AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1454] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user uses it as a reference when cooking.

[1455] Furthermore, when a user browses a recipe, the device sends that information to the server, which then selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools, and send the appropriate advertisements to the device to present to the user.

[1456] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and sends the generated link to the device, allowing the user to easily make a purchase on the related site.

[1457] Users can also input their own emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[1458] Furthermore, the system also has a function to arrange for delivery of missing ingredients based on the list of ingredients in the user's refrigerator and the proposed menu.The server identifies missing ingredients based on the ingredient list and the selected menu, and sends a request to the delivery service via API.

[1459] For example, if the emotion analysis engine detects that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time, which is used to improve the system.

[1460] Prompt Sentence Examples

[1461] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[1462] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1463] Step 1:

[1464] To manage the ingredients in their refrigerator, users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image.

[1465] Input: A captured image of the receipt

[1466] What happens: The user takes a photo of the receipt with their smartphone camera and taps a button in the app.

[1467] Step 2:

[1468] The terminal transmits the receipt image to the server.

[1469] Input: Receipt image

[1470] Specific operation: The receipt image is sent from the device to the server.

[1471] Step 3:

[1472] The server analyzes the received receipt image using an OCR engine (e.g., Tesseract OCR), extracts information about the purchased ingredients, and stores that information in a database.

[1473] Input: Receipt image

[1474] Data processing and calculation: The receipt image is analyzed using an OCR engine to generate text data.

[1475] Output: Extracted ingredient information

[1476] Specific operation: The server uses an OCR engine to analyze the receipt image and saves the ingredient information in a database.

[1477] Step 4:

[1478] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, and emotional state) to the server.

[1479] Input: Ingredient list, user profile information, emotional state

[1480] Specific operation: The user requests a "menu suggestion" from within the app, and the device sends the necessary information to the server.

[1481] Step 5:

[1482] The server uses a generative AI model to generate menu suggestions based on the submitted ingredient list, profile information, and emotional state, and sends them to the device.

[1483] Input: Ingredient list, profile information, emotional state

[1484] Data processing and calculation: The generative AI model generates optimal menu candidates based on the input data.

[1485] Output: Menu candidate list

[1486] Specific operation: The server runs the generative AI model and sends menu suggestions to the device.

[1487] Step 6:

[1488] When the user selects the desired menu, the terminal transmits the information to the server.

[1489] Input: Selected menu information

[1490] Specific operation: The user selects a menu and the device sends that information to the server.

[1491] Step 7:

[1492] Based on the selected menu, the server uses a recipe generation engine to automatically generate the quantities of ingredients and cooking instructions required, and sends them to the terminal.

[1493] Input: Selected menu information

[1494] Data processing and calculation: The recipe generation engine generates quantities and cooking instructions based on the menu.

[1495] Output: Recipe with portion adjustments

[1496] Specific operation: The server executes the recipe generation engine and sends the generated recipe to the device.

[1497] Step 8:

[1498] When a user is browsing recipes, the device sends that information to the server, which then selects appropriate advertisements and sends them to the device.

[1499] Input: Recipe viewing information

[1500] Data processing and calculation: The server selects appropriate advertisements based on the advertisement prediction model.

[1501] Output: Selected ads

[1502] Specific operation: The device sends recipe viewing information, and the server selects appropriate advertisements and sends them to the device.

[1503] Step 9:

[1504] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and send it to the device.

[1505] Input: Desired ingredient information

[1506] Data processing and calculation: The server generates a purchase link using the API of the online shopping site.

[1507] Output: Purchase link

[1508] Specific operation: The device sends a request to purchase ingredients, and the server generates a purchase link and sends it to the device.

[1509] Step 10:

[1510] Users can input their emotions using the in-app camera or microphone, and the device sends this information to a server, which then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state.

[1511] Input: Emotion data

[1512] Data processing and calculation: The server analyzes the emotion data using an emotion analysis engine.

[1513] Output: Emotion analysis results

[1514] Specific operation: The user inputs their emotions, the device sends them to the server, and the server analyzes them.

[1515] Step 11:

[1516] The server optimizes menu suggestions and advertisement displays based on the emotional state.

[1517] Input: Sentiment analysis results

[1518] Data processing and calculation: Propose optimal menus and select advertisements based on emotions.

[1519] Output: Optimized menu suggestions and advertisements

[1520] Specific operation: The server optimizes menu suggestions and advertisement display based on the results of sentiment analysis.

[1521] Step 12:

[1522] The server sends an API request to arrange for delivery of any missing ingredients based on the ingredient list and the proposed menu.

[1523] Input: Ingredient list, suggested meal

[1524] Data processing and calculation: Generates API requests for delivery arrangements.

[1525] Output: Delivery arrangement completion notification

[1526] Specific operation: The server sends an API request to the delivery service to complete the arrangement.

[1527] Prompt Sentence Examples

[1528] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[1529] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1531] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1532] [Fourth embodiment]

[1533] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1534] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1536] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1540] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1541] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1544] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1545] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1546] MODE FOR CARRYING OUT THE INVENTION

[1547] The system of the present invention has the functions of managing ingredients in a refrigerator, proposing optimal menus based on the user's family structure and preferences, generating a shopping list for purchasing necessary ingredients, and providing appropriate advertisements.

[1548] Food management function

[1549] It provides a means to automatically capture purchased ingredient information using a receipt photographed by the user. The user photographs the receipt with their smartphone camera and uploads the image to the app. At this time, the device sends the photographed image to a server. The server analyzes the receipt image, extracts purchased ingredient information, and stores that information in a database. This allows the user to always check the latest ingredient list.

[1550] AI menu suggestion function

[1551] When a user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate an optimal menu using an AI model. It generates multiple menu options and creates a list of ingredients required for each. This allows the user to choose from multiple menu options.

[1552] Automatic recipe generation function

[1553] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities can be adjusted to suit the number of family members and preferences. The generated recipe is sent to the device and can be viewed by the user.

[1554] Advertisement display function

[1555] As a user browses recipes, their device sends that information to a server, which uses the user's profile and recent browsing information to select relevant ads, such as special offers on ingredients or convenient cooking tools. Once a suitable ad is selected, it is sent to the device and presented to the user.

[1556] Shopping list generator

[1557] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a link to purchase the ingredient. The server then sends the link to the device, allowing the user to easily click the link to make a purchase on the relevant site.

[1558] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the ingredients they need. Furthermore, by providing appropriate advertisements, it is possible to provide new value to users.

[1559] The processing flow will be explained below.

[1560] Food management function

[1561] Step 1:

[1562] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[1563] Step 2:

[1564] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user confirms it, the image is sent to the server by pressing the send button.

[1565] Step 3:

[1566] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[1567] Step 4:

[1568] Server: The OCR engine extracts text information from the receipt image and generates data in the format of "3 apples" or "1 liter of milk."

[1569] Step 5:

[1570] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[1571] Step 6:

[1572] Server: Updates the user's ingredient list and sends the latest list information to the device.

[1573] Step 7:

[1574] Device: Display the latest ingredient list on the screen.

[1575] Step 8:

[1576] User: View updated ingredient list in the app.

[1577] ---

[1578] AI menu suggestion function

[1579] Step 1:

[1580] User: Tap the "Menu Suggestion" button on the app.

[1581] Step 2:

[1582] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[1583] Step 3:

[1584] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[1585] Step 4:

[1586] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[1587] Step 5:

[1588] Server: Sends the generated menu options to the terminal.

[1589] Step 6:

[1590] Terminal: Presents multiple menu options to the user.

[1591] Step 7:

[1592] User: Select the desired menu from the displayed menu.

[1593] ---

[1594] Automatic recipe generation function

[1595] Step 1:

[1596] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[1597] Step 2:

[1598] Terminal: Sends the selected menu information and the user's family composition information to the server.

[1599] Step 3:

[1600] Server: Based on the selected menu, launches the recipe generation engine.

[1601] Step 4:

[1602] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members (e.g., "300g chicken, 1 / 2 teaspoon salt...").

[1603] Step 5:

[1604] Server: Sends the generated recipe to the device.

[1605] Step 6:

[1606] Terminal: Displays the generated recipe and cooking instructions to the user.

[1607] Step 7:

[1608] User: Cooks using the displayed recipe as a reference.

[1609] ---

[1610] Advertisement display function

[1611] Step 1:

[1612] Users: See ads while browsing recipes.

[1613] Step 2:

[1614] Device: Sends browsing information to the server.

[1615] Step 3:

[1616] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[1617] Step 4:

[1618] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[1619] Step 5:

[1620] Server: Sends the selected advertisement to the device.

[1621] Step 6:

[1622] Device: Display the received advertisement on the recipe viewing screen.

[1623] Step 7:

[1624] User: View the ad displayed and click to view more information if necessary.

[1625] ---

[1626] Shopping list generator

[1627] Step 1:

[1628] User: Requests to purchase desired ingredients.

[1629] Step 2:

[1630] Terminal: Sends a purchase request to the server.

[1631] Step 3:

[1632] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[1633] Step 4:

[1634] Server: Sends the generated purchase link to the device.

[1635] Step 5:

[1636] Device: Show the user a purchase link.

[1637] Step 6:

[1638] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[1639] Example 1

[1640] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1641] Conventional refrigerator ingredient management and menu suggestion systems required a lot of manual input work, which was time-consuming for users. They also lacked the functionality to suggest menus suited to the user's family structure and preferences, or to easily manage the purchasing procedures for necessary ingredients. Furthermore, there was a need for a way to provide new value by providing appropriate advertisements to users.

[1642] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1643] In this invention, the server includes a means for analyzing receipts photographed by the user and automatically capturing purchased ingredient information, a means for proposing optimal menus based on the user's family structure and preferences, and a means for generating menus using an AI model based on the ingredient list and profile information stored on the server. This allows the user to efficiently manage ingredients in the refrigerator, receive suggestions for optimal menus suited to their family structure and preferences, and smoothly purchase the necessary ingredients.

[1644] "Means of analyzing receipts photographed by users and automatically importing information about purchased ingredients" refers to a function that analyzes images of receipts photographed by users on devices such as smartphones, extracts information about purchased ingredients from the images, and automatically imports the information into a database.

[1645] "Means for suggesting optimal menus based on family structure and preferences" refers to a function that automatically generates and suggests optimal menus based on the user's family structure (for example, number of people and age range) and preferences (preferred dishes and ingredients).

[1646] "Means for generating menus using an AI model based on ingredient lists and profile information stored on a server" refers to a function that stores ingredient lists and user profile information entered by the user or automatically obtained on a server, and automatically generates optimal menus using an AI model based on that information.

[1647] The "recipe generation means for automatically generating the quantities of ingredients required and cooking procedures" is a function for automatically generating and providing the specific quantities of ingredients required and cooking procedures based on the selected menu.

[1648] "Means for selecting relevant advertisements based on the generated recipe and presenting them to the user" refers to a function for selecting highly relevant advertisements (for example, information on special sales of ingredients or advertisements for cooking utensils) based on automatically generated recipe information and displaying them to the user.

[1649] "Means for generating a purchase link using the API of an affiliated online shopping site" refers to a function that uses the API of an affiliated online shopping site to generate a purchase link and provide that link to the user when the user purchases the ingredients they desire.

[1650] MODE FOR CARRYING OUT THE INVENTION

[1651] The system of the present invention utilizes data processing and AI technology to seamlessly perform a series of tasks, from managing ingredients in the refrigerator to suggesting menus, creating recipes, generating shopping lists, and providing advertisements.

[1652] Food management function

[1653] First, the user takes a photo of the receipt for the ingredients they purchased using their smartphone camera. They then upload the image of the receipt to the app. The device then sends the uploaded receipt image to the server. The server uses image analysis software such as Google Cloud Vision API to extract information about the ingredients purchased from the receipt image. The extracted information about the ingredients is saved in a database. This allows the user to always check the latest list of ingredients on the app. As a specific example, when a user takes a photo of a receipt from a supermarket purchase and uploads it to the app, the server automatically extracts and saves information such as "1L of milk, 10 eggs, 4 tomatoes."

[1654] AI menu suggestion function

[1655] When a user requests a "menu suggestion" within the app, the device sends the latest ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. Based on this information, the server uses a generative AI model such as OpenAI's GPT-3 to generate the optimal menu. A specific prompt might be something like, "My family consists of three people, I like Japanese food, and I have chicken and potatoes. Please suggest a menu for me." The server then generates the optimal menu from multiple candidates and provides it to the user. This function allows users to easily obtain the optimal menu.

[1656] Automatic recipe generation function

[1657] When a user selects a desired menu from the menu options provided, the device sends that information to the server. The server then uses IBM Watson's recipe generation engine to automatically generate the specific amounts of ingredients needed and cooking instructions based on the selected menu. The generated recipe is sent to the device and can be viewed by the user on the app. For example, a recipe for "Teriyaki Chicken" may include details such as "200g of chicken, 2 tablespoons of soy sauce, and 1 tablespoon of sugar."

[1658] Advertisement display function

[1659] When a user is browsing a recipe generated by the app, the device sends browsing information to the server. The server selects relevant advertisements based on the user's profile and recent browsing information. These advertisements may include, for example, special sale ingredients or convenient cooking utensils. The selected advertisements are sent to the device in a timely manner and presented to the user. For example, if a user is browsing a recipe for "Mapo Tofu," advertisements for special sale tofu or new cooking utensils may be displayed in the app.

[1660] Shopping list generator

[1661] When a user sends a request to purchase necessary ingredients, the device sends that information to the server. The server uses the API of an affiliated online shopping site (e.g., shopping site API) to generate a purchase link for the necessary ingredients. The generated link is sent to the device, and the user can click on it to easily make a purchase on the related site. For example, in response to a request for "100g of tofu and 1 green onion," the generated purchase link is displayed in the app, and the user can click on it to smoothly make a purchase on the online shopping site.

[1662] The system of the present invention is designed to allow users to efficiently manage ingredients in their refrigerators, suggest optimal menus, and easily purchase the ingredients they need. Furthermore, by providing relevant advertisements, the system can provide new value to users.

[1663] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1664] Step 1: Take a photo of your receipt

[1665] Users take a photo of the receipt for the food they purchased using their smartphone camera, and this image becomes input data for the system.

[1666] Step 2: Upload a receipt image

[1667] The user opens the app and uploads a photo of the receipt, which becomes the input data sent to the server.

[1668] Step 3: Send the receipt image to the server

[1669] The device sends the uploaded receipt image to the server. The server uses image analysis software (e.g., image analysis API) to process the data and extract ingredient information from the receipt image. The input in this step is the receipt image, and the output is ingredient information.

[1670] Step 4: Store purchased ingredients in a database

[1671] The server stores the extracted ingredient information in a database. This process allows users to access the latest ingredient list. The input is the extracted ingredient information, and the output is the ingredient information stored in the database.

[1672] Step 5: Request menu suggestions

[1673] Users can request a "menu suggestion" within the app, which becomes new input data for the system.

[1674] Step 6: Submit user information

[1675] The device sends the latest ingredient list and user profile information (family composition, preferences, etc.) to the server. The ingredient list and profile information are input, and are used in the AI ​​model.

[1676] Step 7: Menu generation using AI models

[1677] The server uses a generative AI model to generate an optimal menu based on the input ingredient list and profile information. The specific input prompt is "Family size is 3 people, preference is Japanese food, ingredients available are chicken and potatoes, please suggest a menu." The output of this step is the generated menu.

[1678] Step 8: Offer menu options

[1679] The server sends the generated menu options to the terminal. The user selects the desired menu from these options. The input is the generated menu options, and the output is the menu options provided to the user.

[1680] Step 9: Send menu selection to server

[1681] When the user selects the desired menu, the terminal sends the selection information to the server. The input is the selected menu, which becomes the basis for recipe generation.

[1682] Step 10: Auto-generating recipes

[1683] The server uses a recipe generation engine (e.g., a recipe generation API) to automatically generate the required ingredient quantities and cooking instructions based on the selected menu. The input is the selected menu, and the output is the specific recipe information.

[1684] Step 11: User Provides Recipe Information

[1685] The server sends the generated recipe information to the device so that the user can view it in the app. The input is the automatically generated recipe information, and the output is the recipe information provided to the user.

[1686] Step 12: Send recipe viewing information to the server

[1687] When a user browses recipes on the app, the device sends browsing information to the server. The input is the user's browsing information, which is used as the basis for selecting ads.

[1688] Step 13: Select relevant ads

[1689] The server selects relevant advertisements based on browsing information and user profile. The input is browsing information and profile information, and the output is the selected advertisement.

[1690] Step 14: User Provides Ad Presentation

[1691] The server sends the selected advertisement to the terminal for presentation to the user. The input is the selected advertisement, and the output is the advertisement provided to the user.

[1692] Step 15: Send purchase request to server

[1693] When a user requests to purchase a desired ingredient, the device sends the information to the server. The input is the user's purchase request, which becomes the basis for generating a purchase link.

[1694] Step 16: Generate a Purchase Link

[1695] The server uses the API of an online shopping site to generate a purchase link for the required ingredients. The input is the purchase request, and the output is the generated purchase link.

[1696] Step 17: User Provided Purchase Link

[1697] The server sends the generated purchase link to the terminal, which the user can click to easily make a purchase on the related site. The input is the generated purchase link, and the output is the purchase link provided to the user.

[1698] (Application example 1)

[1699] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1700] In modern households, managing the ingredients in the refrigerator, proposing optimal menus, and purchasing ingredients are time-consuming and labor-intensive. This makes preparing home-cooked meals complicated, leading to unnecessary food waste and incorrect purchases. Furthermore, users have limited means of obtaining the information they need without receiving advertisements based on individual ingredients. Furthermore, the inability to smoothly integrate with food delivery services reduces the efficiency of home cooking. A new system is needed to solve these issues.

[1701] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1702] In this invention, the server includes an ingredient management unit for managing ingredients in the refrigerator, a recipe acquisition unit for acquiring recipes from external resources, a recipe adjustment unit for automatically adjusting recipe portion sizes based on family composition and preferences, a shopping list generation unit for generating a shopping list for purchasing necessary ingredients, an advertisement presentation unit for presenting advertisements to the user, a receipt analysis unit for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, a menu suggestion unit for generating an optimal menu based on the user's profile information using an AI model, and a delivery collaboration unit for ordering necessary ingredients based on the menu in collaboration with a food delivery service. This allows users to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase and deliver the necessary ingredients. Furthermore, providing appropriate advertisements to users encourages purchasing behavior and improves the efficiency of home cooking.

[1703] The "foodstuff management means" is a system for listing the foodstuffs in the refrigerator and constantly updating and managing the inventory status.

[1704] The "recipe acquisition means" is a mechanism for acquiring cooking recipe information from an external database, the Internet, etc.

[1705] The "recipe adjustment means" is a mechanism for automatically adjusting the amounts of ingredients in the acquired recipe according to the household composition and individual preferences.

[1706] The "shopping list generating means" is a mechanism for automatically generating a list for purchasing necessary ingredients.

[1707] An "advertising presentation means" is a mechanism for displaying highly relevant advertisements to users.

[1708] The "receipt analysis means" is a mechanism that analyzes receipt images taken by the user and automatically updates the list of purchased ingredients based on that information.

[1709] The "menu suggestion method" is a system that uses an AI model to calculate and suggest the optimal menu based on the user's profile information.

[1710] "Delivery collaboration means" is a system that links with a food delivery service to order and request delivery of the necessary ingredients based on the proposed menu.

[1711] MODE FOR CARRYING OUT THE INVENTION

[1712] The present invention is a system that efficiently manages ingredients in a refrigerator, proposes optimal menus, and allows necessary ingredients to be easily purchased and delivered. An embodiment of the present invention will be described below.

[1713] First, to manage the information on ingredients purchased by the user, the user takes a photo of the receipt with the smartphone camera and uploads the image to the app. At this time, the device sends the captured image to the server. The server performs image analysis (using OCR technology) to extract ingredient information from the receipt and saves it in a database. This function allows the user to always manage an up-to-date ingredient list.

[1714] Next, when the user requests a "menu suggestion" within the app, the device sends the ingredient list and the user's profile information to the server. Based on this information, the server uses a generative AI model to generate an optimal menu based on family composition and preferences. At this time, multiple menu options are generated, and the user can choose from these menu options.

[1715] After that, when the user selects the desired menu, the device sends that information to the server, which then uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions based on the selected menu, allowing users to easily obtain recipes that suit the number of family members and their preferences.

[1716] Additionally, when a user is browsing recipes, appropriate advertisements are displayed. The device sends recipe browsing information and the user's profile to the server, and the server selects and presents highly relevant advertisements (such as ingredients on sale or convenient cooking utensils) to the user. This encourages users to make purchases and provides them with even more useful information.

[1717] Furthermore, if the user requests to purchase ingredients, the device sends that information to the server, which then uses the API of a partner food delivery service to generate a link to purchase the ingredients, allowing the user to request delivery with just one click.

[1718] The hardware and software used to build this system includes:

[1719] Hardware: Smartphone (iPhone, Android device)

[1720] software:

[1721] Python: Backend processing for the application

[1722] OpenCV: Receipt image processing and analysis

[1723] Requests: Communication with the server

[1724] For example, a user can purchase new ingredients at the supermarket to prepare dinner and upload the receipt to the app. The app then uses AI to suggest the optimal dinner menu and places an order for the ingredients needed for the user's selected menu with a food delivery service. A detailed recipe is then displayed to help prepare dinner.

[1725] Example prompt sentence:

[1726] 1. "Take a photo of your receipt and upload it."

[1727] 2. "I'll suggest the perfect menu for tonight's dinner."

[1728] 3. "I can quickly order the ingredients I need for this menu."

[1729] 4. "Check out the cookware on sale."

[1730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1731] Step 1:

[1732] The user takes a photo of the receipt with the smartphone camera. Then, they upload the image to the app. The input is the image of the receipt, and the output is its image data. The image data is sent from the device to the server.

[1733] Step 2:

[1734] The server analyzes the received receipt image using OCR technology (e.g., Tesseract OCR). The input is the image data of the receipt, and the output is text data of the extracted ingredient information. This text data is stored in a database.

[1735] Step 3:

[1736] The user requests a "menu suggestion" within the app. The input is the user's request, profile information (family composition, preferences, etc.), and saved ingredient list. The output is this information sent to the server.

[1737] Step 4:

[1738] The server uses the received user profile information and ingredient list to generate an optimal meal plan using a generative AI model (e.g., GPT-3). The input is the profile information and ingredient list, and the output is multiple suggested meal options. These meal options are sent to the device.

[1739] Step 5:

[1740] The user selects the desired menu. The input is the user's menu selection information, and the output is the selection information sent to the server.

[1741] Step 6:

[1742] The server uses a recipe generation engine (e.g., Recipe API) to automatically generate the quantities of ingredients and cooking instructions based on the selected menu. The input is the selected menu information, and the output is the generated recipe information. This recipe information is sent to the device.

[1743] Step 7:

[1744] When a user browses a recipe, the device sends the browsing information to the server. The input is the browsing information and the user's profile information, and the output is the information sent to the server.

[1745] Step 8:

[1746] The server uses the profile information and browsing information to search a database for relevant advertisements (e.g., special offers on ingredients or cooking utensils). The input is the profile information and browsing information, and the output is appropriate advertising information. This advertising information is sent to the terminal and displayed to the user.

[1747] Step 9:

[1748] A user submits an order request to purchase the ingredients they need. The input is the purchase request, and the output is the request information sent to the server.

[1749] Step 10:

[1750] The server uses the API of a partner food delivery service to generate a purchase link for the required ingredients. The input is the purchase request information, and the output is the generated purchase link. This link is sent to the terminal, and the user can click on it to use the delivery service.

[1751] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1752] MODE FOR CARRYING OUT THE INVENTION

[1753] The system of the present invention manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion engine can further improve the user experience.

[1754] Food management function

[1755] Users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This allows users to always check the latest ingredient list.

[1756] AI menu suggestion function

[1757] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, etc.) to the server. The server uses this information to generate menu suggestions using an AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1758] Automatic recipe generation function

[1759] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[1760] Advertisement display function

[1761] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools. Once the appropriate advertisement is selected, it is sent to the device and presented to the user.

[1762] Shopping list generator

[1763] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[1764] Use of emotion engine

[1765] Users can input their emotions using the app's camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[1766] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[1767] With the above functions, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[1768] The processing flow will be explained below.

[1769] A system that combines emotion engines

[1770] Food management function

[1771] Step 1:

[1772] User: Take a photo of the receipt with the smartphone camera and tap the "Upload Receipt" button in the app.

[1773] Step 2:

[1774] Terminal: The captured image of the receipt is temporarily saved on the smartphone, and after the user has checked it, the image is sent to the server when the user presses the send button.

[1775] Step 3:

[1776] Server: Receives the sent receipt image and starts the OCR (Optical Character Recognition) engine.

[1777] Step 4:

[1778] Server: The OCR engine extracts text information from the receipt image and generates data such as "3 apples" and "1 liter of milk."

[1779] Step 5:

[1780] Server: Analyzes the extracted text information and stores the ingredient list in a database.

[1781] Step 6:

[1782] Server: Updates the user's ingredient list and sends the latest list information to the device.

[1783] Step 7:

[1784] Device: Display the latest ingredient list on the screen.

[1785] Step 8:

[1786] User: View updated ingredient list in the app.

[1787] ---

[1788] AI menu suggestion function

[1789] Step 1:

[1790] User: Tap the "Menu Suggestion" button on the app.

[1791] Step 2:

[1792] Device: Sends the user's profile information (family composition, cooking skills, preferences, etc.) and current ingredient list to the server.

[1793] Step 3:

[1794] Server: Based on the received information, it uses an AI model to generate menu suggestions.

[1795] Step 4:

[1796] Server: Generates multiple menu options (e.g., "chicken curry," "stir-fried vegetables," "pasta") and creates a list of ingredients required for each menu.

[1797] Step 5:

[1798] Server: Sends the generated menu options to the terminal.

[1799] Step 6:

[1800] Terminal: Presents multiple menu options to the user.

[1801] Step 7:

[1802] User: Select the desired menu from the displayed menu.

[1803] ---

[1804] Automatic recipe generation function

[1805] Step 1:

[1806] User: Select the desired menu on the app and tap the "Generate Recipe" button.

[1807] Step 2:

[1808] Terminal: Sends the selected menu information and the user's family composition information to the server.

[1809] Step 3:

[1810] Server: Based on the selected menu, launches the recipe generation engine.

[1811] Step 4:

[1812] Server: Generates the necessary ingredient amounts and cooking steps based on the number of family members and their preferences (e.g., "300g chicken, 1 / 2 teaspoon salt").

[1813] Step 5:

[1814] Server: Sends the generated recipe to the device.

[1815] Step 6:

[1816] Terminal: Displays the generated recipe and cooking instructions to the user.

[1817] Step 7:

[1818] User: Cooks using the displayed recipe as a reference.

[1819] ---

[1820] Advertisement display function

[1821] Step 1:

[1822] Users: See ads while browsing recipes.

[1823] Step 2:

[1824] Device: Sends browsing information to the server.

[1825] Step 3:

[1826] Server: Launches the advertising engine based on the user's profile and the recipe they are viewing.

[1827] Step 4:

[1828] Server: Select relevant and appropriate advertisements (e.g., chicken on sale or convenient cooking products).

[1829] Step 5:

[1830] Server: Sends the selected advertisement to the device.

[1831] Step 6:

[1832] Device: Display the received advertisement on the recipe viewing screen.

[1833] Step 7:

[1834] User: View the ad displayed and click to view more information if necessary.

[1835] ---

[1836] Shopping list generator

[1837] Step 1:

[1838] User: Requests to purchase desired ingredients.

[1839] Step 2:

[1840] Terminal: Sends a purchase request to the server.

[1841] Step 3:

[1842] Server: Based on the received ingredient list, a purchase link is generated using the API of a partner online shopping site.

[1843] Step 4:

[1844] Server: Sends the generated purchase link to the device.

[1845] Step 5:

[1846] Device: Show the user a purchase link.

[1847] Step 6:

[1848] User: Clicks on the provided link to go to an online shopping site, adds to cart and purchases ingredients.

[1849] ---

[1850] Use of emotion engine

[1851] Step 1:

[1852] User: Use the camera and microphone in the app to input their own emotions.

[1853] Step 2:

[1854] Terminal: Sends the input emotion data to the server.

[1855] Step 3:

[1856] Server: Utilizes the emotion engine to analyze the user's emotional state.

[1857] Step 4:

[1858] Server: Optimize menu suggestions and advertisement displays based on analyzed emotional information.

[1859] For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time and uses this information to improve the system.

[1860] Through these steps, the system of the present invention is designed to enable users to efficiently manage ingredients in their refrigerators, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing an emotion engine, customization based on the user's emotional state is possible, improving the user experience.

[1861] Example 2

[1862] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1863] Conventional food management systems have struggled to efficiently manage ingredients in the refrigerator, suggest optimal meals based on the user's family composition and preferences, and smoothly purchase the necessary ingredients. Another issue is that they were unable to optimize suggestions or advertisements based on the user's emotional state, and there was a lack of means to improve the user experience.

[1864] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisements, and a menu suggestion means for generating menu candidates using an AI model based on the acquired ingredient list and user profile information. This allows the user to efficiently manage ingredients in the refrigerator, receive optimal menu suggestions, and smoothly purchase the necessary ingredients. Furthermore, by utilizing the emotion engine, the user experience can be further improved.

[1865] "Food in the refrigerator" refers to food and beverages stored in the refrigerator or freezer.

[1866] The "food management means" is a means for efficiently listing the food ingredients stored in the refrigerator and managing that information.

[1867] "External resources" are information sources that exist outside the system, such as databases and websites on the Internet.

[1868] The "recipe acquisition means" is a means for acquiring cooking recipe information from an external resource.

[1869] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the user's family composition and preferences.

[1870] The "shopping list generating means" is a means for automatically generating a list for purchasing necessary ingredients.

[1871] "Advertising presentation means" refers to a means for selecting and presenting appropriate advertisements to users.

[1872] The "receipt analysis means" is a means for analyzing the receipt image taken by the user and automatically updating the information on the purchased ingredients.

[1873] "Emotion analysis means" refers to means for analyzing a user's emotional state and optimizing suggestions and advertisements based on that.

[1874] The "menu suggestion means" is a means of generating optimal menu candidates using an AI model based on the acquired ingredient list and user profile information.

[1875] "Profile information" refers to information such as the user's family structure, preferences, cooking skills, etc.

[1876] An "AI model" is a collection of algorithms and data that uses machine learning techniques to perform a specific task.

[1877] This invention is a system that manages the ingredients in the refrigerator, suggests optimal meals based on the user's family composition, preferences, and emotional state, generates a shopping list for purchasing the necessary ingredients, and provides appropriate advertisements. In particular, the incorporation of an emotion analysis engine can further improve the user experience.

[1878] Hardware and Software

[1879] This system uses the following hardware and software:

[1880] Smartphone: Take and upload receipt images, operate the app.

[1881] Server: Data processing and storage.

[1882] OCR engine: Receipt image analysis (e.g., Tesseract or Google Cloud Vision API).

[1883] Recipe generation engine: Automatically generate the required ingredient quantities and cooking instructions (e.g., custom algorithms using Python libraries).

[1884] AI Model: A generative AI model (e.g., TensorFlow or PyTorch) for menu suggestions.

[1885] Sentiment analysis engine: Analyzes the user's emotional state (e.g., Microsoft Azure Emotion API).

[1886] System Operation

[1887] Food management function

[1888] Users take a photo of their supermarket receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image with an OCR engine, extracts information about the ingredients purchased, and stores this information in a database, automatically updating the user's ingredient list. This ensures that users always have the most up-to-date ingredient list.

[1889] Menu suggestion function

[1890] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, preferences, cooking skills, etc.) to the server. The server uses this information to use an AI model to generate menu suggestions that suit the user. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1891] Automatic recipe generation function

[1892] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user can use it as a reference when cooking.

[1893] Advertisement display function

[1894] When a user browses a recipe, the device sends that information to the server. The server selects relevant advertisements based on the user's profile and browsing information. For example, it might select ingredients on sale or convenient cooking tools. Appropriate advertisements are selected, sent to the device, and then presented to the user.

[1895] Shopping list generator

[1896] When a user makes a request to purchase desired ingredients, the device sends the information to the server, which then generates a purchase link using the API of a partner online shopping site. The generated link is then sent to the device, and the user can click the provided link to easily make a purchase on the related site.

[1897] Use of emotion engine

[1898] Users can input their emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion engine to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if the emotion engine recognizes that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust portion sizes based on the user's profile information.

[1899] Examples and prompts

[1900] Example: When a user takes a photo of a receipt for groceries purchased at the supermarket and uploads it to the app, the system registers the information and displays the latest list of ingredients.

[1901] Example prompt: "I just uploaded the receipt for the groceries I bought at the supermarket."

[1902] Example: When a user requests a menu suggestion, the system will suggest the following dishes based on the ingredients in the refrigerator and the user's preferences: "curry rice," "Chinese rice bowl," and "pasta."

[1903] Example prompt: "Can you suggest some good dishes for dinner tonight?"

[1904] Example: When a user selects the recipe for "curry rice," the system generates ingredient quantities and cooking instructions for four people and sends them to the user.

[1905] Example prompt: "Generate a recipe for curry rice."

[1906] This system allows users to check the latest ingredient list, efficiently manage ingredients, suggest optimal meals, and smoothly purchase the ingredients they need.The system also further improves the user experience by customizing the system based on the user's emotional state.

[1907] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1908] System program processing flow

[1909] Step 1: Upload a receipt image

[1910] What happens: The user takes a photo of the receipt with their smartphone camera.

[1911] Input: Receipt image.

[1912] What happens: A user taps the "Upload Receipt" button in the app.

[1913] Output: The receipt image is sent from the device to the server.

[1914] Step 2: OCR analysis and updating the ingredients list

[1915] Specific operation: The server passes the received receipt image to the OCR engine.

[1916] Input: Receipt image.

[1917] How it works: The server uses an OCR engine (e.g., Tesseract or Google Cloud Vision API) to analyze the receipt image and extract information about the ingredients purchased.

[1918] Output: The extracted ingredient information is saved in the database. An updated ingredient list is generated.

[1919] Step 3: Menu suggestions

[1920] Specific behavior: The user taps the "Menu Suggestion" button.

[1921] Input: Ingredient list, profile information (family composition, preferences, cooking skills, etc.).

[1922] How it works: The device sends the ingredient list and profile information to the server, which uses the AI ​​model to generate meal suggestions.

[1923] Output: Multiple menu options (e.g., "curry rice," "Chinese rice bowl," and "pasta") are generated and sent to the terminal.

[1924] Step 4: Auto-generate and adjust recipes

[1925] Specific behavior: The user selects the desired menu.

[1926] Input: Selected menu information.

[1927] How it works: The device sends the selected menu information to the server, which uses a recipe generation engine to automatically generate the required ingredient quantities and cooking instructions. The quantities are adjusted to suit the number of family members and their preferences.

[1928] Output: The generated recipe (ingredients and cooking instructions) is sent to the device.

[1929] Step 5: Ad Presentation

[1930] Specific operation: When a user is browsing a recipe, the device sends that information to the server.

[1931] Input: User profile, recipe viewing information.

[1932] How it works: The server selects relevant ads based on the user's profile and browsing information.

[1933] Output: Appropriate advertisements (e.g., special offers on ingredients or convenient cooking tools) are sent to the device and displayed.

[1934] Step 6: Generate a shopping list

[1935] Specific behavior: The user makes a request to purchase the desired ingredients.

[1936] Input: Shopping request information.

[1937] How it works: The device sends the request information to the server, which then generates a purchase link using the API of a partner online shopping site.

[1938] Output: The generated purchase link is sent to the device, allowing the user to make a purchase on the associated site.

[1939] Step 7: Use the Emotion Engine

[1940] What it does: Users input their emotions using the in-app camera or microphone.

[1941] Input: Emotion data (images, audio).

[1942] How it works: The device sends this emotion data to a server, which uses an emotion analysis engine (e.g., Microsoft Azure Emotion API) to analyze the emotional state.

[1943] Output: Optimal meal suggestions and advertisements based on the emotional state are generated and sent to the device.

[1944] (Application example 2)

[1945] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1946] Conventional food management and recipe suggestion systems were able to properly manage ingredients in the refrigerator and suggest recipes tailored to the user's preferences and family composition, but they were unable to suggest menus based on the user's emotional state or arrange for delivery of missing ingredients immediately. Furthermore, they lacked functionality that responded to the user's emotional state, such as cooking suggestions and shopping list generation that took into account the user's stress and fatigue, and therefore did not adequately improve the user experience. To solve these issues, there is a need for the development of a system that incorporates emotion analysis to suggest appropriate menus, display advertisements, and arrange for delivery based on the user's emotional state.

[1947] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1948] In this invention, the server includes an ingredient management means for managing ingredients in the refrigerator, a recipe acquisition means for acquiring recipes from external resources, a recipe adjustment means for automatically adjusting the quantities of the acquired recipes based on family composition and preferences, a shopping list generation means for generating a shopping list for purchasing the necessary ingredients, an advertisement presentation means for presenting advertisements to the user, a receipt analysis means for analyzing receipts photographed by the user and automatically updating information on the purchased ingredients, an emotion analysis means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on the emotion, and a delivery arrangement means for arranging delivery of missing ingredients based on the ingredient list and the proposed menu. This allows for efficient management of ingredients in the user's refrigerator and enables optimal menu suggestions based on the emotional state and immediate delivery arrangements for missing ingredients.

[1949] The "food management means" is a means for listing the food ingredients in the refrigerator and efficiently managing the food ingredients that the user has.

[1950] The "recipe acquisition means" is a means for acquiring a cooking recipe from an external resource and incorporating the recipe into the system.

[1951] The "recipe adjustment means" is a means for automatically adjusting the quantities of ingredients in the acquired recipe based on the family structure and the user's preferences.

[1952] The "shopping list generating means" is a means for automatically generating a shopping list for purchasing necessary ingredients.

[1953] "Advertisement presentation means" refers to a means for appropriately presenting relevant advertisements to users.

[1954] The "receipt analysis means" is a means for analyzing an image of a receipt photographed by a user and automatically updating information about purchased ingredients.

[1955] The "emotion analysis means" is a means for analyzing the user's emotional state and optimizing menu suggestions and advertisement display based on that emotion.

[1956] The "delivery arrangement means" is a means for arranging for missing ingredients to be delivered by a delivery service based on the ingredient list and the proposed menu.

[1957] The system that realizes this application example is configured as follows: First, the user takes a photo of the receipt with their smartphone camera and taps the "Upload Receipt" button to upload the receipt image. The device then sends the image to the server. The server then analyzes the received receipt image using an OCR engine (such as Tesseract OCR), extracts information about the ingredients purchased, and stores that information in a database. This allows the user to always check the latest ingredient list.

[1958] Next, when the user requests a "menu suggestion," the device sends the ingredient list and the user's profile information (family composition, cooking skills, preferences, etc.) and emotional state to the server. The server uses this information to generate menu suggestions using a generative AI model. Multiple menu suggestions are then presented to the user, allowing them to choose from them.

[1959] When the user selects the desired menu, the device sends that information to the server. Based on the selected menu, the server uses a recipe generation engine to automatically generate the necessary ingredient quantities and cooking instructions. The quantities are adjusted according to the number of family members and preferences. The generated recipe is sent to the device, and the user uses it as a reference when cooking.

[1960] Furthermore, when a user browses a recipe, the device sends that information to the server, which then selects relevant advertisements based on the user's profile and browsing information. For example, it may select ingredients on sale or convenient cooking tools, and send the appropriate advertisements to the device to present to the user.

[1961] When a user requests to purchase desired ingredients, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and sends the generated link to the device, allowing the user to easily make a purchase on the related site.

[1962] Users can also input their own emotions using the in-app camera or microphone. The device then sends this emotional data to the server. The server then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state and optimizes meal suggestions and advertisement display accordingly. For example, if a user is feeling stressed, the app will suggest simple and easy recipes and foods that have a relaxing effect.

[1963] Furthermore, the system also has a function to arrange for delivery of missing ingredients based on the list of ingredients in the user's refrigerator and the proposed menu.The server identifies missing ingredients based on the ingredient list and the selected menu, and sends a request to the delivery service via API.

[1964] For example, if the emotion analysis engine detects that the user is physically tired, the server will suggest "energy meals that can be prepared in under 30 minutes" and adjust the portion sizes based on the user's profile information. The emotion engine also evaluates the user's satisfaction in real time, which is used to improve the system.

[1965] Prompt Sentence Examples

[1966] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[1967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1968] Step 1:

[1969] To manage the ingredients in their refrigerator, users take a photo of the receipt with their smartphone camera and tap the "Upload Receipt" button in the app to upload the receipt image.

[1970] Input: A captured image of the receipt

[1971] What happens: The user takes a photo of the receipt with their smartphone camera and taps a button in the app.

[1972] Step 2:

[1973] The terminal transmits the receipt image to the server.

[1974] Input: Receipt image

[1975] Specific operation: The receipt image is sent from the device to the server.

[1976] Step 3:

[1977] The server analyzes the received receipt image using an OCR engine (e.g., Tesseract OCR), extracts information about the purchased ingredients, and stores that information in a database.

[1978] Input: Receipt image

[1979] Data processing and calculation: The receipt image is analyzed using an OCR engine to generate text data.

[1980] Output: Extracted ingredient information

[1981] Specific operation: The server uses an OCR engine to analyze the receipt image and saves the ingredient information in a database.

[1982] Step 4:

[1983] When a user requests a "menu suggestion," the device sends a list of ingredients and the user's profile information (family composition, cooking skills, preferences, and emotional state) to the server.

[1984] Input: Ingredient list, user profile information, emotional state

[1985] Specific operation: The user requests a "menu suggestion" from within the app, and the device sends the necessary information to the server.

[1986] Step 5:

[1987] The server uses a generative AI model to generate menu suggestions based on the submitted ingredient list, profile information, and emotional state, and sends them to the device.

[1988] Input: Ingredient list, profile information, emotional state

[1989] Data processing and calculation: The generative AI model generates optimal menu candidates based on the input data.

[1990] Output: Menu candidate list

[1991] Specific operation: The server runs the generative AI model and sends menu suggestions to the device.

[1992] Step 6:

[1993] When the user selects the desired menu, the terminal transmits the information to the server.

[1994] Input: Selected menu information

[1995] Specific operation: The user selects a menu and the device sends that information to the server.

[1996] Step 7:

[1997] Based on the selected menu, the server uses a recipe generation engine to automatically generate the quantities of ingredients and cooking instructions required, and sends them to the terminal.

[1998] Input: Selected menu information

[1999] Data processing and calculation: The recipe generation engine generates quantities and cooking instructions based on the menu.

[2000] Output: Recipe with portion adjustments

[2001] Specific operation: The server executes the recipe generation engine and sends the generated recipe to the device.

[2002] Step 8:

[2003] When a user is browsing recipes, the device sends that information to the server, which then selects appropriate advertisements and sends them to the device.

[2004] Input: Recipe viewing information

[2005] Data processing and calculation: The server selects appropriate advertisements based on the advertisement prediction model.

[2006] Output: Selected ads

[2007] Specific operation: The device sends recipe viewing information, and the server selects appropriate advertisements and sends them to the device.

[2008] Step 9:

[2009] When a user requests to purchase a desired ingredient, the device sends the information to the server, which then uses the API of a partner online shopping site to generate a purchase link and send it to the device.

[2010] Input: Desired ingredient information

[2011] Data processing and calculation: The server generates a purchase link using the API of the online shopping site.

[2012] Output: Purchase link

[2013] Specific operation: The device sends a request to purchase ingredients, and the server generates a purchase link and sends it to the device.

[2014] Step 10:

[2015] Users can input their emotions using the in-app camera or microphone, and the device sends this information to a server, which then uses an emotion analysis engine (such as AWS Comprehend or Google Cloud Natural Language) to analyze the user's emotional state.

[2016] Input: Emotion data

[2017] Data processing and calculation: The server analyzes the emotion data using an emotion analysis engine.

[2018] Output: Emotion analysis results

[2019] Specific operation: The user inputs their emotions, the device sends them to the server, and the server analyzes them.

[2020] Step 11:

[2021] The server optimizes menu suggestions and advertisement displays based on the emotional state.

[2022] Input: Sentiment analysis results

[2023] Data processing and calculation: Propose optimal menus and select advertisements based on emotions.

[2024] Output: Optimized menu suggestions and advertisements

[2025] Specific operation: The server optimizes menu suggestions and advertisement display based on the results of sentiment analysis.

[2026] Step 12:

[2027] The server sends an API request to arrange for delivery of any missing ingredients based on the ingredient list and the proposed menu.

[2028] Input: Ingredient list, suggested meal

[2029] Data processing and calculation: Generates API requests for delivery arrangements.

[2030] Output: Delivery arrangement completion notification

[2031] Specific operation: The server sends an API request to the delivery service to complete the arrangement.

[2032] Prompt Sentence Examples

[2033] For example, "If a user enters today's emotional state as 'feeling stressed,' check the ingredients in the refrigerator and suggest simple, easy-to-make recipes that will have a relaxing effect."

[2034] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2036] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2037] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2038] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2039] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2040] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2041] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2042] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2043] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this ne...

Claims

1. a food management means for managing food ingredients in the refrigerator; a recipe acquisition means for acquiring cooking recipes from an external resource; a recipe adjustment means for automatically adjusting the portion sizes of the acquired recipe based on family composition and preferences; a shopping list generating means for generating a shopping list for purchasing necessary ingredients; an advertisement presentation means for presenting advertisements to users; A receipt analysis method that analyzes receipts photographed by users and automatically updates information on purchased ingredients; A system including:

2. 2. The system according to claim 1, further comprising means for listing ingredients in the refrigerator using the ingredient management means and automatically updating information on purchased ingredients using the receipt analysis means.

3. 2. The system according to claim 1, further comprising a profile setting means for inputting family structure and preferences, and a recipe adjusting means for automatically adjusting recipe quantities based on the profile information.

Images