System

Abstract

To provide a system which can propose menus optimized to individual preferences or lifestyles of users.SOLUTION: The system includes: means for recognizing food and seasonings imaged by a camera; means for collecting menu candidates previously selected by a user, the taste of the user, the allergy information of the user, and schedule information of the user; means for generating a menu candidate by using a generative AI model, and a prompt sentence which instructs generation of a menu cookable from the recognized food ane seasonings on the basis of the menu candidates previously selected, and the taste, the allergy information, and schedule information of the user; and means for notifying a terminal of the user of the generated menu candidate.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There are issues regarding food waste at home and daily menu selection. In particular, there is a need for menu suggestions that take into account the expiration date of ingredients, the number of family members, past menu history, and the season. [Means for solving the problem]

[0005] This invention provides a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, and a means for suggesting the generated menus to users. In particular, the generative AI generates menus taking into account parameters such as the number of family members, past menu history, and season, while the camera recognizes the expiration dates of ingredients and generates menus prioritizing ingredients with the earliest expiration dates. This reduces food waste in the home and eases the burden of daily menu planning and menu selection. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the disclosed technique will be described below with reference to the accompanying drawings.

[0008] do.

[0009] First, the terms used in the following description will be explained.

[0010] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0011] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0012] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0013] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0014] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0015] [First embodiment]

[0016] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] In one embodiment of the present invention, a camera installed in a refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera obtains information about ingredients by reading barcodes or QR codes (registered trademark). It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[0030] "Example 2"

[0031] The generative AI generates a menu based on the recognized ingredients and seasonings. The generative AI generates the optimal menu by taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in a family, it generates a menu for cooking meals for four people. It also estimates the family's favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[0032] "Example 3"

[0033] The generated menu is then suggested to the user. The suggestion method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you get home, a voice guide will guide you through that day's dinner menu, making it possible to make suggestions tailored to the user's lifestyle.

[0034] The processing flow of each embodiment will be described below.

[0035] "Example 1"

[0036] Step 1: A camera installed in the refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera can obtain information about ingredients by reading barcodes or QR codes. It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[0037] "Example 2"

[0038] Step 1: Information about ingredients and seasonings obtained from the camera is sent to the generative AI.

[0039] Step 2: The generative AI generates the optimal menu, taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in the family, it generates a menu for cooking meals for four people. It also estimates the family's likely favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[0040] "Example 3"

[0041] Step 1: Receive the menu generated by the generative AI.

[0042] Step 2: The generated menu is proposed to the user. The proposal method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you return home, a voice guide will guide you through that day's dinner menu, making it possible to make proposals that suit the user's lifestyle.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] Conventional food management in refrigerators and pantries is often done manually, making it difficult to accurately grasp the type, quantity, and expiration date of ingredients. It is also common for ingredients to be overlooked due to expiration dates, resulting in waste. Furthermore, creating menus based on ingredient inventory information is time-consuming, creating a need for efficient food management.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes a camera for recognizing ingredients and seasonings, a calculator for analyzing information on the recognized ingredients and seasonings, a storage device for saving the analysis results, and a display device for displaying the saved analysis results to the user. This allows the type, quantity, and expiration date of ingredients to be accurately determined, enabling efficient ingredient management.

[0048] The "photography device" is a device that captures images to recognize ingredients and seasonings.

[0049] The "computing device" is a device for analyzing information about recognized ingredients and seasonings.

[0050] A "storage device" is a device for saving analysis results.

[0051] A "display device" is a device for displaying the saved analysis results to a user.

[0052] "Image recognition technology" is a technology that detects and classifies specific objects and features from images.

[0053] A "barcode" is a code made up of a series of lines and spaces that contains information about ingredients and seasonings.

[0054] A "QR code" is a two-dimensional matrix code that contains information about ingredients and seasonings.

[0055] "Expiration date" is information indicating the date by which ingredients or seasonings can be safely consumed.

[0056] "Menu generation" is the process of suggesting dishes based on information about ingredients and seasonings.

[0057] MODE FOR CARRYING OUT THE INVENTION

[0058] This invention is a system that uses a camera installed in a refrigerator or pantry to recognize ingredients and seasonings, analyzes and stores that information, and displays it to the user. Specific embodiments of this system are described below.

[0059] server

[0060] The server receives image data sent from the camera installed in the refrigerator or pantry. A general server (e.g., cloud server or on-premise server) can be used as the hardware. HTTP requests are used to receive the image data.

[0061] The server preprocesses the received image data, including adjusting the image resolution and removing noise, using an image processing library such as OpenCV.

[0062] The server then uses image recognition technology to analyze the types, quantities, and expiration dates of ingredients and seasonings. Specifically, it uses machine learning libraries such as TENSORFLOW® to classify ingredients in the image and uses object detection algorithms to estimate the amount of ingredients. It also uses the ZBar library to read barcodes and QR codes to obtain detailed information about ingredients.

[0063] The analysis results are converted to JSON format and stored in a database such as MySQL (registered trademark), allowing information such as the type, quantity, and expiration date of ingredients to be centrally managed.

[0064] Terminal

[0065] The device provides an application that allows users to check refrigerator and pantry inventory information. The hardware used can be a smartphone or tablet. The application runs on iOS or Android (registered trademark) platforms.

[0066] The device retrieves inventory information from the server and displays it to the user. It parses the retrieved JSON data and displays the type, quantity, and expiration date of ingredients in a list format. This allows the user to easily check the inventory information of the refrigerator or pantry.

[0067] User

[0068] Users can check the inventory information of their refrigerator and pantry through the application. For example, a user can open the app and check the type, quantity, and expiration date of ingredients in the refrigerator.

[0069] Users can also create shopping lists based on inventory information. For example, users can select ingredients they are running low on and add them to the shopping list within the app. The list is saved on the device and can be edited as needed.

[0070] Examples and prompts

[0071] A specific example is obtaining information about ingredients in a refrigerator. A camera takes a picture of the ingredients in the refrigerator and sends the image data to a server. The server then uses an image recognition algorithm to analyze the type, quantity, and expiration date of the ingredients, and the analysis results are stored in a database. Users can then check this information through the app.

[0072] An example of a prompt sentence might be:

[0073] "I want to develop an application that retrieves information about ingredients in the refrigerator and displays inventory information. I would like to know how to build a system that sends images taken with a camera to a server and uses image recognition technology to analyze the type, quantity, and expiration date of ingredients."

[0074] In this way, by specifically explaining the system processing from the perspectives of the server, terminal, and user, the overall flow can be understood in detail.

[0075] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0076] Program processing flow

[0077] server

[0078] Step 1:

[0079] Receiving image data

[0080] The server receives image data transmitted from the image capturing device installed in the refrigerator or pantry.

[0081] Input: Image data sent from the imaging device

[0082] Output: Received image data

[0083] Specific operation: The camera periodically takes pictures of the inside of the refrigerator and sends them to the server via Wi-Fi. The server receives the HTTP request and saves the image data.

[0084] Step 2:

[0085] Image preprocessing

[0086] The server pre-processes the received image data.

[0087] Input: Received image data

[0088] Output: Preprocessed image data

[0089] Specific operation: The server adjusts the image resolution, removes noise, and adjusts the image contrast and brightness using OpenCV.

[0090] Step 3:

[0091] Barcode / QR code detection

[0092] The server detects barcodes and QR codes from the image.

[0093] Input: Preprocessed image data

[0094] Output: Information on detected barcodes and QR codes

[0095] What it does: The server uses the ZBar library to scan barcodes and QR codes in an image and extract the code's contents.

[0096] Step 4:

[0097] Database Matching

[0098] The server compares the extracted barcode or QR code information with a database.

[0099] Input: Information of the detected barcode or QR code

[0100] Output: Detailed information about ingredients retrieved from the database

[0101] Specific operation: The server sends a query to the MySQL database to obtain detailed information about the corresponding ingredients (type, quantity, expiration date, etc.).

[0102] Step 5:

[0103] Estimating the type and quantity of ingredients using image recognition

[0104] The server uses image recognition technology to estimate the type and quantity of ingredients.

[0105] Input: Preprocessed image data

[0106] Output: Estimated ingredients and quantities

[0107] How it works: The server uses a TensorFlow model to classify ingredients in the image and an object detection algorithm to estimate the amount of ingredients.

[0108] Step 6:

[0109] Saving analysis results

[0110] The server stores the analysis results in a database.

[0111] Input: Estimated ingredient type and quantity, detailed ingredient information retrieved from the database

[0112] Output: Analysis results stored in a database

[0113] Specific operation: The server converts the parsed results into JSON format and stores them in a MySQL database.

[0114] Terminal

[0115] Step 1:

[0116] Obtaining inventory information

[0117] The terminal obtains inventory information from the server.

[0118] Input: HTTP request to the server

[0119] Output: Stock information obtained from the server

[0120] What it does: When a user opens the app, the device sends an HTTP request to the server to get the latest inventory information.

[0121] Step 2:

[0122] View inventory information

[0123] The terminal displays the acquired inventory information to the user.

[0124] Input: Inventory information obtained from the server

[0125] Output: Inventory information displayed to the user

[0126] Specific operation: The device parses the acquired JSON data and displays the type, quantity, and expiration date of the ingredients in a list format.

[0127] User

[0128] Step 1:

[0129] Check stock information

[0130] Users can check refrigerator and pantry inventory information through the app.

[0131] Input: Stock information displayed on the terminal

[0132] Output: User confirmation action

[0133] Specific operation: When a user opens the app, inventory information is displayed and they can check the type, quantity, and expiration date of ingredients.

[0134] Step 2:

[0135] Creating a shopping list

[0136] The user creates a shopping list based on inventory information.

[0137] Input: Stock information displayed on the terminal

[0138] Output: The created shopping list

[0139] How it works: Users select ingredients they're running low on and add them to a shopping list within the app. The list is saved on the device and can be edited as needed.

[0140] (Application example 1)

[0141] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] In modern households, managing the ingredients and seasonings stored in the refrigerator and pantry is complicated, and it is common for ingredients to pass their expiration date or run out of necessary ingredients. It is also difficult to keep track of ingredient inventory and plan menus efficiently. This results in food waste and increases household food costs. Furthermore, there is a need to reduce food waste by prioritizing the use of ingredients that are close to their expiration date.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0144] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to the user, a means for managing ingredient inventory in real time, a means for notifying the user of ingredients approaching their expiration date, and a means for automatically ordering the necessary ingredients. This improves the efficiency of ingredient inventory management and enables priority use of ingredients approaching their expiration date. Furthermore, automatically ordering necessary ingredients can prevent ingredient shortages and reduce household food costs.

[0145] "Camera for recognizing ingredients and seasonings" refers to a camera that is installed in a refrigerator or pantry to recognize the type, quantity, and expiration date of ingredients and seasonings.

[0146] "Generative AI that generates menus based on recognized ingredients and seasonings" is artificial intelligence that automatically generates appropriate menus based on information about ingredients and seasonings recognized by the camera.

[0147] "Means for proposing the generated menu to the user" refers to means for notifying and proposing the menu generated by the generative AI to the user.

[0148] "Means for managing food inventory in real time" refers to a means for grasping and managing the current inventory status in real time based on the type and quantity of food ingredients recognized by the camera.

[0149] The "means for notifying the user of ingredients whose expiration date is approaching" is a means for monitoring the expiration date of ingredients and notifying the user of ingredients whose expiration date is approaching.

[0150] "Means for automatically ordering necessary ingredients" refers to a means for automatically ordering necessary ingredients from a food delivery service, etc., based on stock availability and expiration dates.

[0151] A system for implementing this invention includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to users, a means for managing ingredient inventory in real time, a means for notifying users of ingredients that are close to their expiration date, and a means for automatically ordering the necessary ingredients.

[0152] Hardware and Software Configuration

[0153] Hardware

[0154] Camera: A camera installed in the refrigerator or pantry to recognize the type, quantity, and expiration date of ingredients and seasonings.

[0155] Smartphone: A device that allows users to operate applications.

[0156] Server: A central processing unit for generative AI and data management.

[0157] software

[0158] OpenCV: Image processing library. Used to analyze images acquired from the camera and recognize the types and quantities of ingredients and seasonings.

[0159] Pyzbar: A barcode / QR code reading library. Used to read barcodes and QR codes on ingredients and seasonings.

[0160] Requests: An HTTP request library that we use to order the ingredients we need from a food delivery service.

[0161] Processing flow

[0162] Camera-based food recognition

[0163] The camera is installed in the refrigerator or pantry and periodically captures images. The captured images are analyzed using OpenCV to identify the type, quantity, and expiration date of ingredients and seasonings. If a barcode or QR code is included, the information is read using Pyzbar.

[0164] Menu generation using generative AI

[0165] The information on the recognized ingredients and seasonings is sent to a server, and a generative AI generates a menu based on this information, taking into account parameters such as the number of family members, past menu history, and the season.

[0166] Menu suggestions and inventory management

[0167] The generated menu is presented to the user via a smartphone application, and the availability of ingredients is managed in real time, with notifications sent to users when any ingredients are nearing their expiration date.

[0168] Automated Orders

[0169] If a required ingredient is in short supply, the server uses the Requests library to automatically place an order with a food delivery service, preventing shortages and reducing household food costs.

[0170] Specific examples

[0171] For example, if there are five tomatoes in the refrigerator and their expiration date is approaching in three days, the camera will recognize this and the generative AI will generate a menu using tomatoes. The generated menu will be suggested to the user through a smartphone application. The user will also be notified that the expiration date is approaching and can order more tomatoes if necessary.

[0172] Prompt Sentence Examples

[0173] "Please create an application that recognizes the type and amount of food in the refrigerator and notifies you when the food is nearing its expiration date. Also, please add a function that automatically orders the necessary ingredients from a food delivery service."

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1:

[0176] Cameras are installed in refrigerators and pantries to periodically capture images.

[0177] Input: Images of ingredients and condiments in your refrigerator or pantry.

[0178] Output: The captured image data.

[0179] Specific operation: The camera takes images at regular intervals and sends the image data to the server.

[0180] Step 2:

[0181] The server uses OpenCV to analyze the captured images and recognize the type, quantity, and expiration date of ingredients and seasonings.

[0182] Input: The captured image data.

[0183] Output: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[0184] Specific operation: The server uses the image processing library OpenCV to identify ingredients and seasonings in the image, and if a barcode or QR code is included, it uses Pyzbar to read that information.

[0185] Step 3:

[0186] The server uses generative AI to generate a menu based on the information on ingredients and seasonings recognized by the server.

[0187] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[0188] Output: The generated menu.

[0189] Specific operation: The server inputs information about ingredients and seasonings into the generative AI, which then generates a menu taking into account parameters such as the number of family members, past menu history, and the season.

[0190] Step 4:

[0191] The server proposes the generated menu to the user through a smartphone application.

[0192] Input: The generated menu.

[0193] Output: The menu suggested to the user.

[0194] Specific operation: The server sends the generated menu to the smartphone application, and the application notifies the user.

[0195] Step 5:

[0196] The server manages food inventory in real time and notifies customers of food items approaching their expiration date.

[0197] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[0198] Output: Real-time inventory status, notifications of ingredients approaching their expiration date.

[0199] Specific operation: The server stores information about ingredients and seasonings in a database, and if any ingredients are nearing their expiration date, it notifies the user via a smartphone application.

[0200] Step 6:

[0201] The server automatically orders the necessary ingredients from a food delivery service.

[0202] Input: Availability and expiration date information.

[0203] Output: An order from a food delivery service.

[0204] Specific operation: The server uses the Requests library to automatically order ingredients that are out of stock from a food delivery service.

[0205] Example 2

[0206] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0207] Conventional menu generation systems make it difficult for users to efficiently use ingredients and seasonings they have on hand, and also make it difficult to generate optimal menus that take into account parameters such as the number of family members, past menu history, and the season. Furthermore, they are unable to generate menus that take into account the expiration dates of ingredients, which leads to the problem of food waste.

[0208] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0209] In this invention, the server includes means for receiving information on ingredients and seasonings entered by the user and saving it in a database, means for sending prompts to the generative AI based on the saved information, and means for returning the generated menu to the user's terminal. This allows the user to efficiently use the ingredients and seasonings on hand and generate an optimal menu taking into account parameters such as the number of family members, past menu history, and the season. Furthermore, since the menu can be generated taking into account the expiration dates of ingredients, food waste can be reduced.

[0210] The "photography device" is a device for recognizing ingredients and seasonings.

[0211] "Generative AI" is AI that generates menus based on recognized ingredients and seasonings.

[0212] The "means for suggesting to the user" is a means for presenting the generated menu to the user.

[0213] The "means for saving in a database" is a means for receiving information on ingredients and seasonings entered by the user and saving it in a database.

[0214] The "means for sending a prompt" is a means for sending a prompt to the generative artificial intelligence based on the stored information.

[0215] The "means for returning" is a means for returning the generated menu to the user's terminal.

[0216] "Number of family members" is information about the number of family members to be taken into consideration when creating a menu.

[0217] "Past menu history" is history information of menus that have been generated in the past.

[0218] "Season" is information about the current season that is taken into consideration when generating a menu.

[0219] "Expiration date" is information indicating the expiration date of a food ingredient.

[0220] The present invention is a system that recognizes ingredients and seasonings and generates an optimal menu based on the ingredients and seasonings. A specific embodiment of this system will be described below.

[0221] Hardware and software used

[0222] Hardware: Server, terminal (PC, smartphone, tablet, etc.), imaging device

[0223] Software: Generative AI models (e.g., GPT-4®), database management systems (e.g., MySQL), and user interfaces (e.g., web applications).

[0224] System configuration

[0225] 1. Imaging equipment

[0226] This is a device for recognizing ingredients and seasonings. The user takes a photo of the ingredients or seasonings at hand and sends that information to the system.

[0227] 2. Server

[0228] The server receives the information about ingredients and seasonings entered by the user and stores it in a database.

[0229] The server sends prompts to the generative AI model based on the stored information.

[0230] The server returns the generated menu to the user's terminal.

[0231] 3. Generative AI Models

[0232] The generative AI model generates the optimal menu based on prompts sent from the server.

[0233] The generative AI model takes into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[0234] 4. Database

[0235] The database stores information entered by the user, such as ingredients and seasonings, the number of family members, past menu history, and the season.

[0236] 5. User Interface

[0237] The user interface is a web application that allows users to input information about ingredients and seasonings.

[0238] The user interface presents the generated menu to the user.

[0239] Specific examples

[0240] Prompt Sentence Examples

[0241] "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu items include curry, stew, and stir-fry. The current season is autumn."

[0242] System Operation

[0243] 1. The user inputs ingredients and seasonings

[0244] The user uses the terminal to input the types and amounts of ingredients and seasonings currently on hand.

[0245] Example: "200g chicken, 1 carrot, 1 onion, 50ml soy sauce"

[0246] 2. The server receives the input data and stores it in the database

[0247] The server receives the information about ingredients and seasonings sent by the user and stores it in a database.

[0248] Example: Save as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce" in a MySQL database.

[0249] 3. The server sends a prompt to the generative AI model

[0250] The server generates and sends prompts to the generative AI model, including parameters entered by the user, such as information on ingredients and seasonings, the number of family members, past menu history, and the season.

[0251] Example: "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn."

[0252] 4. Generative AI model generates optimal menu

[0253] The generative AI model generates the optimal menu based on the prompts received, taking into account parameters such as the type and quantity of ingredients, expiration dates, number of family members, past menu history, and the season.

[0254] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[0255] 5. The server returns the generated menu to the user

[0256] The server returns the menu received from the generative AI model to the user's device.

[0257] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[0258] 6. User checks the menu and starts cooking

[0259] The user checks the menu generated on the terminal and starts cooking.

[0260] Example: "Check out the recipe for boiled chicken and vegetables and start cooking."

[0261] In this way, the system uses a generative AI model to generate the optimal menu based on the ingredients and seasoning information entered by the user and provides it to the user.

[0262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0263] Step 1:

[0264] The user inputs ingredients and seasonings

[0265] Input: The user uses a device (PC, smartphone, tablet, etc.) to input the types and quantities of ingredients and seasonings currently on hand.

[0266] Specific operation: The user enters information such as "200g of chicken, 1 carrot, 1 onion, 50ml of soy sauce" into an input form in a web application and clicks the submit button.

[0267] Output: The information about the ingredients and seasonings entered is sent to the server.

[0268] Step 2:

[0269] The server receives the input data and stores it in the database

[0270] Input: Ingredient and seasoning information sent by the user

[0271] Specific operation: The server receives the HTTP request, analyzes its contents, and stores it in a database (e.g., MySQL) as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce."

[0272] Output: Information on ingredients and seasonings stored in the database

[0273] Step 3:

[0274] The server sends prompts to the generative AI model

[0275] Input: Parameters such as ingredients and seasoning information stored in the database, number of family members, past menu history, and seasons

[0276] Specific operation: The server retrieves the necessary information from the database and sends the prompt statement, "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn," to the generative AI model (e.g., GPT-4).

[0277] Output: The prompt sent to the generative AI model

[0278] Step 4:

[0279] Generative AI models generate optimal menus

[0280] Input: Prompt sent from the server

[0281] How it works: The generative AI model analyzes the prompt and generates a menu such as "Chicken and vegetable stew, and a salad made with seasonal autumn ingredients," taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[0282] Output: The generated menu

[0283] Step 5:

[0284] The server returns the generated menu to the user

[0285] Input: Menu received from the generative AI model

[0286] Specific operation: The server sends the generated menu to the user's device as an HTTP response and displays it on the web application.

[0287] Output: Menu displayed on the user's device

[0288] Step 6:

[0289] The user checks the menu and starts cooking

[0290] Input: The menu displayed on the user's device

[0291] Specific behavior: The user checks the recipe displayed on the web application for "Chicken and vegetable stew, salad using seasonal autumn ingredients" and starts cooking by following the necessary steps.

[0292] Output: Specific instructions and ingredients for the user to start cooking

[0293] (Application example 2)

[0294] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] Conventional systems that generate menus by recognizing ingredients and seasonings require users to manually purchase additional ingredients and seasonings required for the generated menu, which is inconvenient. Furthermore, there is a lack of a way to maximize the use of ingredients and seasonings that users already have and reduce waste. Furthermore, there is a need for a way to quickly obtain additional ingredients and seasonings based on the generated menu.

[0296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0297] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, and a means for ordering additional ingredients and seasonings required for the generated menu. This allows the user to make the most of the ingredients and seasonings they have and reduce waste. Furthermore, additional ingredients and seasonings can be quickly obtained based on the generated menu, improving user convenience.

[0298] A "camera for recognizing ingredients and seasonings" is a device that captures images of ingredients and seasonings owned by the user and analyzes them as digital data.

[0299] "Generative AI" is an artificial intelligence system that automatically generates the optimal menu for the user based on recognized information about ingredients and seasonings.

[0300] "Means for proposing the generated menu to the user" refers to an interface or function for notifying and displaying the menu generated by the generative AI to the user.

[0301] "Means for ordering additional ingredients and condiments required for the generated menu" refers to a function that automatically identifies missing ingredients and condiments based on the generated menu and orders them through a food delivery service.

[0302] "Family size" is the number of people in the user's household that will be taken into account when generating the menu.

[0303] "Past menu history" is a record of menus previously created or consumed by a user, data that generative AI uses to learn about a user's preferences and tendencies.

[0304] "Season" refers to information about the current time or season that is taken into consideration when generating a menu, and is a parameter for giving priority to using ingredients that are in season.

[0305] "Expiration date" is information indicating the date by which ingredients and seasonings can be safely consumed, and is used to determine which ingredients to prioritize when creating a menu.

[0306] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0307] System configuration

[0308] The system consists of the following major components:

[0309] 1. Camera for recognizing ingredients and seasonings

[0310] 2. Generative AI

[0311] 3. A means of suggesting the generated menu to the user

[0312] 4. A way to order any additional ingredients or condiments needed for the generated menu.

[0313] Program processing

[0314] Camera for recognizing ingredients and seasonings

[0315] A camera is used to recognize ingredients and seasonings that the user owns. The camera takes a picture of the ingredients in the refrigerator or pantry, and the image is analyzed as digital data. Specifically, image processing libraries such as OpenCV are used to recognize ingredients and seasonings.

[0316] Generative AI

[0317] Based on the information on recognized ingredients and seasonings, a generative AI generates the optimal menu. The generative AI generates the menu taking into account parameters such as the number of family members, past menu history, and the season. For example, an AI model built using TensorFlow or PyTorch can play this role.

[0318] A means of suggesting the generated menu to the user

[0319] The generated menu is presented to the user through the smartphone application, and the user can view the generated menu through the application interface.

[0320] A means to order any additional ingredients or condiments required for the generated menu

[0321] Based on the generated menu, missing ingredients and condiments can be automatically identified and ordered through a food delivery service, for example, using a specific API to order ingredients online.

[0322] Specific examples

[0323] The ingredients in the user's refrigerator are tomatoes, cheese, and basil. There are four people in the family, and past menu history includes pasta and salad. The current season is summer. Based on this information, generative AI is used to suggest the optimal menu. The generated menu is a Margherita pizza, which requires additional pizza dough and olive oil. These additional ingredients are ordered through a food delivery service.

[0324] Prompt Sentence Examples

[0325] The ingredients in the user's refrigerator are tomatoes, cheese, and basil. There are four people in the family, and their past menu history includes pasta and salad. The current season is summer. Based on this information, please use generative AI to suggest the best menu.

[0326] In this way, users can maximize the use of ingredients and seasonings they have and reduce waste, and can quickly obtain additional ingredients and seasonings based on the generated menu, improving user convenience.

[0327] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0328] Step 1:

[0329] The user uses the smartphone camera to take a picture of the ingredients and seasonings in the refrigerator or pantry.

[0330] Input: Images of ingredients and seasonings

[0331] Output: Image data

[0332] Specific operation: The user launches the camera app on their smartphone and takes a picture of ingredients and seasonings in the refrigerator or pantry. The captured image is saved on the smartphone.

[0333] Step 2:

[0334] The server uses an image processing library (e.g., OpenCV) to recognize ingredients and seasonings from the captured image.

[0335] Input: Image data

[0336] Output: A list of recognized ingredients and seasonings

[0337] Specific operation: The server receives the image data and performs image analysis using OpenCV. It extracts the characteristics of ingredients and seasonings and outputs the recognition results in list format.

[0338] Step 3:

[0339] The server uses a generative AI model (e.g., TensorFlow or PyTorch) to generate the optimal menu, taking into account parameters such as recognized ingredients and seasonings, number of family members, past menu history, and seasonings.

[0340] Input: List of recognized ingredients and seasonings, number of family members, past menu history, seasonal information

[0341] Output: The generated menu

[0342] Specific operation: The server receives inputs such as a list of recognized ingredients and seasonings, the number of family members, past menu history, and seasonal information, and uses a generative AI model to generate an optimal menu. The generated menu is then output in list format.

[0343] Step 4:

[0344] The server proposes the generated menu to the user through a smartphone application.

[0345] Input: The generated menu

[0346] Output: Menu suggested to the user

[0347] Specific operation: The server sends the generated menu to the smartphone application and notifies the user, who then checks the menu through the application interface.

[0348] Step 5:

[0349] Based on the generated menu, the server automatically identifies missing ingredients and condiments and orders them through a food delivery service.

[0350] Input: The generated menu

[0351] Output: Additional ingredients and condiments ordered

[0352] What it does: The server analyzes the generated menu and identifies any missing ingredients or condiments, which are then automatically ordered using the food delivery service's API.

[0353] In this way, users can maximize the use of ingredients and seasonings they have and reduce waste, and can quickly obtain additional ingredients and seasonings based on the generated menu, improving user convenience.

[0354] Example 3

[0355] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Conventional menu suggestion systems were unable to fully consider the user's individual preferences, allergy information, or schedule information, making it difficult to suggest the optimal menu for the user. Furthermore, if the suggested menu did not suit the user's lifestyle, there was the problem of having to resubmit the request, which was time-consuming. Furthermore, since the system was unable to suggest menus that took into account the expiration dates of ingredients, food waste could occur.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0358] In this invention, the server includes: a means for recognizing ingredients and seasonings photographed by a camera; The system includes a means for collecting menu candidates previously selected by a user, the user's preferences, allergy information, and schedule information, a means for generating menu candidates using a generative AI model and a prompt that instructs the system to generate a menu that can be cooked from the recognized ingredients and seasonings based on the recognized ingredients and seasonings, the previously selected menu candidates, the user's preferences, allergy information, and schedule information, and a means for notifying the user's terminal of the generated menu candidates, thereby enabling the system to propose optimal menus that match the user's individual preferences and lifestyle. In addition, the collecting means further collects the number of family members, past menu history, or season, and the generating means generates menu candidates using a prompt sentence instructing the means to generate a menu that can be cooked from the recognized ingredients and seasonings based on the collected number of family members, past menu history, or season, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model. The server may also include a means for recognizing the expiration dates of ingredients, and generate a menu by giving priority to ingredients with the earliest expiration dates. This allows for menu suggestions that take the expiration dates of ingredients into consideration, thereby reducing food waste.

[0359] A "camera for recognizing ingredients and seasonings" is a device that uses image analysis technology to recognize the type, shape, color, expiration date, etc. of ingredients and seasonings.

[0360] A "generative AI model" is an artificial intelligence system that generates the optimal menu for a user based on collected data.

[0361] "Means for suggesting to the user" refers to a method or device for notifying the user of the generated menu, and includes notifications on smartphones or tablets, guidance via voice assistants, etc.

[0362] "User's past selection data" is information about menu items and ingredients selected by the user in the past.

[0363] "Preferences" is information about ingredients and types of dishes that the user particularly likes.

[0364] "Allergy information" is information about ingredients to which the user has an allergy.

[0365] "Schedule information" is information about the user's schedule and time, and is obtained from a calendar application or the like.

[0366] A "generative AI model" is an artificial intelligence model that generates prompts and suggests optimal menus based on collected data.

[0367] A "terminal" is an electronic device used by a user, such as a smartphone or tablet.

[0368] The "notification means" refers to a method or device for notifying the user of the generated menu, and includes push notifications and voice guidance.

[0369] This invention is a system that suggests optimal menus to users, and includes a camera for recognizing ingredients and seasonings, generative AI, means for making suggestions to users, means for collecting the user's past selection data, preferences, allergy information, and schedule information, means for generating optimal menus using a generative AI model, and means for notifying the user of the generated menu on their terminal.

[0370] Hardware and software used

[0371] Hardware: Smartphones, tablets, cameras

[0372] Software: Generative AI models, voice assistants

[0373] Data processing and calculation

[0374] server

[0375] The server recognizes the ingredients and seasonings the user has, as captured by the camera. The server also collects the user's past menu selection data, preferences, allergy information, and schedule information. The server also collects the number of family members, past menu history, and current season. This information is entered by the user through a smartphone app or obtained from a calendar app, etc. The collected information is stored in a database and input into the generative AI model.

[0376] The server uses a generative AI model to generate menu options that can be cooked from the recognized ingredients and seasonings based on the collected data.

[0377] Example prompt sentence: "Ingredients owned by the user: pork, lettuce Condiments owned by the user: salt, pepper User's past selection data: curry, stew, ginger pork User preference: Light User allergy information: Egg allergy Schedule information: Regular work today, off tomorrow Number of people in the family: 3 Past menu history: Spaghetti, rice ball set meal, ramen Season: Summer Considering this information, please suggest a menu for tonight's dinner."

[0378] The generated menu candidates are transmitted from the server to the user's terminal.

[0379] Terminal

[0380] The device (smartphone or tablet) receives multiple menu options sent from the server. The received menu is displayed to the user as a notification. For example, when you wake up in the morning, a notification such as "Today's dinner is chicken curry or shrimp fried rice" is displayed on your smartphone. If the device has a voice assistant function, it can also guide you by voice, saying "Today's dinner is chicken curry or shrimp fried rice."

[0381] User

[0382] The user checks the menu options displayed on their smartphone or tablet. If the user finds a menu option they like, they select it. They can then prepare the selected menu option for dinner. If the user does not like any of the suggested menu options, they can send a request to the server again and receive new menu options.

[0383] Specific examples

[0384] 1. At 7:00 a.m., a notification appears on the user's smartphone saying, "Today's dinner is chicken curry or shrimp fried rice."

[0385] 2. When the user returns home, the smart speaker will say, "Welcome home. Tonight's dinner is chicken curry or shrimp fried rice."

[0386] 3. If the user does not like either the chicken curry or the shrimp fried rice, they can send a request from their smartphone saying, "Please suggest a new menu item," and the server will generate new menu options and notify them again.

[0387] In this way, the system collects user information, uses the generative AI model to propose an optimal menu, and notifies the user. The user can check the proposed menu and resubmit the request if necessary. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[0388] Step 1: Collect user information

[0389] Subject: Server

[0390] The server recognizes the ingredients and seasonings the user owns from images taken with the user's smartphone camera. The server also collects the user's past menu selections, preferences, allergy information, and schedule information. The server also collects the number of family members, past menu history, and current season. This information is entered by the user through a smartphone app or obtained from a calendar app, etc.

[0391] Input: Information entered by the user into a smartphone app, schedule information obtained from a calendar app

[0392] Data processing: Store the entered information in a database and convert it into the required format.

[0393] Output: User information stored in the database

[0394] Specific behavior:

[0395] The user takes a photo of ingredients and seasonings that the user has with the camera of their smartphone. Users enter their preferences and allergy information into a smartphone app.

[0396] The server retrieves schedule information from the calendar app. The server obtains the number of people in the user's family, past menu history, and the current season.

[0397] The server stores this information in a database.

[0398] Step 2: Submit a request to create a menu

[0399] Subject: Terminal

[0400] When a user requests a menu, the device (smartphone or tablet) sends a request to the server, which includes information such as the user's ID and the current date and time.

[0401] Input: User ID, current date and time

[0402] Data processing: Converting request data into a format to send to the server

[0403] Output: Request data sent to the server

[0404] Specific behavior:

[0405] The user presses a button on a smartphone app that says, "Suggest today's dinner menu."

[0406] The device sends a request to the server containing the user's ID and the current date and time.

[0407] Step 3: Create a menu

[0408] Subject: Server

[0409] The server uses a generative AI model to generate an optimal menu for the user. Specifically, the server generates a prompt sentence instructing the user to generate a menu that can be cooked from the recognized ingredients and seasonings based on the collected information on the number of family members, past menu history, or season, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and then uses the prompt sentence and the generative AI model to suggest an optimal menu.

[0410] Input: User information stored in the database, prompt text

[0411] Data processing: Input a prompt sentence into the generative AI model to generate multiple menu options

[0412] Output: Multiple generated menu options

[0413] Specific behavior:

[0414] The server inputs a prompt sentence into the generative AI model.

[0415] The generative AI model returns suggested menu options to the server.

[0416] Step 4: Menu Notifications

[0417] Subject: Terminal

[0418] The terminal receives the multiple menu options sent from the server and notifies the user of them in a manner that varies depending on the user's settings.

[0419] Input: Menu suggestions sent from the server

[0420] Data processing: Convert the received menu into a notification format

[0421] Output: Menu suggestions notified to the user

[0422] Specific behavior:

[0423] The terminal receives the menu candidates sent from the server.

[0424] Depending on the user's settings, the device will display "Chicken curry or shrimp fried rice for dinner tonight" in the notification bar.

[0425] If the device has a voice assistant function, it will say aloud, "Today's dinner is chicken curry, shrimp fried rice, or shrimp fried rice."

[0426] Step 5: Menu review and feedback

[0427] Subject: User

[0428] The user checks the notified menu candidates, and if they like it, selects it and prepares it as is. If they do not like it, they can send the request again and receive new menu candidates.

[0429] Input: Notified menu, user feedback

[0430] Data transformation: Generate new requests based on user feedback

[0431] Output: New menu request

[0432] Specific behavior:

[0433] The user checks the notification on their smartphone and sees that it says, "Today's dinner is chicken curry or shrimp fried rice."

[0434] If the user doesn't like the menu, they can press the "Suggest a new menu" button within the app.

[0435] The terminal again sends a request to the server and receives a new menu.

[0436] Note that an emotion engine, which will be described later, may be used to recognize the user's emotion and generate a cookable menu by further taking the user's emotion into consideration. Specifically, the server further includes means for recognizing the user's emotion using an emotion engine, and the generating means generates menu candidates using a prompt sentence instructing the generation of a cookable menu from the recognized ingredients and seasonings based on the recognized emotion of the user, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model.

[0437] (Application example 3)

[0438] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0439] Conventional menu suggestion systems have difficulty proposing optimal menus for users because they do not fully consider user preferences or past order history. Furthermore, there are limited ways to effectively notify users of suggested menus, preventing flexible suggestions tailored to the user's lifestyle. Furthermore, they are unable to suggest menus that take into account the expiration dates of ingredients, which can result in food waste.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0441] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, a means for generating a menu taking into account the user's preferences and past order history, and a means for notifying or announcing the generated menu by voice. This allows for optimal menu suggestions that take into account the user's preferences and past order history, allowing for flexible suggestions tailored to the user's lifestyle. Furthermore, menu suggestions that take into account the expiration dates of ingredients can reduce food waste.

[0442] A "camera for recognizing ingredients and seasonings" is a photographic device used to identify the type, shape, color, etc. of ingredients and seasonings.

[0443] "Generative AI" is an artificial intelligence system that automatically generates appropriate menus based on recognized information about ingredients and seasonings.

[0444] The "means for proposing the generated menu to the user" refers to a method or device for notifying or displaying the generated menu to the user.

[0445] "Means for generating a menu taking into consideration the user's preferences and past order history" refers to a method or device for generating an optimal menu based on the user's past order history and preference data.

[0446] The "means for notifying or providing voice guidance of the generated menu" refers to a method or device for notifying or providing voice guidance of the generated menu to the user.

[0447] "Parameters such as the number of family members, past menu history, and season" are elements such as the number of family members, past menu history, and season that are taken into consideration when generating a menu.

[0448] "Means for recognizing the expiration date of ingredients" refers to a method or device for identifying the expiration date of ingredients and creating a menu based on that information.

[0449] The "means for generating a menu by giving priority to ingredients with the earliest expiration date" refers to a method or device for generating a menu by giving priority to ingredients with the nearest expiration date.

[0450] A system for implementing this invention includes a camera for recognizing ingredients and seasonings, a generation AI, a means for suggesting the generated menu to the user, a means for generating a menu taking into account the user's preferences and past ordering history, and a means for notifying or providing audio guidance about the generated menu.

[0451] The server first uses a camera to recognize ingredients and seasonings. The camera is used to identify the type, shape, color, etc. of the ingredients and seasonings. The recognized information is sent to the generative AI.

[0452] Generative AI automatically generates an appropriate menu based on the recognized information on ingredients and seasonings, taking into account parameters such as the user's preferences, past order history, number of family members, and season. This allows the system to generate the optimal menu for the user.

[0453] The generated menu is then proposed to the user. This can be done by sending a notification to a device such as a smartphone or tablet, or by voice guidance. For example, when you wake up in the morning, you can receive a notification on your smartphone suggesting that day's dinner menu. Or, when you get home, you can receive a voice prompt with that day's dinner menu, making it possible to make proposals tailored to the user's lifestyle.

[0454] The camera also recognizes the expiration date of ingredients, allowing the system to prioritize ingredients with the earliest expiration date when generating menus, thereby reducing food waste.

[0455] For example, if a user likes "sushi, ramen, tempura" and their past order history is "sushi, ramen," the generative AI is likely to suggest "tempura." This suggestion is sent to the user's smartphone at 7:00 AM.

[0456] An example of a prompt to input to a generative AI model is as follows:

[0457] The user's preferences are "Sushi, Ramen, Tempura" and their past order history is "Sushi, Ramen." Please suggest the best dinner menu for this user.

[0458] This system makes it possible to propose optimal menus that take into account the user's preferences and past order history, allowing for flexible proposals that fit the user's lifestyle. It also reduces food waste by proposing menus that take into account the expiration dates of ingredients.

[0459] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0460] Step 1:

[0461] The server uses a camera to recognize ingredients and seasonings. The camera is used to identify the type, shape, color, etc. of ingredients and seasonings. The input is image data captured by the camera, and the output is information about the recognized ingredients and seasonings. This information is then processed using an image processing algorithm.

[0462] Step 2:

[0463] The server sends the information about the recognized ingredients and seasonings to the generative AI. The input is the information about the ingredients and seasonings obtained from the camera, and the output is the data passed to the generative AI. The data is sent via an API.

[0464] Step 3:

[0465] Generative AI automatically generates appropriate menus based on recognized information about ingredients and seasonings. Inputs include information about ingredients and seasonings, parameters such as the user's preferences, past order history, number of family members, and seasonings. The output is a generated menu. The generative AI model analyzes this data based on prompt statements to generate the optimal menu.

[0466] Step 4:

[0467] The server proposes the generated menu to the user. The input is the menu obtained from the generative AI, and the output is the menu that is notified or voice-guided to the user. The notification is sent to a smartphone or tablet, and the voice guidance is provided using speech synthesis software.

[0468] Step 5:

[0469] The server uses a camera to recognize the expiration date of ingredients. The input is image data captured by the camera, and the output is the recognized expiration date. The expiration date data is extracted using an image processing algorithm.

[0470] Step 6:

[0471] The generative AI generates menus by prioritizing ingredients with the earliest expiration date. The inputs include expiration date information, information on ingredients and seasonings, user preferences, past order history, number of family members, seasonings, and other parameters. The output is an optimal menu that takes expiration dates into consideration. The generative AI model analyzes this data based on prompt statements and generates a menu that takes expiration dates into consideration.

[0472] Step 7:

[0473] The user checks the proposed menu and places an order if necessary. The input is a menu announced or voice-guided by the server, and the output is the user's order information. The user places an order using a smartphone or tablet.

[0474] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0475] "Example 1"

[0476] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes a user's emotions and generates a menu based on those emotions. Specifically, if the user expresses joy or excitement, the emotion engine assumes it is a special occasion or celebration and generates a luxurious menu. Conversely, if the user expresses sadness or depression, the emotion engine assumes they are not feeling well and generates a menu that is easy to digest.

[0477] "Example 2"

[0478] The emotion engine can also change the way menu suggestions are made based on the user's emotions.

[0479] For example, if a user expresses emotions such as joy or excitement, the emotion engine will suggest a menu item by voice and share that joy. Conversely, if a user expresses emotions such as sadness or depression, the emotion engine will silently send a notification to a device such as a smartphone or tablet. This makes it possible to provide services that take into account the user's emotions.

[0480] "Example 3"

[0481] Furthermore, the emotion engine can learn changes in a user's emotions and optimize menu generation and suggestion methods accordingly. For example, if a user tends to always select a particular ingredient after expressing happiness, the emotion engine can learn that tendency and prioritize generating a menu that includes that ingredient the next time the user expresses happiness. This allows the system to better understand the user's emotions and preferences and provide a more personalized service.

[0482] The processing flow of each embodiment will be described below.

[0483] "Example 1"

[0484] Step 1: The emotion engine recognizes the user's emotion.

[0485] Step 2: The emotion engine generates a menu based on the user's emotions. Specifically, if the user expresses joy or excitement, the emotion engine assumes it's a special occasion or celebration and generates a luxurious menu. Conversely, if the user expresses sadness or depression, the emotion engine assumes they're not feeling well and generates a menu that's easy to digest.

[0486] "Example 2"

[0487] Step 1: The emotion engine recognizes the user's emotion.

[0488] Step 2: The emotion engine changes the menu suggestion method based on the user's emotions. For example, if the user expresses happiness or excitement, the emotion engine will suggest a menu by voice, sharing that joy. Conversely, if the user expresses sadness or depression, the emotion engine will silently send a notification to the device, such as a smartphone or tablet.

[0489] "Example 3"

[0490] Step 1: The emotion engine recognizes the user's emotion.

[0491] Step 2: The emotion engine learns the user's emotional changes.

[0492] Step 3: The emotion engine optimizes menu generation and suggestion methods based on its learning. For example, if a user tends to always select a particular ingredient after expressing happiness, the emotion engine will learn that tendency and prioritize generating a menu that includes that ingredient the next time the user expresses happiness.

[0493] Example 1

[0494] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0495] Conventional food management systems and menu generation systems are limited to recognizing ingredients and suggesting menus, and have the problem of not being able to suggest menus that take the user's emotions into account. Furthermore, menu suggestions that take into account the expiration date of ingredients are insufficient, which can result in food waste. Furthermore, menu generation has not taken into account parameters such as the number of family members, past menu history, or season. To solve these issues, a system that recognizes the user's emotions and adjusts the menu based on them is needed.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0497] In this invention, the server includes a photographing device for recognizing ingredients and seasonings, a generative artificial intelligence for generating a menu based on the recognized ingredients and seasonings, a means for proposing the generated menu to a user, an emotion analysis device for recognizing the user's emotion, and a means for adjusting the menu based on the user's emotion. This makes it possible to propose a menu that takes the user's emotion into consideration, reducing ingredient waste and realizing more personalized menu proposals.

[0498] A "photography device" is a device that captures images or videos to recognize ingredients and seasonings.

[0499] "Generative AI" is an AI technology that generates menus based on recognized ingredients and seasonings.

[0500] The "means for suggesting" is a means for notifying or displaying the generated menu to the user.

[0501] An "emotion analysis device" is a device that analyzes data such as facial expressions and voice to recognize a user's emotions.

[0502] The "adjusting means" is a means for appropriately changing or adjusting the generated menu based on the user's feelings.

[0503] "Expiration date" is information indicating the date by which ingredients or seasonings can be safely consumed.

[0504] "Parameters" are factors that affect menu generation, such as the number of family members, past menu history, and season.

[0505] The present invention relates to a system that recognizes ingredients and seasonings and generates a menu based on a user's emotions. The system includes a photographing device, generative artificial intelligence, an emotion analysis device, a suggestion unit, and an adjustment unit.

[0506] System configuration

[0507] 1. Imaging equipment

[0508] The server receives image data transmitted from the image capturing device installed in the refrigerator or pantry.

[0509] The camera captures information about ingredients by reading barcodes and QR codes, and uses image recognition technology to estimate the type and quantity of ingredients.

[0510] 2. Generative Artificial Intelligence

[0511] The server generates a menu using generative artificial intelligence based on the information about ingredients acquired from the camera.

[0512] Generative AI generates menus taking into account parameters such as the number of family members, past menu history, and the season.

[0513] 3. Emotion analysis device

[0514] The device captures the user's facial expressions and voice through a camera and microphone.

[0515] The server analyzes the user's emotions using an emotion analysis device, for example, using facial expression recognition technology or voice analysis technology.

[0516] 4. Proposed measures

[0517] The server provides a means for notifying or displaying the generated menu to the user, for example, by displaying the menu through a smartphone app or display.

[0518] 5. Adjustment measures

[0519] The server provides a means to appropriately modify or adjust the generated menu based on the user's sentiment.

[0520] If the user shows emotions of joy or excitement, the emotion analysis device generates a luxurious menu for a special occasion or celebration. Conversely, if the user shows emotions of sadness or depression, the device generates a menu that is easy to digest and assumes that the user is not feeling well.

[0521] Specific examples

[0522] Example 1: A camera installed in a refrigerator reads the barcode and acquires information about "milk" and "eggs." The server stores this information in a database.

[0523] How it works: The camera scans the barcodes of the "Milk" and "Eggs" items in the refrigerator and sends the information to the server, which stores it in a database and updates the inventory.

[0524] Example 2: A user smiles into the device's camera. The server uses an emotion analyzer to recognize the emotion "joy" and generates a luxurious dinner menu.

[0525] How it works: The device's camera captures the user's smile and sends the data to the server, which uses an emotion analyzer to analyze the emotion of "happiness" and suggest a luxurious dinner menu.

[0526] Prompt Sentence Examples

[0527] Prompt 1: "Use a camera to identify ingredients in the refrigerator and suggest menu items based on the user's emotions."

[0528] Prompt 2: "If the user is sad, generate a digestible menu."

[0529] This system supports daily meal preparation by automating ingredient management and menu suggestions based on the user's emotions.

[0530] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0531] Step 1:

[0532] Ingredient recognition

[0533] Input: Image data from imaging devices installed in refrigerators and pantries.

[0534] Specific operation: The camera scans the barcodes and QR codes on ingredients and seasonings in the refrigerator and sends the information to the server.

[0535] Data processing: The server analyzes the received image data and extracts information about ingredients from barcodes and QR codes.

[0536] Output: Information on the extracted ingredients (type, quantity, expiration date, etc.).

[0537] Step 2:

[0538] Saving food information

[0539] Input: Ingredient information extracted in step 1.

[0540] Specific operation: The server saves the extracted ingredient information in the database and updates the inventory information.

[0541] Data processing: Store information such as the type of ingredients, quantity, and expiration date in a database.

[0542] Output: Updated database inventory information.

[0543] Step 3:

[0544] Emotion recognition

[0545] Input: User facial and voice data captured from the device's camera and microphone.

[0546] Specific operation: When the user speaks to the camera or shows facial expressions, the device captures the data and sends it to the server.

[0547] Data processing: The server uses an emotion analysis device to analyze facial expressions and voice data to identify the user's emotions.

[0548] Output: The user's emotional state (happy, sad, excited, depressed, etc.).

[0549] Step 4:

[0550] Generate Menu

[0551] Input: Inventory information updated in step 2 and the user's emotional state identified in step 3.

[0552] Specific operation: The server uses generative artificial intelligence to generate a menu based on inventory information and the user's emotional state.

[0553] Data processing: Menus are generated taking into account parameters such as the number of family members, past menu history, and season.

[0554] Output: The generated menu.

[0555] Step 5:

[0556] Menu suggestions

[0557] Input: The menu generated in step 4.

[0558] Specific operation: The server provides a means to notify or display the generated menu to the user, for example, by displaying the menu through a smartphone app or display.

[0559] Data processing: Converts menu information into an appropriate format for display.

[0560] Output: The menu suggested to the user.

[0561] Step 6:

[0562] Menu adjustments

[0563] Input: The menu proposed in step 5 and user feedback.

[0564] Specific Actions: The server provides a means to appropriately change or adjust the generated menu based on the user's emotions.

[0565] Data processing: Analyze user feedback and regenerate or adjust menus as needed.

[0566] Output: Adjusted menu.

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] Conventional food management systems were able to recognize ingredients and seasonings in the refrigerator or pantry and generate menus, but they lacked the ability to suggest menus that took the user's emotions into account or arrange food delivery based on the generated menus. As a result, they were unable to suggest appropriate menus based on the user's emotions and physical condition, and it was difficult to arrange those menus quickly. This made it difficult to improve user satisfaction.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0571] In this invention, the server includes a camera means for recognizing ingredients and seasonings, a generative AI means for generating a menu based on the recognized ingredients and seasonings, a means for proposing the generated menu to the user, an emotion recognition means for recognizing the user's emotion, a means for adjusting the menu based on the emotion recognition means, and a means for arranging food delivery based on the adjusted menu. This makes it possible to propose an appropriate menu based on the user's emotion and physical condition, and further to quickly arrange the menu.

[0572] A "camera for recognizing ingredients and seasonings" is a photographic device that is installed in a refrigerator or pantry to recognize the type, quantity, expiration date, etc. of ingredients and seasonings.

[0573] "Generative AI" is an artificial intelligence system that generates appropriate menus based on information about recognized ingredients and seasonings.

[0574] "Means for proposing the generated menu to the user" refers to an interface or device for notifying and proposing the menu generated by generative AI to the user.

[0575] "Emotion recognition means for recognizing user emotions" refers to technology or devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[0576] The "means for adjusting the menu based on the emotion recognition means" is a system for appropriately changing and adjusting the generated menu according to the recognized user emotion.

[0577] "Means for arranging food delivery based on a coordinated menu" means a system for ordering and arranging ingredients and dishes through a food delivery service based on a coordinated menu.

[0578] A system for implementing this invention is configured as follows. First, a camera installed in the refrigerator or pantry recognizes ingredients and seasonings. The camera obtains information about ingredients by reading barcodes or QR codes, and estimates the type and quantity of ingredients using image recognition technology. This camera can use, for example, the "cv2.VideoCapture" or "pyzbar" library.

[0579] Next, the information on the recognized ingredients and seasonings is sent to a generative AI, which generates an appropriate menu. The generative AI uses the "RecipeGenerator" class, for example, to generate a menu based on the ingredient information. Parameters such as the number of family members, past menu history, and the season are also taken into account.

[0580] The generated menu is presented to the user via a smartphone application, tablet device, or other device, allowing the user to check the generated menu.

[0581] Furthermore, an emotion recognition means is used to recognize the user's emotions. The emotion recognition means, for example, uses the "EmotionRecognizer" class to analyze and recognize emotions from the user's facial expressions and voice. Based on the recognized emotions, the generated menu is appropriately adjusted. For example, if the user shows emotions of joy or excitement, a luxurious menu is suggested, whereas if the user shows emotions of sadness or depression, a menu that is easy to digest is suggested.

[0582] A means for arranging food delivery is used based on the adjusted menu. The food delivery arrangement means uses, for example, the "FoodDeliveryService" class to order and arrange ingredients and dishes based on the adjusted menu. This allows users to quickly receive appropriate meals from the comfort of their own homes.

[0583] As a concrete example, a user scans the ingredients in their refrigerator with their camera and takes a photo of their face with their smartphone camera. The app retrieves that information, suggests the best menu based on the user's emotions, and arranges food delivery based on that menu. An example prompt is, "Scan the ingredients in your refrigerator with your camera and take a photo of your face with your smartphone camera. The app will suggest the best menu based on your emotions and arrange food delivery based on that menu."

[0584] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0585] Step 1:

[0586] The server obtains information about ingredients and seasonings from cameras installed in the refrigerator or pantry. The camera obtains information about ingredients by reading barcodes and QR codes, and uses image recognition technology to estimate the type and quantity of ingredients. The input is video data from the camera, and the output is information about the recognized ingredients and seasonings. Specifically, the server analyzes the video data using the "cv2.VideoCapture" and "pyzbar" libraries and extracts ingredient information.

[0587] Step 2:

[0588] The server sends the acquired information on ingredients and seasonings to the generative AI. The generative AI generates a menu based on the ingredient information. The input is information on ingredients and seasonings, and the output is the generated menu. Specifically, it uses the "RecipeGenerator" class to generate an appropriate menu based on the ingredient information. Parameters such as the number of family members, past menu history, and season are also taken into consideration.

[0589] Step 3:

[0590] The server proposes the generated menu to the user. The proposal method is a smartphone application or a tablet device. The input is the generated menu, and the output is the menu proposal to the user. Specifically, the server notifies the smartphone or tablet device of the generated menu so that the user can check it.

[0591] Step 4:

[0592] The device uses an emotion recognition means to recognize the user's emotions. The emotion recognition means analyzes and recognizes emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the recognized emotion. Specifically, it uses the "EmotionRecognizer" class to analyze the user's facial expressions and voice and recognize emotions.

[0593] Step 5:

[0594] The server adjusts the generated menu based on the recognized emotion. The input is the recognized emotion and the generated menu, and the output is the adjusted menu. Specifically, the generated menu is modified and adjusted according to the recognized emotion. For example, if the user expresses joy or excitement, a luxurious menu is suggested, while if the user expresses sadness or depression, a menu that is easy to digest is suggested.

[0595] Step 6:

[0596] The server arranges food delivery based on the adjusted menu. The input is the adjusted menu, and the output is the food delivery order status. Specifically, it uses the "FoodDeliveryService" class to order and arrange ingredients and dishes based on the adjusted menu. This allows users to quickly receive appropriate meals from the comfort of their own home.

[0597] Example 2

[0598] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0599] Conventional menu generation systems can generate menus based on the recognition of ingredients and seasonings, but they have the problem of not being able to propose menus that take into account parameters such as the user's emotions, the number of family members, past menu history, and the season. Also, because the proposal method cannot be changed according to the user's emotions, it is difficult to increase user satisfaction.

[0600] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0601] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, and an emotion engine for changing the menu suggestion method based on the user's emotions. This makes it possible to generate menus taking into account parameters such as the number of family members, past menu history, and season, and also to change the suggestion method according to the user's emotions.

[0602] The "imaging device" is a device used to recognize ingredients and seasonings.

[0603] "Generative AI" is AI that generates menus based on recognized ingredients and seasonings.

[0604] The "proposing means" is a means for proposing the generated menu to the user.

[0605] The "emotion engine" is an engine for changing the way menus are suggested based on the user's emotions.

[0606] "Parameters" are factors that affect menu generation, such as the number of family members, past menu history, and season.

[0607] "Expiration date" refers to the date by which food can be safely consumed.

[0608] The present invention relates to a system for recognizing ingredients and seasonings and generating menus based on the ingredients and seasonings. Specific embodiments of this system will be described below.

[0609] System configuration

[0610] The system consists of the following major components:

[0611] 1. Imaging equipment:

[0612] This is a device for recognizing ingredients and seasonings. For example, it can use the camera built into a smartphone or tablet.

[0613] 2. Generative AI:

[0614] It is an artificial intelligence that generates menus based on recognized ingredients and seasonings. Specifically, it is implemented using machine learning libraries such as Python and TensorFlow. GPT-4 and other generative AI models are used.

[0615] 3. Proposed method:

[0616] This is a means for suggesting the generated menu to the user. For example, there are methods such as displaying the menu on the screen of a smartphone or tablet, or suggesting it by voice.

[0617] 4. Emotion Engine:

[0618] It is an engine that changes the way it suggests menus based on the user's emotions, for example, if the user is happy or excited, it will make audible suggestions, and if the user is sad or depressed, it will send silent notifications.

[0619] System Operation

[0620] Data collection:

[0621] Using a smartphone app or web interface, users input information about the ingredients and seasonings they currently have on hand—for example, "tomatoes, chicken, basil"—as well as additional parameters such as the number of people in their household, past menu history, and the season.

[0622] Data transmission:

[0623] The device sends the data entered by the user to the server, along with additional parameters such as the number of family members, past menu history, and season.

[0624] Data preprocessing:

[0625] The server receives and standardizes the data sent from the device, for example, by checking the type and quantity of ingredients and taking expiration dates into account.

[0626] Menu generation:

[0627] The server uses generative artificial intelligence to generate menus based on the collected data, taking into account parameters such as the type and quantity of ingredients, expiration dates, number of family members, past menu history, and the season to create the optimal menu.

[0628] Applying the Emotion Engine:

[0629] The server uses an emotion engine to determine how to suggest menu items based on the user's emotions, for example, suggesting menu items through voice if the user is expressing happiness or excitement, or sending silent notifications if the user is expressing sadness or depression.

[0630] Menu suggestion:

[0631] The server sends the generated menu to the device. The device then presents the menu received from the server to the user. For example, a recipe for "Chicken pasta with tomatoes and basil" is displayed on the smartphone screen. If the user shows a happy emotion, the device will suggest by voice, "Let's make chicken pasta with tomatoes and basil today!"

[0632] Examples of concrete examples and prompts

[0633] Examples:

[0634] A user inputs "tomato, chicken, basil" through a smartphone app and sets the number of people in their family as four. The device sends this information to the server. The server recognizes from past menu history that the family tends to like Italian food, and taking into account that it is summer, generates the following menu:

[0635] Menu: Chicken pasta with tomato and basil

[0636] Recipe: Stir-fry chicken, add tomatoes and basil, and toss with pasta

[0637] The server uses an emotion engine to detect that the user is expressing joy, and suggests in voice, "Let's make chicken pasta with tomatoes and basil today!" The device plays this voice message to the user, while simultaneously displaying the recipe on the smartphone screen.

[0638] Example prompt sentence:

[0639] "Generate a menu using tomatoes, chicken, and basil to make a meal for a family of four. Past menu history shows that your family prefers Italian food, and it's summer."

[0640] In this way, the system can generate optimal menus based on the user's input data and additional parameters, and suggest them to the user in an appropriate way using the emotion engine.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The user inputs data on ingredients and seasonings.

[0644] Using a smartphone app or web interface, users input information about the ingredients and seasonings they currently have on hand—for example, "tomatoes, chicken, and basil"—as well as additional parameters such as the number of family members, past menu history, and season. Input data includes the type of ingredient, quantity, expiration date, number of family members, past menu history, and season.

[0645] Step 2:

[0646] The terminal sends the input data to the server.

[0647] The terminal sends the data entered by the user to the server, along with additional parameters such as the number of family members, past menu history, and season. The input data includes the type of ingredient, amount, expiration date, number of family members, past menu history, and season, and the output is the data sent to the server.

[0648] Step 3:

[0649] The server receives the data and performs pre-processing.

[0650] The server receives data sent from the terminal. It standardizes the received data and organizes it taking into account the type and quantity of ingredients and their expiration date. For example, it checks specific quantities such as 3 tomatoes, 500 grams of chicken, and 10 basil leaves. The input data is the data sent from the terminal, and the output is standardized data.

[0651] Step 4:

[0652] The server generates the menu using a generative AI model.

[0653] The server generates a menu using a generative AI model (e.g., GPT-4) based on the collected data. It generates the optimal menu by taking into account parameters such as the type and amount of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in a family, it generates a menu for cooking meals for four people. The input data is standardized data, and the output is the generated menu.

[0654] Step 5:

[0655] The server uses an emotion engine to determine the proposal method.

[0656] The server uses an emotion engine to determine how to suggest a menu based on the user's emotions. For example, if the user expresses joy or excitement, the emotion engine will suggest a menu by voice. Conversely, if the user expresses sadness or depression, the emotion engine will silently send a notification to a device such as a smartphone or tablet. The input data is the user's emotion information and the generated menu, and the output is the suggestion method.

[0657] Step 6:

[0658] The server sends the generated menu to the terminal.

[0659] The server sends the generated menu to the terminal. The data sent includes specific recipes and cooking instructions. The input data is the generated menu and the proposed method, and the output is the data sent to the terminal.

[0660] Step 7:

[0661] The terminal presents the user with a menu.

[0662] The device presents the menu received from the server to the user. For example, a recipe for "Chicken pasta with tomatoes and basil" is displayed on the smartphone screen. If the user shows a happy emotion, the device will suggest by voice, "Let's make chicken pasta with tomatoes and basil today!" The input data is the data received from the server, and the output is the menu presented to the user.

[0663] (Application example 2)

[0664] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0665] Conventional menu generation systems can generate menus based on the recognition of ingredients and seasonings, but they cannot respond to user emotions or missing ingredient requests. Therefore, there is a need for menu suggestions that take user emotions into consideration and for automated ordering of necessary ingredients. Furthermore, menu generation that takes into account parameters such as the user's family composition, past menu history, and seasons is also insufficient.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0667] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, an emotion engine for recognizing the user's emotions, a means for changing the menu suggestion method based on the user's emotions, and a delivery service linkage means for ordering missing ingredients. This makes it possible to propose a menu that takes the user's emotions into consideration and automate the ordering of necessary ingredients.

[0668] A "camera for recognizing ingredients and seasonings" is a device that captures the type and condition of ingredients and seasonings as images and analyzes them.

[0669] "Generative AI" is artificial intelligence that automatically generates optimal menus based on information about recognized ingredients and seasonings.

[0670] "Means for proposing the generated menu to the user" refers to a method or device for informing the user of the menu generated by the generative AI.

[0671] An "emotion engine for recognizing a user's emotions" is a device or software that analyzes the user's emotions from their facial expressions, voice, etc., and recognizes their state.

[0672] The "means for changing the method of suggesting menu items based on the user's emotions" refers to a method or device for changing the method of suggesting menu items in accordance with the user's emotions recognized by the emotion engine.

[0673] A "delivery service collaboration means for ordering missing ingredients" is a method or device for collaborating with an external delivery service to order ingredients required for a generated menu when those ingredients are missing.

[0674] "Family size" is the number of people in the user's household that will be taken into account when generating the menu.

[0675] "Past menu history" is a record of menu items that a user has previously selected or consumed.

[0676] "Season" refers to information about the current time or season that is taken into consideration when generating a menu.

[0677] A "use-by date" is the date by which food ingredients or seasonings can be safely consumed, after which the quality may deteriorate.

[0678] The following system configuration will be described as an embodiment of the present invention.

[0679] System Configuration

[0680] 1. Hardware Configuration

[0681] Camera: A camera is used to recognize ingredients and seasonings. The camera can be a built-in camera on a smartphone or tablet.

[0682] Smartphone or tablet: Used as a device for users to operate the system.

[0683] Server: A server is used to run the generative AI, emotion engine, and delivery service integration means.

[0684] 2. Software Configuration

[0685] Generative AI: Artificial intelligence that automatically generates optimal menus based on recognized information about ingredients and seasonings. Implemented in programming languages ​​such as Python.

[0686] Emotion engine: Software that analyzes the user's emotions from facial expressions, voice, etc. and recognizes their state. It uses libraries such as EmotionEngine.

[0687] Delivery service integration: API integration function for ordering missing ingredients. Uses DeliveryService API, etc.

[0688] Processing flow

[0689] 1. Recognizing ingredients

[0690] The user takes a photo of the food in the refrigerator using the camera on their smartphone or tablet.

[0691] Images taken by the camera are sent to the server, where ingredients and seasonings are recognized using the FoodRecognition library.

[0692] 2. Creating a Menu

[0693] The generative AI generates the optimal menu based on parameters such as recognized ingredients and seasoning information, number of family members, past menu history, and season.

[0694] The generated menu is displayed on the user's smartphone or tablet.

[0695] 3. Emotion-based recommendations

[0696] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions.

[0697] The way the menu is suggested changes depending on the user's emotions. For example, if the user is happy, suggestions are made by voice, and if the user is depressed, suggestions are made by notification.

[0698] 4. Ordering missing ingredients

[0699] If any ingredients required for the generated menu are missing, the missing ingredients are ordered using a delivery service linkage means.

[0700] Orders are placed via the user's smartphone or tablet.

[0701] Specific examples

[0702] The user takes a photo of the ingredients in the refrigerator with their smartphone. The app recognizes the ingredients and generates a menu taking into account the number of family members and past menu history. If the user is happy, it will suggest "Today's menu is curry!" via voice. If the necessary ingredients are missing, it will place an order with a food delivery service.

[0703] Prompt Sentence Examples

[0704] Your goal is to take a photo of the ingredients in the refrigerator taken by the user and generate the optimal menu taking into account the number of family members, past menu history, and the season.You also need to change the way you suggest menus depending on the user's emotions, and order food from a food delivery service if the necessary ingredients are in short supply.

[0705] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0706] Step 1:

[0707] The user takes a photo of the food in the refrigerator using the camera on their smartphone or tablet.

[0708] Input: Images of ingredients in the refrigerator

[0709] Output: Captured image data

[0710] Specific operation: The user launches the camera app and takes a picture of the food in the refrigerator. The image is saved on the device.

[0711] Step 2:

[0712] The terminal transmits the captured image data to the server.

[0713] Input: Captured image data

[0714] Output: Image data sent to the server

[0715] Specific operation: The device uploads image data to a server via the Internet.

[0716] Step 3:

[0717] The server receives the image data and uses the FoodRecognition library to recognize ingredients and seasonings.

[0718] Input: Image data sent to the server

[0719] Output: A list of recognized ingredients and seasonings

[0720] How it works: The server analyzes the image data and identifies ingredients and seasonings using the FoodRecognition library. The recognition results are saved in a list format.

[0721] Step 4:

[0722] The server uses generative AI to generate the optimal menu based on parameters such as information on ingredients and seasonings recognized, the number of family members, past menu history, and the season.

[0723] Input: List of recognized ingredients and seasonings, number of family members, past menu history, seasonal information

[0724] Output: The generated menu

[0725] Specific operation: The server inputs the necessary parameters into the generative AI and generates the optimal menu. The generated menu is saved in the database.

[0726] Step 5:

[0727] The server transmits the generated menu to the terminal, which displays the menu.

[0728] Input: The generated menu

[0729] Output: Menu displayed on the terminal

[0730] Specific operation: The server sends the generated menu to the terminal, and the terminal displays the menu to the user.

[0731] Step 6:

[0732] The device captures the user's facial expressions and voice using a camera and microphone and sends them to a server.

[0733] Input: User facial and voice data

[0734] Output: Facial expression and voice data sent to the server

[0735] Specific operation: The device uses the camera and microphone to capture the user's facial expressions and voice, and uploads the data to the server.

[0736] Step 7:

[0737] The server uses an emotion engine to analyze and recognize the user's emotions.

[0738] Input: Facial expression and voice data sent to the server

[0739] Output: Perceived user emotion

[0740] Specific operation: The server uses the emotion engine to analyze facial and voice data to identify the user's emotions.

[0741] Step 8:

[0742] The server changes the way it suggests menus depending on the user's emotions and notifies the terminal.

[0743] Input: Recognized user sentiment, generated menu

[0744] Output: A method for suggesting menu items based on user emotions

[0745] Specific operation: Based on the user's emotion, the server decides whether to suggest the menu by voice or notification, and sends instructions to the device. The device then suggests the menu according to the instructions.

[0746] Step 9:

[0747] The server checks whether ingredients necessary for the generated menu are in short supply, and if so, places an order using a delivery service linkage means.

[0748] Input: Generated menu, user's ingredient inventory information

[0749] Output: Order for missing ingredients

[0750] Specific operation: The server compares the generated menu with the user's ingredient inventory information to identify any missing ingredients, and uses the delivery service API to order the missing ingredients.

[0751] Example 3

[0752] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0753] Conventional menu generation systems have the problem that it is difficult to make personalized suggestions because they do not sufficiently consider the user's preferences and emotions. Also, because the menu is not optimized according to changes in the user's emotions, it is not possible to increase user satisfaction.

[0754] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0755] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to the user, an emotion engine for collecting and learning user emotion data, and a means for optimizing menu generation based on the data learned by the emotion engine. This makes it possible to propose personalized menus that take into account the user's preferences and emotions, thereby increasing user satisfaction.

[0756] The "camera" is a photographic device used to recognize ingredients and seasonings.

[0757] "Generative AI" is artificial intelligence that generates menus based on recognized ingredients and seasonings.

[0758] The "means for suggesting" is a method or device for notifying or guiding the user about the generated menu.

[0759] The "emotion engine" is a system for collecting and learning user emotional data.

[0760] The "optimizing means" refers to a method or device for optimizing menu generation based on data learned by the emotion engine.

[0761] "Ingredients" are the ingredients used in cooking.

[0762] A "seasoning" is a substance used to adjust the flavor of food.

[0763] A "menu" is a list of dishes suggested to the user.

[0764] A "user" is someone who uses the system.

[0765] "Data" is a collection of information that is processed by a system.

[0766] This invention is a system that proposes personalized menus to users. The system includes a camera for recognizing ingredients and seasonings, a generative AI that generates menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to users, an emotion engine that collects and learns user emotion data, and a means for optimizing menu generation based on the data learned by the emotion engine.

[0767] The server first uses a camera to recognize ingredients and seasonings. For example, a camera inside the refrigerator takes a picture of the type and quantity of ingredients and sends that data to the server. The server then inputs the recognized ingredient and seasoning data into a generative AI, which then generates a menu taking into account the user's preferences and past selection history. For example, the generative AI might suggest curry rice based on data that shows the user has previously preferred curry rice.

[0768] Next, the server selects the means by which to present the generated menu to the user. For example, in the morning, it may select a method of sending a notification to the smartphone, and in the evening, it may select a method of providing voice guidance. The device notifies the user of the menu sent from the server. For example, when the user wakes up in the morning, a notification may appear on the smartphone saying, "How about curry rice for dinner tonight?" And when the user returns home, a voice prompt may say, "Welcome home. Tonight's dinner is curry rice."

[0769] Furthermore, the server collects user emotional data and uses the emotion engine to learn. For example, if a user expresses joy in response to a menu item, the data is recorded. The emotion engine then optimizes the next menu generation based on the collected emotional data. For example, if a user tends to select a specific ingredient (e.g., an ingredient in curry rice) after expressing joy, the server will prioritize generating a menu that includes that ingredient.

[0770] An example prompt is "Generate a menu that includes ingredients that users tend to select after expressing the emotion of pleasure."

[0771] In this way, the system can deeply understand the user's feelings and preferences and provide more personalized services. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0772] Step 1:

[0773] The server uses a camera to recognize ingredients and seasonings. The camera takes a picture of the ingredients in the refrigerator and sends the image data to the server. The server then uses an image recognition algorithm to identify the type and amount of ingredients and seasonings. The input is the image data from the camera, and the output is a list of recognized ingredients and seasonings.

[0774] Step 2:

[0775] The server inputs the recognized ingredients and seasoning data into a generative AI to generate a menu. The generative AI creates a menu taking into account the user's preferences and past selection history. For example, it suggests curry rice based on data that indicates "the user has previously preferred curry rice." The input is a list of recognized ingredients and seasonings and the user's history data, and the output is the generated menu.

[0776] Step 3:

[0777] The server selects the method for proposing the generated menu to the user. It decides whether to send a notification to the user's smartphone or provide voice guidance depending on the user's lifestyle and time of day. For example, in the morning, it selects the method of sending a notification to the smartphone, and in the evening, it selects the method of providing voice guidance. The input is the generated menu and the user's situation data, and the output is the selection of the proposal method.

[0778] Step 4:

[0779] The device notifies the user of the menu sent from the server. For example, when you wake up in the morning, a notification appears on your smartphone saying, "How about curry rice for dinner tonight?". Also, when you return home, a voice message will say, "Welcome home. Tonight's dinner is curry rice." The input is the suggested method and menu from the server, and the output is a notification or voice guidance to the user.

[0780] Step 5:

[0781] The server collects user emotion data and uses the emotion engine to learn. For example, if a user expresses joy in response to a menu item, the server records the data. The input is the user's emotion feedback, and the output is the learned emotion data.

[0782] Step 6:

[0783] The server optimizes the next menu generation based on the data learned by the emotion engine. For example, if a user tends to select a specific ingredient (such as an ingredient in curry rice) after expressing joy, the server will prioritize generating a menu that includes that ingredient. The input is the learned emotion data, and the output is an optimized menu.

[0784] (Application example 3)

[0785] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0786] Conventional menu generation systems generate menus simply based on the user's perception of ingredients and seasonings, without considering the user's emotions or lifestyle. As a result, they are unable to propose the optimal menu for the user, making it difficult to provide personalized services. Furthermore, because they are unable to propose menus that respond to changes in the user's emotions, they are unable to increase user satisfaction.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0788] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to the user, an emotion engine for analyzing the user's emotions, and a means for optimizing menu generation based on the emotion engine. This makes it possible to propose personalized menus according to the user's emotions and lifestyle, thereby increasing user satisfaction.

[0789] A "camera for recognizing ingredients and seasonings" is a device that captures the type and condition of ingredients and seasonings as images and analyzes them.

[0790] "Generative AI" is artificial intelligence that automatically generates appropriate menus based on information about recognized ingredients and seasonings.

[0791] The "means for proposing the generated menu to the user" refers to a device or software for presenting the generated menu to the user by means of notification, voice guidance, or the like.

[0792] An "emotion engine that analyzes user emotions" is software or hardware that analyzes the user's emotional state and optimizes menu generation based on that information.

[0793] The "means for optimizing menu generation based on an emotion engine" refers to a device or software for generating an optimal menu for a user based on emotion information obtained from the emotion engine.

[0794] A system for implementing this invention has the following configuration: The server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to the user, an emotion engine for analyzing the user's emotions, and a means for optimizing menu generation based on the emotion engine.

[0795] Hardware and Software Configuration

[0796] Camera: A device that captures images of the type and condition of ingredients and seasonings and analyzes them. For example, a smartphone camera or a dedicated image recognition camera can be used.

[0797] Generative AI: This is artificial intelligence that automatically generates appropriate menus based on recognized information about ingredients and seasonings. For example, OpenAI's GPT-3 model is used.

[0798] Emotion engine: Software or hardware that analyzes the user's emotional state and optimizes menu generation based on that information. For example, a sentiment analysis library such as TextBlob is used.

[0799] Notification means: A device or software that presents the generated menu to the user by means of notification or voice guidance. For example, the notification function of a smartphone or a voice assistant is used.

[0800] Data processing and calculation

[0801] 1. Ingredient Recognition: Images captured by the camera are analyzed to identify the type of ingredients and seasonings. This is done using image recognition technology.

[0802] 2. Sentiment Analysis: Analyze the user's input text or voice data and calculate a sentiment score. Use a sentiment analysis library such as TextBlob.

[0803] 3. Menu generation: A generative AI generates a menu based on the emotion scores and recognized ingredient information. It uses OpenAI's GPT-3 model.

[0804] 4. Notification: The generated menu is sent to your smartphone via text message or voice guidance.

[0805] Specific examples

[0806] For example, if a user types, "I'm feeling very happy today!", the emotion engine analyzes this text and calculates a high emotion score. Next, the generative AI generates the optimal menu for the user based on this emotion score and the ingredient information recognized by the camera. The generated menu is displayed on the smartphone as "Today's dinner suggestion: steak and salad."

[0807] Prompt Sentence Examples

[0808] "Please suggest a menu for dinner tonight. Sentiment score is 0.8."

[0809] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0810] Step 1:

[0811] The server uses a camera to capture images of ingredients and seasonings. The input is the image captured by the camera, and the output is image data. This image data is used to identify the type and condition of the ingredients and seasonings.

[0812] Step 2:

[0813] The server uses image recognition technology to analyze the acquired image data and identify the types of ingredients and seasonings. The input is the image data acquired in step 1, and the output is a list of recognized ingredients and seasonings. This list is used to generate the subsequent menu.

[0814] Step 3:

[0815] The user inputs text or voice data that indicates their emotional state into the terminal. The input is the user's text or voice data, and the output is data that indicates their emotional state. This data is analyzed by the emotion engine.

[0816] Step 4:

[0817] The server uses an emotion engine to analyze the user's input data and calculate an emotion score. The input is the data indicating the emotional state obtained in step 3, and the output is an emotion score. This emotion score is used to optimize menu generation.

[0818] Step 5:

[0819] The server generates a menu based on the list of ingredients and seasonings recognized using generative AI and their emotion scores. The input is the list of ingredients and seasonings obtained in step 2 and the emotion scores obtained in step 4, and the output is the generated menu. This menu is then proposed to the user.

[0820] Step 6:

[0821] The server notifies the terminal of the generated menu. The input is the menu generated in step 5, and the output is a notification message to be displayed on the terminal. The notification is made by a text message or a voice guidance.

[0822] Step 7:

[0823] The user checks the notification message displayed on the terminal and decides whether to accept the proposed menu. The input is the menu notified in step 6, and the output is the user's selection. If the user accepts the menu, the meal is prepared based on that menu.

[0824] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt containing an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in the data format of voice data, text data, etc.

[0826] Here, inference can refer to, for example, analysis, classification, prediction, and / or summarization.

[0827] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0828] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0829] [Second embodiment]

[0830] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0831] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0832] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0833] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0834] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0835] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0836] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0837] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0838] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0839] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0840] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0841] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0842] "Example 1"

[0843] In one embodiment of the present invention, a camera installed in a refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera obtains information about ingredients by reading barcodes or QR codes. It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[0844] "Example 2"

[0845] The generative AI generates a menu based on the recognized ingredients and seasonings. The generative AI generates the optimal menu by taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in a family, it generates a menu for cooking meals for four people. It also estimates the family's favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[0846] "Example 3"

[0847] The generated menu is then suggested to the user. The suggestion method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you get home, a voice guide will guide you through that day's dinner menu, making it possible to make suggestions tailored to the user's lifestyle.

[0848] The processing flow of each embodiment will be described below.

[0849] "Example 1"

[0850] Step 1: A camera installed in the refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera can obtain information about ingredients by reading barcodes or QR codes. It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[0851] "Example 2"

[0852] Step 1: Information about ingredients and seasonings obtained from the camera is sent to the generative AI.

[0853] Step 2: The generative AI generates the optimal menu, taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in the family, it generates a menu for cooking meals for four people. It also estimates the family's likely favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[0854] "Example 3"

[0855] Step 1: Receive the menu generated by the generative AI.

[0856] Step 2: The generated menu is proposed to the user. The proposal method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you return home, a voice guide will guide you through that day's dinner menu, making it possible to make proposals that suit the user's lifestyle.

[0857] Example 1

[0858] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0859] Conventional food management in refrigerators and pantries is often done manually, making it difficult to accurately grasp the type, quantity, and expiration date of ingredients. It is also common for ingredients to be overlooked due to expiration dates, resulting in waste. Furthermore, creating menus based on ingredient inventory information is time-consuming, creating a need for efficient food management.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0861] In this invention, the server includes a camera for recognizing ingredients and seasonings, a calculator for analyzing information on the recognized ingredients and seasonings, a storage device for saving the analysis results, and a display device for displaying the saved analysis results to the user. This allows the type, quantity, and expiration date of ingredients to be accurately determined, enabling efficient ingredient management.

[0862] The "photography device" is a device that captures images to recognize ingredients and seasonings.

[0863] The "computing device" is a device for analyzing information about recognized ingredients and seasonings.

[0864] A "storage device" is a device for saving analysis results.

[0865] A "display device" is a device for displaying the saved analysis results to a user.

[0866] "Image recognition technology" is a technology that detects and classifies specific objects and features from images.

[0867] A "barcode" is a code made up of a series of lines and spaces that contains information about ingredients and seasonings.

[0868] A "QR code" is a two-dimensional matrix code that contains information about ingredients and seasonings.

[0869] "Expiration date" is information indicating the date by which ingredients or seasonings can be safely consumed.

[0870] "Menu generation" is the process of suggesting dishes based on information about ingredients and seasonings.

[0871] MODE FOR CARRYING OUT THE INVENTION

[0872] This invention is a system that uses a camera installed in a refrigerator or pantry to recognize ingredients and seasonings, analyzes and stores that information, and displays it to the user. Specific embodiments of this system are described below.

[0873] server

[0874] The server receives image data sent from the camera installed in the refrigerator or pantry. A general server (e.g., cloud server or on-premise server) can be used as the hardware. HTTP requests are used to receive the image data.

[0875] The server preprocesses the received image data, including adjusting the image resolution and removing noise, using an image processing library such as OpenCV.

[0876] The server then uses image recognition technology to analyze the types, quantities, and expiration dates of ingredients and seasonings. Specifically, it uses machine learning libraries such as TensorFlow to classify ingredients in the image and uses object detection algorithms to estimate the amount of ingredients. It also uses the ZBar library to read barcodes and QR codes to obtain detailed information about ingredients.

[0877] The analysis results are converted to JSON format and stored in a database such as MySQL, allowing information such as the type, quantity, and expiration date of ingredients to be centrally managed.

[0878] Terminal

[0879] The device provides an application that allows users to check refrigerator and pantry inventory information. The hardware used can be a smartphone or tablet. The application runs on iOS or Android platforms.

[0880] The device retrieves inventory information from the server and displays it to the user. It parses the retrieved JSON data and displays the type, quantity, and expiration date of ingredients in a list format. This allows the user to easily check the inventory information of the refrigerator or pantry.

[0881] User

[0882] Users can check the inventory information of their refrigerator and pantry through the application. For example, a user can open the app and check the type, quantity, and expiration date of ingredients in the refrigerator.

[0883] Users can also create shopping lists based on inventory information. For example, users can select ingredients they are running low on and add them to the shopping list within the app. The list is saved on the device and can be edited as needed.

[0884] Examples and prompts

[0885] A specific example is obtaining information about ingredients in a refrigerator. A camera takes a picture of the ingredients in the refrigerator and sends the image data to a server. The server then uses an image recognition algorithm to analyze the type, quantity, and expiration date of the ingredients, and the analysis results are stored in a database. Users can then check this information through the app.

[0886] An example of a prompt sentence might be:

[0887] "I want to develop an application that retrieves information about ingredients in the refrigerator and displays inventory information. I would like to know how to build a system that sends images taken with a camera to a server and uses image recognition technology to analyze the type, quantity, and expiration date of ingredients."

[0888] In this way, by specifically explaining the system processing from the perspectives of the server, terminal, and user, the overall flow can be understood in detail.

[0889] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0890] Program processing flow

[0891] server

[0892] Step 1:

[0893] Receiving image data

[0894] The server receives image data transmitted from the image capturing device installed in the refrigerator or pantry.

[0895] Input: Image data sent from the imaging device

[0896] Output: Received image data

[0897] Specific operation: The camera periodically takes pictures of the inside of the refrigerator and sends them to the server via Wi-Fi. The server receives the HTTP request and saves the image data.

[0898] Step 2:

[0899] Image preprocessing

[0900] The server pre-processes the received image data.

[0901] Input: Received image data

[0902] Output: Preprocessed image data

[0903] Specific operation: The server adjusts the image resolution, removes noise, and adjusts the image contrast and brightness using OpenCV.

[0904] Step 3:

[0905] Barcode / QR code detection

[0906] The server detects barcodes and QR codes from the image.

[0907] Input: Preprocessed image data

[0908] Output: Information on detected barcodes and QR codes

[0909] What it does: The server uses the ZBar library to scan barcodes and QR codes in an image and extract the code's contents.

[0910] Step 4:

[0911] Database Matching

[0912] The server compares the extracted barcode or QR code information with a database.

[0913] Input: Information of the detected barcode or QR code

[0914] Output: Detailed information about ingredients retrieved from the database

[0915] Specific operation: The server sends a query to the MySQL database to obtain detailed information about the corresponding ingredients (type, quantity, expiration date, etc.).

[0916] Step 5:

[0917] Estimating the type and quantity of ingredients using image recognition

[0918] The server uses image recognition technology to estimate the type and quantity of ingredients.

[0919] Input: Preprocessed image data

[0920] Output: Estimated ingredients and quantities

[0921] How it works: The server uses a TensorFlow model to classify ingredients in the image and an object detection algorithm to estimate the amount of ingredients.

[0922] Step 6:

[0923] Saving analysis results

[0924] The server stores the analysis results in a database.

[0925] Input: Estimated ingredient type and quantity, detailed ingredient information retrieved from the database

[0926] Output: Analysis results stored in a database

[0927] Specific operation: The server converts the parsed results into JSON format and stores them in a MySQL database.

[0928] Terminal

[0929] Step 1:

[0930] Obtaining inventory information

[0931] The terminal obtains inventory information from the server.

[0932] Input: HTTP request to the server

[0933] Output: Stock information obtained from the server

[0934] What it does: When a user opens the app, the device sends an HTTP request to the server to get the latest inventory information.

[0935] Step 2:

[0936] View inventory information

[0937] The terminal displays the acquired inventory information to the user.

[0938] Input: Inventory information obtained from the server

[0939] Output: Inventory information displayed to the user

[0940] Specific operation: The device parses the acquired JSON data and displays the type, quantity, and expiration date of the ingredients in a list format.

[0941] User

[0942] Step 1:

[0943] Check stock information

[0944] Users can check refrigerator and pantry inventory information through the app.

[0945] Input: Stock information displayed on the terminal

[0946] Output: User confirmation action

[0947] Specific operation: When a user opens the app, inventory information is displayed and they can check the type, quantity, and expiration date of ingredients.

[0948] Step 2:

[0949] Creating a shopping list

[0950] The user creates a shopping list based on inventory information.

[0951] Input: Stock information displayed on the terminal

[0952] Output: The created shopping list

[0953] How it works: Users select ingredients they're running low on and add them to a shopping list within the app. The list is saved on the device and can be edited as needed.

[0954] (Application example 1)

[0955] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0956] In modern households, managing the ingredients and seasonings stored in the refrigerator and pantry is complicated, and it is common for ingredients to pass their expiration date or run out of necessary ingredients. It is also difficult to keep track of ingredient inventory and plan menus efficiently. This results in food waste and increases household food costs. Furthermore, there is a need to reduce food waste by prioritizing the use of ingredients that are close to their expiration date.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0958] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to the user, a means for managing ingredient inventory in real time, a means for notifying the user of ingredients approaching their expiration date, and a means for automatically ordering the necessary ingredients. This improves the efficiency of ingredient inventory management and enables priority use of ingredients approaching their expiration date. Furthermore, automatically ordering necessary ingredients can prevent ingredient shortages and reduce household food costs.

[0959] "Camera for recognizing ingredients and seasonings" refers to a camera that is installed in a refrigerator or pantry to recognize the type, quantity, and expiration date of ingredients and seasonings.

[0960] "Generative AI that generates menus based on recognized ingredients and seasonings" is artificial intelligence that automatically generates appropriate menus based on information about ingredients and seasonings recognized by the camera.

[0961] "Means for proposing the generated menu to the user" refers to means for notifying and proposing the menu generated by the generative AI to the user.

[0962] "Means for managing food inventory in real time" refers to a means for grasping and managing the current inventory status in real time based on the type and quantity of food ingredients recognized by the camera.

[0963] The "means for notifying the user of ingredients whose expiration date is approaching" is a means for monitoring the expiration date of ingredients and notifying the user of ingredients whose expiration date is approaching.

[0964] "Means for automatically ordering necessary ingredients" refers to a means for automatically ordering necessary ingredients from a food delivery service, etc., based on stock availability and expiration dates.

[0965] A system for implementing this invention includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to users, a means for managing ingredient inventory in real time, a means for notifying users of ingredients that are close to their expiration date, and a means for automatically ordering the necessary ingredients.

[0966] Hardware and Software Configuration

[0967] Hardware

[0968] Camera: A camera installed in the refrigerator or pantry to recognize the type, quantity, and expiration date of ingredients and seasonings.

[0969] Smartphone: A device that allows users to operate applications.

[0970] Server: A central processing unit for generative AI and data management.

[0971] software

[0972] OpenCV: Image processing library. Used to analyze images acquired from the camera and recognize the types and quantities of ingredients and seasonings.

[0973] Pyzbar: A barcode / QR code reading library. Used to read barcodes and QR codes on ingredients and seasonings.

[0974] Requests: An HTTP request library that we use to order the ingredients we need from a food delivery service.

[0975] Processing flow

[0976] Camera-based food recognition

[0977] The camera is installed in the refrigerator or pantry and periodically captures images. The captured images are analyzed using OpenCV to identify the type, quantity, and expiration date of ingredients and seasonings. If a barcode or QR code is included, the information is read using Pyzbar.

[0978] Menu generation using generative AI

[0979] The information on the recognized ingredients and seasonings is sent to a server, and a generative AI generates a menu based on this information, taking into account parameters such as the number of family members, past menu history, and the season.

[0980] Menu suggestions and inventory management

[0981] The generated menu is presented to the user via a smartphone application, and the availability of ingredients is managed in real time, with notifications sent to users when any ingredients are nearing their expiration date.

[0982] Automated Orders

[0983] If a required ingredient is in short supply, the server uses the Requests library to automatically place an order with a food delivery service, preventing shortages and reducing household food costs.

[0984] Specific examples

[0985] For example, if there are five tomatoes in the refrigerator and their expiration date is approaching in three days, the camera will recognize this and the generative AI will generate a menu using tomatoes. The generated menu will be suggested to the user through a smartphone application. The user will also be notified that the expiration date is approaching and can order more tomatoes if necessary.

[0986] Prompt Sentence Examples

[0987] "Please create an application that recognizes the type and amount of food in the refrigerator and notifies you when the food is nearing its expiration date. Also, please add a function that automatically orders the necessary ingredients from a food delivery service."

[0988] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0989] Step 1:

[0990] Cameras are installed in refrigerators and pantries to periodically capture images.

[0991] Input: Images of ingredients and condiments in your refrigerator or pantry.

[0992] Output: The captured image data.

[0993] Specific operation: The camera takes images at regular intervals and sends the image data to the server.

[0994] Step 2:

[0995] The server uses OpenCV to analyze the captured images and recognize the type, quantity, and expiration date of ingredients and seasonings.

[0996] Input: The captured image data.

[0997] Output: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[0998] Specific operation: The server uses the image processing library OpenCV to identify ingredients and seasonings in the image, and if a barcode or QR code is included, it uses Pyzbar to read that information.

[0999] Step 3:

[1000] The server uses generative AI to generate a menu based on the information on ingredients and seasonings recognized by the server.

[1001] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[1002] Output: The generated menu.

[1003] Specific operation: The server inputs information about ingredients and seasonings into the generative AI, which then generates a menu taking into account parameters such as the number of family members, past menu history, and the season.

[1004] Step 4:

[1005] The server proposes the generated menu to the user through a smartphone application.

[1006] Input: The generated menu.

[1007] Output: The menu suggested to the user.

[1008] Specific operation: The server sends the generated menu to the smartphone application, and the application notifies the user.

[1009] Step 5:

[1010] The server manages food inventory in real time and notifies customers of food items approaching their expiration date.

[1011] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[1012] Output: Real-time inventory status, notifications of ingredients approaching their expiration date.

[1013] Specific operation: The server stores information about ingredients and seasonings in a database, and if any ingredients are nearing their expiration date, it notifies the user via a smartphone application.

[1014] Step 6:

[1015] The server automatically orders the necessary ingredients from a food delivery service.

[1016] Input: Availability and expiration date information.

[1017] Output: An order from a food delivery service.

[1018] Specific operation: The server uses the Requests library to automatically order ingredients that are out of stock from a food delivery service.

[1019] Example 2

[1020] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1021] Conventional menu generation systems make it difficult for users to efficiently use ingredients and seasonings they have on hand, and also make it difficult to generate optimal menus that take into account parameters such as the number of family members, past menu history, and the season. Furthermore, they are unable to generate menus that take into account the expiration dates of ingredients, which leads to the problem of food waste.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1023] In this invention, the server includes means for receiving information on ingredients and seasonings entered by the user and saving it in a database, means for sending prompts to the generative AI based on the saved information, and means for returning the generated menu to the user's terminal. This allows the user to efficiently use the ingredients and seasonings on hand and generate an optimal menu taking into account parameters such as the number of family members, past menu history, and the season. Furthermore, since the menu can be generated taking into account the expiration dates of ingredients, food waste can be reduced.

[1024] The "photography device" is a device for recognizing ingredients and seasonings.

[1025] "Generative AI" is AI that generates menus based on recognized ingredients and seasonings.

[1026] The "means for suggesting to the user" is a means for presenting the generated menu to the user.

[1027] The "means for saving in a database" is a means for receiving information on ingredients and seasonings entered by the user and saving it in a database.

[1028] The "means for sending a prompt" is a means for sending a prompt to the generative artificial intelligence based on the stored information.

[1029] The "means for returning" is a means for returning the generated menu to the user's terminal.

[1030] "Number of family members" is information about the number of family members to be taken into consideration when creating a menu.

[1031] "Past menu history" is history information of menus that have been generated in the past.

[1032] "Season" is information about the current season that is taken into consideration when generating a menu.

[1033] "Expiration date" is information indicating the expiration date of a food ingredient.

[1034] The present invention is a system that recognizes ingredients and seasonings and generates an optimal menu based on the ingredients and seasonings. A specific embodiment of this system will be described below.

[1035] Hardware and software used

[1036] Hardware: Server, terminal (PC, smartphone, tablet, etc.), imaging device

[1037] Software: Generative AI models (e.g., GPT-4), database management systems (e.g., MySQL), and user interfaces (e.g., web applications)

[1038] System configuration

[1039] 1. Imaging equipment

[1040] This is a device for recognizing ingredients and seasonings. The user takes a photo of the ingredients or seasonings at hand and sends that information to the system.

[1041] 2. Server

[1042] The server receives the information about ingredients and seasonings entered by the user and stores it in a database.

[1043] The server sends prompts to the generative AI model based on the stored information.

[1044] The server returns the generated menu to the user's terminal.

[1045] 3. Generative AI Models

[1046] The generative AI model generates the optimal menu based on prompts sent from the server.

[1047] The generative AI model takes into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[1048] 4. Database

[1049] The database stores information entered by the user, such as ingredients and seasonings, the number of family members, past menu history, and the season.

[1050] 5. User Interface

[1051] The user interface is a web application that allows users to input information about ingredients and seasonings.

[1052] The user interface presents the generated menu to the user.

[1053] Specific examples

[1054] Prompt Sentence Examples

[1055] "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu items include curry, stew, and stir-fry. The current season is autumn."

[1056] System Operation

[1057] 1. The user inputs ingredients and seasonings

[1058] The user uses the terminal to input the types and amounts of ingredients and seasonings currently on hand.

[1059] Example: "200g chicken, 1 carrot, 1 onion, 50ml soy sauce"

[1060] 2. The server receives the input data and stores it in the database

[1061] The server receives the information about ingredients and seasonings sent by the user and stores it in a database.

[1062] Example: Save as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce" in a MySQL database.

[1063] 3. The server sends a prompt to the generative AI model

[1064] The server generates and sends prompts to the generative AI model, including parameters entered by the user, such as information on ingredients and seasonings, the number of family members, past menu history, and the season.

[1065] Example: "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn."

[1066] 4. Generative AI model generates optimal menu

[1067] The generative AI model generates the optimal menu based on the prompts received, taking into account parameters such as the type and quantity of ingredients, expiration dates, number of family members, past menu history, and the season.

[1068] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[1069] 5. The server returns the generated menu to the user

[1070] The server returns the menu received from the generative AI model to the user's device.

[1071] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[1072] 6. User checks the menu and starts cooking

[1073] The user checks the menu generated on the terminal and starts cooking.

[1074] Example: "Check out the recipe for boiled chicken and vegetables and start cooking."

[1075] In this way, the system uses a generative AI model to generate the optimal menu based on the ingredients and seasoning information entered by the user and provides it to the user.

[1076] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The user inputs ingredients and seasonings

[1079] Input: The user uses a device (PC, smartphone, tablet, etc.) to input the types and quantities of ingredients and seasonings currently on hand.

[1080] Specific operation: The user enters information such as "200g of chicken, 1 carrot, 1 onion, 50ml of soy sauce" into an input form in a web application and clicks the submit button.

[1081] Output: The information about the ingredients and seasonings entered is sent to the server.

[1082] Step 2:

[1083] The server receives the input data and stores it in the database

[1084] Input: Ingredient and seasoning information sent by the user

[1085] Specific operation: The server receives the HTTP request, analyzes its contents, and stores it in a database (e.g., MySQL) as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce."

[1086] Output: Information on ingredients and seasonings stored in the database

[1087] Step 3:

[1088] The server sends prompts to the generative AI model

[1089] Input: Parameters such as ingredients and seasoning information stored in the database, number of family members, past menu history, and seasons

[1090] Specific operation: The server retrieves the necessary information from the database and sends the prompt statement, "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn," to the generative AI model (e.g., GPT-4).

[1091] Output: The prompt sent to the generative AI model

[1092] Step 4:

[1093] Generative AI models generate optimal menus

[1094] Input: Prompt sent from the server

[1095] How it works: The generative AI model analyzes the prompt and generates a menu such as "Chicken and vegetable stew, and a salad made with seasonal autumn ingredients," taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[1096] Output: The generated menu

[1097] Step 5:

[1098] The server returns the generated menu to the user

[1099] Input: Menu received from the generative AI model

[1100] Specific operation: The server sends the generated menu to the user's device as an HTTP response and displays it on the web application.

[1101] Output: Menu displayed on the user's device

[1102] Step 6:

[1103] The user checks the menu and starts cooking

[1104] Input: The menu displayed on the user's device

[1105] Specific behavior: The user checks the recipe displayed on the web application for "Chicken and vegetable stew, salad using seasonal autumn ingredients" and starts cooking by following the necessary steps.

[1106] Output: Specific instructions and ingredients for the user to start cooking

[1107] (Application example 2)

[1108] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1109] Conventional systems that generate menus by recognizing ingredients and seasonings require users to manually purchase additional ingredients and seasonings required for the generated menu, which is inconvenient. Furthermore, there is a lack of a way to maximize the use of ingredients and seasonings that users already have and reduce waste. Furthermore, there is a need for a way to quickly obtain additional ingredients and seasonings based on the generated menu.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1111] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, and a means for ordering additional ingredients and seasonings required for the generated menu. This allows the user to make the most of the ingredients and seasonings they have and reduce waste. Furthermore, additional ingredients and seasonings can be quickly obtained based on the generated menu, improving user convenience.

[1112] A "camera for recognizing ingredients and seasonings" is a device that captures images of ingredients and seasonings owned by the user and analyzes them as digital data.

[1113] "Generative AI" is an artificial intelligence system that automatically generates the optimal menu for the user based on recognized information about ingredients and seasonings.

[1114] "Means for proposing the generated menu to the user" refers to an interface or function for notifying and displaying the menu generated by the generative AI to the user.

[1115] "Means for ordering additional ingredients and condiments required for the generated menu" refers to a function that automatically identifies missing ingredients and condiments based on the generated menu and orders them through a food delivery service.

[1116] "Family size" is the number of people in the user's household that will be taken into account when generating the menu.

[1117] "Past menu history" is a record of menus previously created or consumed by a user, data that generative AI uses to learn about a user's preferences and tendencies.

[1118] "Season" refers to information about the current time or season that is taken into consideration when generating a menu, and is a parameter for giving priority to using ingredients that are in season.

[1119] "Expiration date" is information indicating the date by which ingredients and seasonings can be safely consumed, and is used to determine which ingredients to prioritize when creating a menu.

[1120] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1121] System configuration

[1122] The system consists of the following major components:

[1123] 1. Camera for recognizing ingredients and seasonings

[1124] 2. Generative AI

[1125] 3. A means of suggesting the generated menu to the user

[1126] 4. A way to order any additional ingredients or condiments needed for the generated menu.

[1127] Program processing

[1128] Camera for recognizing ingredients and seasonings

[1129] A camera is used to recognize ingredients and seasonings that the user owns. The camera takes a picture of the ingredients in the refrigerator or pantry, and the image is analyzed as digital data. Specifically, image processing libraries such as OpenCV are used to recognize ingredients and seasonings.

[1130] Generative AI

[1131] Based on the information on recognized ingredients and seasonings, a generative AI generates the optimal menu. The generative AI generates the menu taking into account parameters such as the number of family members, past menu history, and the season. For example, an AI model built using TensorFlow or PyTorch can play this role.

[1132] A means of suggesting the generated menu to the user

[1133] The generated menu is presented to the user through the smartphone application, and the user can view the generated menu through the application interface.

[1134] A means to order any additional ingredients or condiments required for the generated menu

[1135] Based on the generated menu, missing ingredients and condiments can be automatically identified and ordered through a food delivery service, for example, using a specific API to order ingredients online.

[1136] Specific examples

[1137] The ingredients in the user's refrigerator are tomatoes, cheese, and basil. There are four people in the family, and past menu history includes pasta and salad. The current season is summer. Based on this information, generative AI is used to suggest the optimal menu. The generated menu is a Margherita pizza, which requires additional pizza dough and olive oil. These additional ingredients are ordered through a food delivery service.

[1138] Prompt Sentence Examples

[1139] The ingredients in the user's refrigerator are tomatoes, cheese, and basil. There are four people in the family, and their past menu history includes pasta and salad. The current season is summer. Based on this information, please use generative AI to suggest the best menu.

[1140] In this way, users can maximize the use of ingredients and seasonings they have and reduce waste, and can quickly obtain additional ingredients and seasonings based on the generated menu, improving user convenience.

[1141] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1142] Step 1:

[1143] The user uses the smartphone camera to take a picture of the ingredients and seasonings in the refrigerator or pantry.

[1144] Input: Images of ingredients and seasonings

[1145] Output: Image data

[1146] Specific operation: The user launches the camera app on their smartphone and takes a picture of ingredients and seasonings in the refrigerator or pantry. The image is then saved on the smartphone.

[1147] Step 2:

[1148] The server uses an image processing library (e.g., OpenCV) to recognize ingredients and seasonings from the captured image.

[1149] Input: Image data

[1150] Output: A list of recognized ingredients and seasonings

[1151] Specific operation: The server receives the image data and performs image analysis using OpenCV. It extracts the characteristics of ingredients and seasonings and outputs the recognition results in list format.

[1152] Step 3:

[1153] The server uses a generative AI model (e.g., TensorFlow or PyTorch) to generate the optimal menu, taking into account parameters such as recognized ingredients and seasonings, number of family members, past menu history, and seasonings.

[1154] Input: List of recognized ingredients and seasonings, number of family members, past menu history, seasonal information

[1155] Output: The generated menu

[1156] Specific operation: The server receives inputs such as a list of recognized ingredients and seasonings, the number of family members, past menu history, and seasonal information, and uses a generative AI model to generate an optimal menu. The generated menu is then output in list format.

[1157] Step 4:

[1158] The server proposes the generated menu to the user through a smartphone application.

[1159] Input: The generated menu

[1160] Output: Menu suggested to the user

[1161] Specific operation: The server sends the generated menu to the smartphone application and notifies the user, who then checks the menu through the application interface.

[1162] Step 5:

[1163] Based on the generated menu, the server automatically identifies missing ingredients and condiments and orders them through a food delivery service.

[1164] Input: The generated menu

[1165] Output: Additional ingredients and condiments ordered

[1166] What it does: The server analyzes the generated menu and identifies any missing ingredients or condiments, which are then automatically ordered using the food delivery service's API.

[1167] In this way, users can maximize the use of ingredients and seasonings they have and reduce waste, and can quickly obtain additional ingredients and seasonings based on the generated menu, improving user convenience.

[1168] Example 3

[1169] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1170] Conventional menu suggestion systems were unable to fully consider the user's individual preferences, allergy information, or schedule information, making it difficult to suggest the optimal menu for the user. Furthermore, if the suggested menu did not suit the user's lifestyle, there was the problem of having to resubmit the request, which was time-consuming. Furthermore, since the system was unable to suggest menus that took into account the expiration dates of ingredients, food waste could occur.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1172] In this invention, the server includes: a means for recognizing ingredients and seasonings photographed by a camera; The system includes a means for collecting menu candidates previously selected by a user, the user's preferences, allergy information, and schedule information, a means for generating menu candidates using a generative AI model and a prompt that instructs the system to generate a menu that can be cooked from the recognized ingredients and seasonings based on the recognized ingredients and seasonings, the previously selected menu candidates, the user's preferences, allergy information, and schedule information, and a means for notifying the user's terminal of the generated menu candidates, thereby enabling the system to propose optimal menus that match the user's individual preferences and lifestyle. In addition, the collecting means further collects the number of family members, past menu history, or season, and the generating means generates menu candidates using a prompt sentence instructing the means to generate a menu that can be cooked from the recognized ingredients and seasonings based on the collected number of family members, past menu history, or season, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model. The server may also include a means for recognizing the expiration dates of ingredients, and generate a menu by giving priority to ingredients with the earliest expiration dates. This allows for menu suggestions that take the expiration dates of ingredients into consideration, thereby reducing food waste.

[1173] A "camera for recognizing ingredients and seasonings" is a device that uses image analysis technology to recognize the type, shape, color, expiration date, etc. of ingredients and seasonings.

[1174] A "generative AI model" is an artificial intelligence system that generates the optimal menu for a user based on collected data.

[1175] "Means for suggesting to the user" refers to a method or device for notifying the user of the generated menu, and includes notifications on smartphones or tablets, guidance via voice assistants, etc.

[1176] "User's past selection data" is information about menu items and ingredients selected by the user in the past.

[1177] "Preferences" is information about ingredients and types of dishes that the user particularly likes.

[1178] "Allergy information" is information about ingredients to which the user has an allergy.

[1179] "Schedule information" is information about the user's schedule and time, and is obtained from a calendar application or the like.

[1180] A "generative AI model" is an artificial intelligence model that generates prompts and suggests optimal menus based on collected data.

[1181] A "terminal" is an electronic device used by a user, such as a smartphone or tablet.

[1182] The "notification means" refers to a method or device for notifying the user of the generated menu, and includes push notifications and voice guidance.

[1183] This invention is a system that suggests optimal menus to users, and includes a camera for recognizing ingredients and seasonings, generative AI, means for making suggestions to users, means for collecting the user's past selection data, preferences, allergy information, and schedule information, means for generating optimal menus using a generative AI model, and means for notifying the user of the generated menu on their terminal.

[1184] Hardware and software used

[1185] Hardware: Smartphones, tablets, cameras

[1186] Software: Generative AI models, voice assistants

[1187] Data processing and calculation

[1188] server

[1189] The server recognizes the ingredients and seasonings the user has, as captured by the camera. The server also collects the user's past menu selection data, preferences, allergy information, and schedule information. The server also collects the number of family members, past menu history, and current season. This information is entered by the user through a smartphone app or obtained from a calendar app, etc. The collected information is stored in a database and input into the generative AI model.

[1190] The server uses a generative AI model to generate menu options that can be cooked from the recognized ingredients and seasonings based on the collected data.

[1191] Example prompt sentence: "Ingredients owned by the user: pork, lettuce Condiments owned by the user: salt, pepper User's past selection data: curry, stew, ginger pork User preference: Light User allergy information: Egg allergy Schedule information: Regular work today, off tomorrow Number of people in the family: 3 Past menu history: Spaghetti, rice ball set meal, ramen Season: Summer Considering this information, please suggest a menu for tonight's dinner."

[1192] The generated menu candidates are transmitted from the server to the user's terminal.

[1193] Terminal

[1194] The device (smartphone or tablet) receives multiple menu options sent from the server. The received menu is displayed to the user as a notification. For example, when you wake up in the morning, a notification such as "Today's dinner is chicken curry or shrimp fried rice" is displayed on your smartphone. If the device has a voice assistant function, it can also guide you by voice, saying "Today's dinner is chicken curry or shrimp fried rice."

[1195] User

[1196] The user checks the menu options displayed on their smartphone or tablet. If the user finds a menu option they like, they select it. They can then prepare the selected menu option for dinner. If the user does not like any of the suggested menu options, they can send a request to the server again and receive new menu options.

[1197] Specific examples

[1198] 1. At 7:00 a.m., a notification appears on the user's smartphone saying, "Today's dinner is chicken curry or shrimp fried rice."

[1199] 2. When the user returns home, the smart speaker will say, "Welcome home. Tonight's dinner is chicken curry or shrimp fried rice."

[1200] 3. If the user does not like either the chicken curry or the shrimp fried rice, they can send a request from their smartphone saying, "Please suggest a new menu item," and the server will generate new menu options and notify them again.

[1201] In this way, the system collects user information, uses the generative AI model to propose an optimal menu, and notifies the user. The user can check the proposed menu and resubmit the request if necessary. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1202] Step 1: Collect user information

[1203] Subject: Server

[1204] The server recognizes the ingredients and seasonings the user owns from images taken with the user's smartphone camera. The server also collects the user's past menu selections, preferences, allergy information, and schedule information. The server also collects the number of family members, past menu history, and current season. This information is entered by the user through a smartphone app or obtained from a calendar app, etc.

[1205] Input: Information entered by the user into a smartphone app, schedule information obtained from a calendar app

[1206] Data processing: Store the entered information in a database and convert it into the required format.

[1207] Output: User information stored in the database

[1208] Specific behavior:

[1209] The user takes a photo of ingredients and seasonings that the user has with the camera of their smartphone. Users enter their preferences and allergy information into a smartphone app.

[1210] The server retrieves schedule information from the calendar app. The server obtains the number of people in the user's family, past menu history, and the current season.

[1211] The server stores this information in a database.

[1212] Step 2: Submit a request to create a menu

[1213] Subject: Terminal

[1214] When a user requests a menu, the device (smartphone or tablet) sends a request to the server, which includes information such as the user's ID and the current date and time.

[1215] Input: User ID, current date and time

[1216] Data processing: Converting request data into a format to send to the server

[1217] Output: Request data sent to the server

[1218] Specific behavior:

[1219] The user presses a button on a smartphone app that says, "Suggest today's dinner menu."

[1220] The device sends a request to the server containing the user's ID and the current date and time.

[1221] Step 3: Create a menu

[1222] Subject: Server

[1223] The server uses a generative AI model to generate an optimal menu for the user. Specifically, the server generates a prompt sentence instructing the user to generate a menu that can be cooked from the recognized ingredients and seasonings based on the collected information on the number of family members, past menu history, or season, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and then uses the prompt sentence and the generative AI model to suggest an optimal menu.

[1224] Input: User information stored in the database, prompt text

[1225] Data processing: Input a prompt sentence into the generative AI model to generate multiple menu options

[1226] Output: Multiple generated menu options

[1227] Specific behavior:

[1228] The server inputs a prompt sentence into the generative AI model.

[1229] The generative AI model returns suggested menu options to the server.

[1230] Step 4: Menu Notifications

[1231] Subject: Terminal

[1232] The terminal receives the multiple menu options sent from the server and notifies the user of them in a manner that varies depending on the user's settings.

[1233] Input: Menu suggestions sent from the server

[1234] Data processing: Convert the received menu into a notification format

[1235] Output: Menu suggestions notified to the user

[1236] Specific behavior:

[1237] The terminal receives the menu candidates sent from the server.

[1238] Depending on the user's settings, the device will display "Chicken curry or shrimp fried rice for dinner tonight" in the notification bar.

[1239] If the device has a voice assistant function, it will say aloud, "Today's dinner is chicken curry, shrimp fried rice, or shrimp fried rice."

[1240] Step 5: Menu review and feedback

[1241] Subject: User

[1242] The user checks the notified menu candidates, and if they like it, selects it and prepares it as is. If they do not like it, they can send the request again and receive new menu candidates.

[1243] Input: Notified menu, user feedback

[1244] Data transformation: Generate new requests based on user feedback

[1245] Output: New menu request

[1246] Specific behavior:

[1247] The user checks the notification on their smartphone and sees that it says, "Today's dinner is chicken curry or shrimp fried rice."

[1248] If the user doesn't like the menu, they can press the "Suggest a new menu" button within the app.

[1249] The terminal again sends a request to the server and receives a new menu.

[1250] Note that an emotion engine, which will be described later, may be used to recognize the user's emotion and generate a cookable menu by further taking the user's emotion into consideration. Specifically, the server further includes means for recognizing the user's emotion using an emotion engine, and the generating means generates menu candidates using a prompt sentence instructing the generation of a cookable menu from the recognized ingredients and seasonings based on the recognized emotion of the user, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model.

[1251] (Application example 3)

[1252] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1253] Conventional menu suggestion systems have difficulty proposing optimal menus for users because they do not fully consider user preferences or past order history. Furthermore, there are limited ways to effectively notify users of suggested menus, preventing flexible suggestions tailored to the user's lifestyle. Furthermore, they are unable to suggest menus that take into account the expiration dates of ingredients, which can result in food waste.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1255] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, a means for generating a menu taking into account the user's preferences and past order history, and a means for notifying or announcing the generated menu by voice. This allows for optimal menu suggestions that take into account the user's preferences and past order history, allowing for flexible suggestions tailored to the user's lifestyle. Furthermore, menu suggestions that take into account the expiration dates of ingredients can reduce food waste.

[1256] A "camera for recognizing ingredients and seasonings" is a photographic device used to identify the type, shape, color, etc. of ingredients and seasonings.

[1257] "Generative AI" is an artificial intelligence system that automatically generates appropriate menus based on recognized information about ingredients and seasonings.

[1258] The "means for proposing the generated menu to the user" refers to a method or device for notifying or displaying the generated menu to the user.

[1259] "Means for generating a menu taking into consideration the user's preferences and past order history" refers to a method or device for generating an optimal menu based on the user's past order history and preference data.

[1260] The "means for notifying or providing voice guidance of the generated menu" refers to a method or device for notifying or providing voice guidance of the generated menu to the user.

[1261] "Parameters such as the number of family members, past menu history, and season" are elements such as the number of family members, past menu history, and season that are taken into consideration when generating a menu.

[1262] "Means for recognizing the expiration date of ingredients" refers to a method or device for identifying the expiration date of ingredients and creating a menu based on that information.

[1263] The "means for generating a menu by giving priority to ingredients with the earliest expiration date" refers to a method or device for generating a menu by giving priority to ingredients with the nearest expiration date.

[1264] A system for implementing this invention includes a camera for recognizing ingredients and seasonings, a generation AI, a means for suggesting the generated menu to the user, a means for generating a menu taking into account the user's preferences and past ordering history, and a means for notifying or providing audio guidance about the generated menu.

[1265] The server first uses a camera to recognize ingredients and seasonings. The camera is used to identify the type, shape, color, etc. of the ingredients and seasonings. The recognized information is sent to the generative AI.

[1266] Generative AI automatically generates an appropriate menu based on the recognized information on ingredients and seasonings, taking into account parameters such as the user's preferences, past order history, number of family members, and season. This allows the system to generate the optimal menu for the user.

[1267] The generated menu is then proposed to the user. This can be done by sending a notification to a device such as a smartphone or tablet, or by voice guidance. For example, when you wake up in the morning, you can receive a notification on your smartphone suggesting that day's dinner menu. Or, when you get home, you can receive a voice prompt with that day's dinner menu, making it possible to make proposals tailored to the user's lifestyle.

[1268] The camera also recognizes the expiration date of ingredients, allowing the system to prioritize ingredients with the earliest expiration date when generating menus, thereby reducing food waste.

[1269] For example, if a user likes "sushi, ramen, tempura" and their past order history is "sushi, ramen," the generative AI is likely to suggest "tempura." This suggestion is sent to the user's smartphone at 7:00 AM.

[1270] An example of a prompt to input to a generative AI model is as follows:

[1271] The user's preferences are "Sushi, Ramen, Tempura" and their past order history is "Sushi, Ramen." Please suggest the best dinner menu for this user.

[1272] This system makes it possible to propose optimal menus that take into account the user's preferences and past order history, allowing for flexible proposals that fit the user's lifestyle. It also reduces food waste by proposing menus that take into account the expiration dates of ingredients.

[1273] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1274] Step 1:

[1275] The server uses a camera to recognize ingredients and seasonings. The camera is used to identify the type, shape, color, etc. of ingredients and seasonings. The input is image data captured by the camera, and the output is information about the recognized ingredients and seasonings. This information is then processed using an image processing algorithm.

[1276] Step 2:

[1277] The server sends the information about the recognized ingredients and seasonings to the generative AI. The input is the information about the ingredients and seasonings obtained from the camera, and the output is the data passed to the generative AI. The data is sent via an API.

[1278] Step 3:

[1279] Generative AI automatically generates appropriate menus based on recognized information about ingredients and seasonings. Inputs include information about ingredients and seasonings, parameters such as the user's preferences, past order history, number of family members, and seasonings. The output is a generated menu. The generative AI model analyzes this data based on prompt statements to generate the optimal menu.

[1280] Step 4:

[1281] The server proposes the generated menu to the user. The input is the menu obtained from the generative AI, and the output is the menu that is notified or voice-guided to the user. The notification is sent to a smartphone or tablet, and the voice guidance is provided using speech synthesis software.

[1282] Step 5:

[1283] The server uses a camera to recognize the expiration date of ingredients. The input is image data captured by the camera, and the output is the recognized expiration date. The expiration date data is extracted using an image processing algorithm.

[1284] Step 6:

[1285] The generative AI generates menus by prioritizing ingredients with the earliest expiration date. The inputs include expiration date information, information on ingredients and seasonings, user preferences, past order history, number of family members, seasonings, and other parameters. The output is an optimal menu that takes expiration dates into consideration. The generative AI model analyzes this data based on prompt statements and generates a menu that takes expiration dates into consideration.

[1286] Step 7:

[1287] The user checks the proposed menu and places an order if necessary. The input is a menu announced or voice-guided by the server, and the output is the user's order information. The user places the order using a smartphone or tablet.

[1288] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1289] "Example 1"

[1290] One embodiment of the present invention is a system that incorporates an emotion engine. This system recognizes a user's emotions and generates a menu based on those emotions. Specifically, if the user expresses joy or excitement, the emotion engine assumes it is a special occasion or celebration and generates a luxurious menu. Conversely, if the user expresses sadness or depression, the emotion engine assumes they are not feeling well and generates a menu that is easy to digest.

[1291] "Example 2"

[1292] The emotion engine can also change the way it suggests menu items based on the user's emotions. For example, if the user expresses joy or excitement, the emotion engine will suggest menu items by voice, sharing that joy. Conversely, if the user expresses sadness or depression, the emotion engine will silently send a notification to the user's smartphone, tablet, or other device. This makes it possible to provide services that take the user's emotions into consideration.

[1293] "Example 3"

[1294] Furthermore, the emotion engine can learn changes in a user's emotions and optimize menu generation and suggestion methods accordingly. For example, if a user tends to always select a particular ingredient after expressing happiness, the emotion engine can learn that tendency and prioritize generating a menu that includes that ingredient the next time the user expresses happiness. This allows the system to better understand the user's emotions and preferences and provide a more personalized service.

[1295] The processing flow of each embodiment will be described below.

[1296] "Example 1"

[1297] Step 1: The emotion engine recognizes the user's emotion.

[1298] Step 2: The emotion engine generates a menu based on the user's emotions. Specifically, if the user expresses joy or excitement, the emotion engine assumes it's a special occasion or celebration and generates a luxurious menu. Conversely, if the user expresses sadness or depression, the emotion engine assumes they're not feeling well and generates a menu that's easy to digest.

[1299] "Example 2"

[1300] Step 1: The emotion engine recognizes the user's emotion.

[1301] Step 2: The emotion engine changes the menu suggestion method based on the user's emotions. For example, if the user expresses happiness or excitement, the emotion engine will suggest a menu by voice, sharing that joy. Conversely, if the user expresses sadness or depression, the emotion engine will silently send a notification to the device, such as a smartphone or tablet.

[1302] "Example 3"

[1303] Step 1: The emotion engine recognizes the user's emotion.

[1304] Step 2: The emotion engine learns the user's emotional changes.

[1305] Step 3: Optimize menu generation and suggestion methods based on the results learned by the emotion engine.

[1306] For example, if a user tends to always select a particular ingredient after expressing a happy emotion, the emotion engine will learn that tendency and prioritize generating a menu that includes that ingredient the next time the user expresses a happy emotion.

[1307] Example 1

[1308] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1309] Conventional food management systems and menu generation systems are limited to recognizing ingredients and suggesting menus, and have the problem of not being able to suggest menus that take the user's emotions into account. Furthermore, menu suggestions that take into account the expiration date of ingredients are insufficient, which can result in food waste. Furthermore, menu generation has not taken into account parameters such as the number of family members, past menu history, or season. To solve these issues, a system that recognizes the user's emotions and adjusts the menu based on them is needed.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1311] In this invention, the server includes a photographing device for recognizing ingredients and seasonings, a generative artificial intelligence for generating a menu based on the recognized ingredients and seasonings, a means for proposing the generated menu to a user, an emotion analysis device for recognizing the user's emotion, and a means for adjusting the menu based on the user's emotion. This makes it possible to propose a menu that takes the user's emotion into consideration, reducing ingredient waste and realizing more personalized menu proposals.

[1312] A "photography device" is a device that captures images or videos to recognize ingredients and seasonings.

[1313] "Generative AI" is an AI technology that generates menus based on recognized ingredients and seasonings.

[1314] The "means for suggesting" is a means for notifying or displaying the generated menu to the user.

[1315] An "emotion analysis device" is a device that analyzes data such as facial expressions and voice to recognize a user's emotions.

[1316] The "adjusting means" is a means for appropriately changing or adjusting the generated menu based on the user's feelings.

[1317] "Expiration date" is information indicating the date by which ingredients or seasonings can be safely consumed.

[1318] "Parameters" are factors that affect menu generation, such as the number of family members, past menu history, and season.

[1319] The present invention relates to a system that recognizes ingredients and seasonings and generates a menu based on a user's emotions. The system includes a photographing device, generative artificial intelligence, an emotion analysis device, a suggestion unit, and an adjustment unit.

[1320] System configuration

[1321] 1. Imaging equipment

[1322] The server receives image data transmitted from the image capturing device installed in the refrigerator or pantry.

[1323] The camera captures information about ingredients by reading barcodes and QR codes, and uses image recognition technology to estimate the type and quantity of ingredients.

[1324] 2. Generative Artificial Intelligence

[1325] The server generates a menu using generative artificial intelligence based on the information about ingredients acquired from the camera.

[1326] Generative AI generates menus taking into account parameters such as the number of family members, past menu history, and the season.

[1327] 3. Emotion analysis device

[1328] The device captures the user's facial expressions and voice through a camera and microphone.

[1329] The server analyzes the user's emotions using an emotion analysis device, for example, using facial expression recognition technology or voice analysis technology.

[1330] 4. Proposed measures

[1331] The server provides a means for notifying or displaying the generated menu to the user, for example, by displaying the menu through a smartphone app or display.

[1332] 5. Adjustment measures

[1333] The server provides a means to appropriately modify or adjust the generated menu based on the user's sentiment.

[1334] If the user shows emotions of joy or excitement, the emotion analysis device generates a luxurious menu for a special occasion or celebration. Conversely, if the user shows emotions of sadness or depression, the device generates a menu that is easy to digest and assumes that the user is not feeling well.

[1335] Specific examples

[1336] Example 1: A camera installed in a refrigerator reads the barcode and acquires information about "milk" and "eggs." The server stores this information in a database.

[1337] How it works: The camera scans the barcodes of the "Milk" and "Eggs" items in the refrigerator and sends the information to the server, which stores it in a database and updates the inventory.

[1338] Example 2: A user smiles into the device's camera. The server uses an emotion analyzer to recognize the emotion "joy" and generates a luxurious dinner menu.

[1339] How it works: The device's camera captures the user's smile and sends the data to the server, which uses an emotion analyzer to analyze the emotion of "happiness" and suggest a luxurious dinner menu.

[1340] Prompt Sentence Examples

[1341] Prompt 1: "Use a camera to identify ingredients in the refrigerator and suggest menu items based on the user's emotions."

[1342] Prompt 2: "If the user is sad, generate a digestible menu."

[1343] This system supports daily meal preparation by automating ingredient management and menu suggestions based on the user's emotions.

[1344] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1345] Step 1:

[1346] Ingredient recognition

[1347] Input: Image data from imaging devices installed in refrigerators and pantries.

[1348] Specific operation: The camera scans the barcodes and QR codes on ingredients and seasonings in the refrigerator and sends the information to the server.

[1349] Data processing: The server analyzes the received image data and extracts information about ingredients from barcodes and QR codes.

[1350] Output: Information on the extracted ingredients (type, quantity, expiration date, etc.).

[1351] Step 2:

[1352] Saving food information

[1353] Input: Ingredient information extracted in step 1.

[1354] Specific operation: The server saves the extracted ingredient information in the database and updates the inventory information.

[1355] Data processing: Store information such as the type of ingredients, quantity, and expiration date in a database.

[1356] Output: Updated database inventory information.

[1357] Step 3:

[1358] Emotion recognition

[1359] Input: User facial and voice data captured from the device's camera and microphone.

[1360] Specific operation: When the user speaks to the camera or shows facial expressions, the device captures the data and sends it to the server.

[1361] Data processing: The server uses an emotion analysis device to analyze facial expressions and voice data to identify the user's emotions.

[1362] Output: The user's emotional state (happy, sad, excited, depressed, etc.).

[1363] Step 4:

[1364] Generate Menu

[1365] Input: Inventory information updated in step 2 and the user's emotional state identified in step 3.

[1366] Specific operation: The server uses generative artificial intelligence to generate a menu based on inventory information and the user's emotional state.

[1367] Data processing: Menus are generated taking into account parameters such as the number of family members, past menu history, and season.

[1368] Output: The generated menu.

[1369] Step 5:

[1370] Menu suggestions

[1371] Input: The menu generated in step 4.

[1372] Specific operation: The server provides a means to notify or display the generated menu to the user, for example, by displaying the menu through a smartphone app or display.

[1373] Data processing: Converts menu information into an appropriate format for display.

[1374] Output: The menu suggested to the user.

[1375] Step 6:

[1376] Menu adjustments

[1377] Input: The menu proposed in step 5 and user feedback.

[1378] Specific Actions: The server provides a means to appropriately change or adjust the generated menu based on the user's emotions.

[1379] Data processing: Analyze user feedback and regenerate or adjust menus as needed.

[1380] Output: Adjusted menu.

[1381] (Application example 1)

[1382] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1383] Conventional food management systems were able to recognize ingredients and seasonings in the refrigerator or pantry and generate menus, but they lacked the ability to suggest menus that took the user's emotions into account or arrange food delivery based on the generated menus. As a result, they were unable to suggest appropriate menus based on the user's emotions and physical condition, and it was difficult to arrange those menus quickly. This made it difficult to improve user satisfaction.

[1384] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1385] In this invention, the server includes a camera means for recognizing ingredients and seasonings, a generative AI means for generating a menu based on the recognized ingredients and seasonings, a means for proposing the generated menu to the user, an emotion recognition means for recognizing the user's emotion, a means for adjusting the menu based on the emotion recognition means, and a means for arranging food delivery based on the adjusted menu. This makes it possible to propose an appropriate menu based on the user's emotion and physical condition, and further to quickly arrange the menu.

[1386] A "camera for recognizing ingredients and seasonings" is a photographic device that is installed in a refrigerator or pantry to recognize the type, quantity, expiration date, etc. of ingredients and seasonings.

[1387] "Generative AI" is an artificial intelligence system that generates appropriate menus based on information about recognized ingredients and seasonings.

[1388] "Means for proposing the generated menu to the user" refers to an interface or device for notifying and proposing the menu generated by generative AI to the user.

[1389] "Emotion recognition means for recognizing user emotions" refers to technology or devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[1390] The "means for adjusting the menu based on the emotion recognition means" is a system for appropriately changing and adjusting the generated menu according to the recognized user emotion.

[1391] "Means for arranging food delivery based on a coordinated menu" means a system for ordering and arranging ingredients and dishes through a food delivery service based on a coordinated menu.

[1392] A system for implementing this invention is configured as follows. First, a camera installed in the refrigerator or pantry recognizes ingredients and seasonings. The camera obtains information about ingredients by reading barcodes or QR codes, and estimates the type and quantity of ingredients using image recognition technology. This camera can use, for example, the "cv2.VideoCapture" or "pyzbar" library.

[1393] Next, the information on the recognized ingredients and seasonings is sent to a generative AI, which generates an appropriate menu. The generative AI uses the "RecipeGenerator" class, for example, to generate a menu based on the ingredient information. Parameters such as the number of family members, past menu history, and the season are also taken into account.

[1394] The generated menu is presented to the user via a smartphone application, tablet device, or other device, allowing the user to check the generated menu.

[1395] Furthermore, an emotion recognition means is used to recognize the user's emotions. The emotion recognition means, for example, uses the "EmotionRecognizer" class to analyze and recognize emotions from the user's facial expressions and voice. Based on the recognized emotions, the generated menu is appropriately adjusted. For example, if the user shows emotions of joy or excitement, a luxurious menu is suggested, whereas if the user shows emotions of sadness or depression, a menu that is easy to digest is suggested.

[1396] A means for arranging food delivery is used based on the adjusted menu. The food delivery arrangement means uses, for example, the "FoodDeliveryService" class to order and arrange ingredients and dishes based on the adjusted menu. This allows users to quickly receive appropriate meals from the comfort of their own homes.

[1397] As a concrete example, a user scans the ingredients in their refrigerator with their camera and takes a photo of their face with their smartphone camera. The app retrieves that information, suggests the best menu based on the user's emotions, and arranges food delivery based on that menu. An example prompt is, "Scan the ingredients in your refrigerator with your camera and take a photo of your face with your smartphone camera. The app will suggest the best menu based on your emotions and arrange food delivery based on that menu."

[1398] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1399] Step 1:

[1400] The server obtains information about ingredients and seasonings from cameras installed in the refrigerator or pantry. The camera obtains information about ingredients by reading barcodes and QR codes, and uses image recognition technology to estimate the type and quantity of ingredients. The input is video data from the camera, and the output is information about the recognized ingredients and seasonings. Specifically, the server analyzes the video data using the "cv2.VideoCapture" and "pyzbar" libraries and extracts ingredient information.

[1401] Step 2:

[1402] The server sends the acquired information on ingredients and seasonings to the generative AI. The generative AI generates a menu based on the ingredient information. The input is information on ingredients and seasonings, and the output is the generated menu. Specifically, it uses the "RecipeGenerator" class to generate an appropriate menu based on the ingredient information. Parameters such as the number of family members, past menu history, and season are also taken into consideration.

[1403] Step 3:

[1404] The server proposes the generated menu to the user. The proposal method is a smartphone application or a tablet device. The input is the generated menu, and the output is the menu proposal to the user. Specifically, the server notifies the smartphone or tablet device of the generated menu so that the user can check it.

[1405] Step 4:

[1406] The device uses an emotion recognition means to recognize the user's emotions. The emotion recognition means analyzes and recognizes emotions from the user's facial expressions and voice. The input is the user's facial expression and voice data, and the output is the recognized emotion. Specifically, it uses the "EmotionRecognizer" class to analyze the user's facial expressions and voice and recognize emotions.

[1407] Step 5:

[1408] The server adjusts the generated menu based on the recognized emotion. The input is the recognized emotion and the generated menu, and the output is the adjusted menu. Specifically, the generated menu is modified and adjusted according to the recognized emotion. For example, if the user expresses joy or excitement, a luxurious menu is suggested, while if the user expresses sadness or depression, a menu that is easy to digest is suggested.

[1409] Step 6:

[1410] The server arranges food delivery based on the adjusted menu. The input is the adjusted menu, and the output is the food delivery order status. Specifically, it uses the "FoodDeliveryService" class to order and arrange ingredients and dishes based on the adjusted menu. This allows users to quickly receive appropriate meals from the comfort of their own home.

[1411] Example 2

[1412] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1413] Conventional menu generation systems can generate menus based on the recognition of ingredients and seasonings, but they have the problem of not being able to propose menus that take into account parameters such as the user's emotions, the number of family members, past menu history, and the season. Also, because the proposal method cannot be changed according to the user's emotions, it is difficult to increase user satisfaction.

[1414] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1415] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, and an emotion engine for changing the menu suggestion method based on the user's emotions. This makes it possible to generate menus taking into account parameters such as the number of family members, past menu history, and season, and also to change the suggestion method according to the user's emotions.

[1416] The "imaging device" is a device used to recognize ingredients and seasonings.

[1417] "Generative AI" is AI that generates menus based on recognized ingredients and seasonings.

[1418] The "proposing means" is a means for proposing the generated menu to the user.

[1419] The "emotion engine" is an engine for changing the way menus are suggested based on the user's emotions.

[1420] "Parameters" are factors that affect menu generation, such as the number of family members, past menu history, and season.

[1421] "Expiration date" refers to the date by which food can be safely consumed.

[1422] The present invention relates to a system for recognizing ingredients and seasonings and generating menus based on the ingredients and seasonings. Specific embodiments of this system will be described below.

[1423] System configuration

[1424] The system consists of the following major components:

[1425] 1. Imaging equipment:

[1426] This is a device for recognizing ingredients and seasonings. For example, it can use the camera built into a smartphone or tablet.

[1427] 2. Generative AI:

[1428] It is an artificial intelligence that generates menus based on recognized ingredients and seasonings. Specifically, it is implemented using machine learning libraries such as Python and TensorFlow. GPT-4 and other generative AI models are used.

[1429] 3. Proposed method:

[1430] This is a means for suggesting the generated menu to the user. For example, there are methods such as displaying the menu on the screen of a smartphone or tablet, or suggesting it by voice.

[1431] 4. Emotion Engine:

[1432] It is an engine that changes the way it suggests menus based on the user's emotions, for example, if the user is happy or excited, it will make audible suggestions, and if the user is sad or depressed, it will send silent notifications.

[1433] System Operation

[1434] Data collection:

[1435] Using a smartphone app or web interface, users input information about the ingredients and seasonings they currently have on hand—for example, "tomatoes, chicken, basil"—as well as additional parameters such as the number of people in their household, past menu history, and the season.

[1436] Data transmission:

[1437] The device sends the data entered by the user to the server, along with additional parameters such as the number of family members, past menu history, and season.

[1438] Data preprocessing:

[1439] The server receives and standardizes the data sent from the device, for example, by checking the type and quantity of ingredients and taking expiration dates into account.

[1440] Menu generation:

[1441] The server uses generative artificial intelligence to generate menus based on the collected data, taking into account parameters such as the type and quantity of ingredients, expiration dates, number of family members, past menu history, and the season to create the optimal menu.

[1442] Applying the Emotion Engine:

[1443] The server uses an emotion engine to determine how to suggest menu items based on the user's emotions, for example, suggesting menu items through voice if the user is expressing happiness or excitement, or sending silent notifications if the user is expressing sadness or depression.

[1444] Menu suggestion:

[1445] The server sends the generated menu to the device. The device then presents the menu received from the server to the user. For example, a recipe for "Chicken pasta with tomatoes and basil" is displayed on the smartphone screen. If the user shows a happy emotion, the device will suggest by voice, "Let's make chicken pasta with tomatoes and basil today!"

[1446] Examples of concrete examples and prompts

[1447] Examples:

[1448] A user inputs "tomato, chicken, basil" through a smartphone app and sets the number of people in their family as four. The device sends this information to the server. The server recognizes from past menu history that the family tends to like Italian food, and taking into account that it is summer, generates the following menu:

[1449] Menu: Chicken pasta with tomato and basil

[1450] Recipe: Stir-fry chicken, add tomatoes and basil, and toss with pasta

[1451] The server uses an emotion engine to detect that the user is expressing joy, and suggests in voice, "Let's make chicken pasta with tomatoes and basil today!" The device plays this voice message to the user, while simultaneously displaying the recipe on the smartphone screen.

[1452] Example prompt sentence:

[1453] "Generate a menu using tomatoes, chicken, and basil to make a meal for a family of four. Past menu history shows that your family prefers Italian food, and it's summer."

[1454] In this way, the system can generate optimal menus based on the user's input data and additional parameters, and suggest them to the user in an appropriate way using the emotion engine.

[1455] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1456] Step 1:

[1457] The user inputs data on ingredients and seasonings.

[1458] Using a smartphone app or web interface, users input information about the ingredients and seasonings they currently have on hand—for example, "tomatoes, chicken, and basil"—as well as additional parameters such as the number of family members, past menu history, and season. Input data includes the type of ingredient, quantity, expiration date, number of family members, past menu history, and season.

[1459] Step 2:

[1460] The terminal sends the input data to the server.

[1461] The terminal sends the data entered by the user to the server, along with additional parameters such as the number of family members, past menu history, and season. The input data includes the type of ingredient, amount, expiration date, number of family members, past menu history, and season, and the output is the data sent to the server.

[1462] Step 3:

[1463] The server receives the data and performs pre-processing.

[1464] The server receives data sent from the terminal. It standardizes the received data and organizes it taking into account the type and quantity of ingredients and their expiration date. For example, it checks specific quantities such as 3 tomatoes, 500 grams of chicken, and 10 basil leaves. The input data is the data sent from the terminal, and the output is standardized data.

[1465] Step 4:

[1466] The server generates the menu using a generative AI model.

[1467] The server generates a menu using a generative AI model (e.g., GPT-4) based on the collected data. It generates the optimal menu by taking into account parameters such as the type and amount of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in a family, it generates a menu for cooking meals for four people. The input data is standardized data, and the output is the generated menu.

[1468] Step 5:

[1469] The server uses an emotion engine to determine the proposal method.

[1470] The server uses an emotion engine to determine how to suggest a menu based on the user's emotions. For example, if the user expresses joy or excitement, the emotion engine will suggest a menu by voice. Conversely, if the user expresses sadness or depression, the emotion engine will silently send a notification to a device such as a smartphone or tablet. The input data is the user's emotion information and the generated menu, and the output is the suggestion method.

[1471] Step 6:

[1472] The server sends the generated menu to the terminal.

[1473] The server sends the generated menu to the terminal. The data sent includes specific recipes and cooking instructions. The input data is the generated menu and the proposed method, and the output is the data sent to the terminal.

[1474] Step 7:

[1475] The terminal presents the user with a menu.

[1476] The device presents the menu received from the server to the user. For example, a recipe for "Chicken pasta with tomatoes and basil" is displayed on the smartphone screen. If the user shows a happy emotion, the device will suggest by voice, "Let's make chicken pasta with tomatoes and basil today!" The input data is the data received from the server, and the output is the menu presented to the user.

[1477] (Application example 2)

[1478] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1479] Conventional menu generation systems can generate menus based on the recognition of ingredients and seasonings, but they cannot respond to user emotions or missing ingredient requests. Therefore, there is a need for menu suggestions that take user emotions into consideration and for automated ordering of necessary ingredients. Furthermore, menu generation that takes into account parameters such as the user's family composition, past menu history, and seasons is also insufficient.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1481] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, an emotion engine for recognizing the user's emotions, a means for changing the menu suggestion method based on the user's emotions, and a delivery service linkage means for ordering missing ingredients. This makes it possible to propose a menu that takes the user's emotions into consideration and automate the ordering of necessary ingredients.

[1482] A "camera for recognizing ingredients and seasonings" is a device that captures the type and condition of ingredients and seasonings as images and analyzes them.

[1483] "Generative AI" is artificial intelligence that automatically generates optimal menus based on information about recognized ingredients and seasonings.

[1484] "Means for proposing the generated menu to the user" refers to a method or device for informing the user of the menu generated by the generative AI.

[1485] An "emotion engine for recognizing a user's emotions" is a device or software that analyzes the user's emotions from their facial expressions, voice, etc., and recognizes their state.

[1486] The "means for changing the method of suggesting menu items based on the user's emotions" refers to a method or device for changing the method of suggesting menu items in accordance with the user's emotions recognized by the emotion engine.

[1487] A "delivery service collaboration means for ordering missing ingredients" is a method or device for collaborating with an external delivery service to order ingredients required for a generated menu when those ingredients are missing.

[1488] "Family size" is the number of people in the user's household that will be taken into account when generating the menu.

[1489] "Past menu history" is a record of menu items that a user has previously selected or consumed.

[1490] "Season" refers to information about the current time or season that is taken into consideration when generating a menu.

[1491] A "use-by date" is the date by which food ingredients or seasonings can be safely consumed, after which the quality may deteriorate.

[1492] The following system configuration will be described as an embodiment of the present invention.

[1493] System Configuration

[1494] 1. Hardware Configuration

[1495] Camera: A camera is used to recognize ingredients and seasonings. The camera can be a built-in camera on a smartphone or tablet.

[1496] Smartphone or tablet: Used as a device for users to operate the system.

[1497] Server: A server is used to run the generative AI, emotion engine, and delivery service integration means.

[1498] 2. Software Configuration

[1499] Generative AI: Artificial intelligence that automatically generates optimal menus based on recognized information about ingredients and seasonings. Implemented in programming languages ​​such as Python.

[1500] Emotion engine: Software that analyzes the user's emotions from facial expressions, voice, etc. and recognizes their state. It uses libraries such as EmotionEngine.

[1501] Delivery service integration: API integration function for ordering missing ingredients. Uses DeliveryService API, etc.

[1502] Processing flow

[1503] 1. Recognizing ingredients

[1504] The user takes a photo of the food in the refrigerator using the camera on their smartphone or tablet.

[1505] Images taken by the camera are sent to the server, where ingredients and seasonings are recognized using the FoodRecognition library.

[1506] 2. Creating a menu

[1507] The generative AI generates the optimal menu based on parameters such as recognized ingredients and seasoning information, number of family members, past menu history, and season.

[1508] The generated menu is displayed on the user's smartphone or tablet.

[1509] 3. Emotion-based recommendations

[1510] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions.

[1511] The way the menu is suggested changes depending on the user's emotions. For example, if the user is happy, suggestions are made by voice, and if the user is depressed, suggestions are made by notification.

[1512] 4. Ordering missing ingredients

[1513] If any ingredients required for the generated menu are missing, the missing ingredients are ordered using a delivery service linkage means.

[1514] Orders are placed via the user's smartphone or tablet.

[1515] Specific examples

[1516] The user takes a photo of the ingredients in the refrigerator with their smartphone. The app recognizes the ingredients and generates a menu taking into account the number of family members and past menu history. If the user is happy, it will suggest "Today's menu is curry!" via voice. If the necessary ingredients are missing, it will place an order with a food delivery service.

[1517] Prompt Sentence Examples

[1518] Your goal is to take a photo of the ingredients in the refrigerator taken by the user and generate the optimal menu taking into account the number of family members, past menu history, and the season.You also need to change the way you suggest menus depending on the user's emotions, and order food from a food delivery service if the necessary ingredients are in short supply.

[1519] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1520] Step 1:

[1521] The user takes a photo of the food in the refrigerator using the camera on their smartphone or tablet.

[1522] Input: Images of ingredients in the refrigerator

[1523] Output: Captured image data

[1524] Specific operation: The user launches the camera app and takes a picture of the food in the refrigerator. The image is saved on the device.

[1525] Step 2:

[1526] The terminal transmits the captured image data to the server.

[1527] Input: Captured image data

[1528] Output: Image data sent to the server

[1529] Specific operation: The device uploads image data to a server via the Internet.

[1530] Step 3:

[1531] The server receives the image data and uses the FoodRecognition library to recognize ingredients and seasonings.

[1532] Input: Image data sent to the server

[1533] Output: A list of recognized ingredients and seasonings

[1534] How it works: The server analyzes the image data and identifies ingredients and seasonings using the FoodRecognition library. The recognition results are saved in a list format.

[1535] Step 4:

[1536] The server uses generative AI to generate the optimal menu based on parameters such as information on ingredients and seasonings recognized, the number of family members, past menu history, and the season.

[1537] Input: List of recognized ingredients and seasonings, number of family members, past menu history, seasonal information

[1538] Output: The generated menu

[1539] Specific operation: The server inputs the necessary parameters into the generative AI and generates the optimal menu. The generated menu is saved in the database.

[1540] Step 5:

[1541] The server transmits the generated menu to the terminal, which displays the menu.

[1542] Input: The generated menu

[1543] Output: Menu displayed on the terminal

[1544] Specific operation: The server sends the generated menu to the terminal, and the terminal displays the menu to the user.

[1545] Step 6:

[1546] The device captures the user's facial expressions and voice using a camera and microphone and sends them to a server.

[1547] Input: User facial and voice data

[1548] Output: Facial expression and voice data sent to the server

[1549] Specific operation: The device uses the camera and microphone to capture the user's facial expressions and voice, and uploads the data to the server.

[1550] Step 7:

[1551] The server uses an emotion engine to analyze and recognize the user's emotions.

[1552] Input: Facial expression and voice data sent to the server

[1553] Output: Perceived user emotion

[1554] Specific operation: The server uses the emotion engine to analyze facial and voice data to identify the user's emotions.

[1555] Step 8:

[1556] The server changes the way it suggests menus depending on the user's emotions and notifies the terminal.

[1557] Input: Recognized user sentiment, generated menu

[1558] Output: A method for suggesting menu items based on user emotions

[1559] Specific operation: Based on the user's emotion, the server decides whether to suggest the menu by voice or notification, and sends instructions to the device. The device then suggests the menu according to the instructions.

[1560] Step 9:

[1561] The server checks whether ingredients necessary for the generated menu are in short supply, and if so, places an order using a delivery service linkage means.

[1562] Input: Generated menu, user's ingredient inventory information

[1563] Output: Order for missing ingredients

[1564] Specific operation: The server compares the generated menu with the user's ingredient inventory information to identify any missing ingredients, and uses the delivery service API to order the missing ingredients.

[1565] Example 3

[1566] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1567] Conventional menu generation systems have the problem that it is difficult to make personalized suggestions because they do not sufficiently consider the user's preferences and emotions. Also, because the menu is not optimized according to changes in the user's emotions, it is not possible to increase user satisfaction.

[1568] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1569] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to the user, an emotion engine for collecting and learning user emotion data, and a means for optimizing menu generation based on the data learned by the emotion engine. This makes it possible to propose personalized menus that take into account the user's preferences and emotions, thereby increasing user satisfaction.

[1570] The "camera" is a photographic device used to recognize ingredients and seasonings.

[1571] "Generative AI" is artificial intelligence that generates menus based on recognized ingredients and seasonings.

[1572] The "means for suggesting" is a method or device for notifying or guiding the user about the generated menu.

[1573] The "emotion engine" is a system for collecting and learning user emotional data.

[1574] The "optimizing means" refers to a method or device for optimizing menu generation based on data learned by the emotion engine.

[1575] "Ingredients" are the ingredients used in cooking.

[1576] A "seasoning" is a substance used to adjust the flavor of food.

[1577] A "menu" is a list of dishes suggested to the user.

[1578] A "user" is someone who uses the system.

[1579] "Data" is a collection of information that is processed by a system.

[1580] This invention is a system that proposes personalized menus to users. The system includes a camera for recognizing ingredients and seasonings, a generative AI that generates menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to users, an emotion engine that collects and learns user emotion data, and a means for optimizing menu generation based on the data learned by the emotion engine.

[1581] The server first uses a camera to recognize ingredients and seasonings. For example, a camera inside the refrigerator takes a picture of the type and quantity of ingredients and sends that data to the server. The server then inputs the recognized ingredient and seasoning data into a generative AI, which then generates a menu taking into account the user's preferences and past selection history. For example, the generative AI might suggest curry rice based on data that shows the user has previously preferred curry rice.

[1582] Next, the server selects the means by which to present the generated menu to the user. For example, in the morning, it may select a method of sending a notification to the smartphone, and in the evening, it may select a method of providing voice guidance. The device notifies the user of the menu sent from the server. For example, when the user wakes up in the morning, a notification may appear on the smartphone saying, "How about curry rice for dinner tonight?" And when the user returns home, a voice prompt may say, "Welcome home. Tonight's dinner is curry rice."

[1583] Furthermore, the server collects user emotional data and uses the emotion engine to learn. For example, if a user expresses joy in response to a menu item, the data is recorded. The emotion engine then optimizes the next menu generation based on the collected emotional data. For example, if a user tends to select a specific ingredient (e.g., an ingredient in curry rice) after expressing joy, the server will prioritize generating a menu that includes that ingredient.

[1584] An example prompt is "Generate a menu that includes ingredients that users tend to select after expressing the emotion of pleasure."

[1585] In this way, the system can deeply understand the user's feelings and preferences and provide more personalized services. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1586] Step 1:

[1587] The server uses a camera to recognize ingredients and seasonings. The camera takes a picture of the ingredients in the refrigerator and sends the image data to the server. The server then uses an image recognition algorithm to identify the type and amount of ingredients and seasonings. The input is the image data from the camera, and the output is a list of recognized ingredients and seasonings.

[1588] Step 2:

[1589] The server inputs the recognized ingredients and seasoning data into a generative AI to generate a menu. The generative AI creates a menu taking into account the user's preferences and past selection history. For example, it suggests curry rice based on data that indicates "the user has previously preferred curry rice." The input is a list of recognized ingredients and seasonings and the user's history data, and the output is the generated menu.

[1590] Step 3:

[1591] The server selects the method for proposing the generated menu to the user. It decides whether to send a notification to the user's smartphone or provide voice guidance depending on the user's lifestyle and time of day. For example, in the morning, it selects the method of sending a notification to the smartphone, and in the evening, it selects the method of providing voice guidance. The input is the generated menu and the user's situation data, and the output is the selection of the proposal method.

[1592] Step 4:

[1593] The device notifies the user of the menu sent from the server. For example, when you wake up in the morning, a notification appears on your smartphone saying, "How about curry rice for dinner tonight?". Also, when you return home, a voice message will say, "Welcome home. Tonight's dinner is curry rice." The input is the suggested method and menu from the server, and the output is a notification or voice guidance to the user.

[1594] Step 5:

[1595] The server collects user emotion data and uses the emotion engine to learn. For example, if a user expresses joy in response to a menu item, the server records the data. The input is the user's emotion feedback, and the output is the learned emotion data.

[1596] Step 6:

[1597] The server optimizes the next menu generation based on the data learned by the emotion engine. For example, if a user tends to select a specific ingredient (such as an ingredient in curry rice) after expressing joy, the server will prioritize generating a menu that includes that ingredient. The input is the learned emotion data, and the output is an optimized menu.

[1598] (Application example 3)

[1599] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1600] Conventional menu generation systems generate menus simply based on the user's perception of ingredients and seasonings, without considering the user's emotions or lifestyle. As a result, they are unable to propose the optimal menu for the user, making it difficult to provide personalized services. Furthermore, because they are unable to propose menus that respond to changes in the user's emotions, they are unable to increase user satisfaction.

[1601] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1602] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for proposing the generated menus to the user, an emotion engine for analyzing the user's emotions, and a means for optimizing menu generation based on the emotion engine. This makes it possible to propose personalized menus according to the user's emotions and lifestyle, thereby increasing user satisfaction.

[1603] A "camera for recognizing ingredients and seasonings" is a device that captures the type and condition of ingredients and seasonings as images and analyzes them.

[1604] "Generative AI" is artificial intelligence that automatically generates appropriate menus based on information about recognized ingredients and seasonings.

[1605] The "means for proposing the generated menu to the user" refers to a device or software for presenting the generated menu to the user by means of notification, voice guidance, or the like.

[1606] An "emotion engine that analyzes user emotions" is software or hardware that analyzes the user's emotional state and optimizes menu generation based on that information.

[1607] The "means for optimizing menu generation based on an emotion engine" refers to a device or software for generating an optimal menu for a user based on emotion information obtained from the emotion engine.

[1608] A system for implementing this invention has the following configuration: The server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to the user, an emotion engine for analyzing the user's emotions, and a means for optimizing menu generation based on the emotion engine.

[1609] Hardware and Software Configuration

[1610] Camera: A device that captures images of the type and condition of ingredients and seasonings and analyzes them. For example, a smartphone camera or a dedicated image recognition camera can be used.

[1611] Generative AI: This is artificial intelligence that automatically generates appropriate menus based on recognized ingredients and seasoning information. For example, OpenAI's GPT-3 model is used.

[1612] Emotion engine: Software or hardware that analyzes the user's emotional state and optimizes menu generation based on that information. For example, a sentiment analysis library such as TextBlob is used.

[1613] Notification means: A device or software that presents the generated menu to the user by means of notification or voice guidance. For example, the notification function of a smartphone or a voice assistant is used.

[1614] Data processing and calculation

[1615] 1. Ingredient Recognition: Images captured by the camera are analyzed to identify the type of ingredients and seasonings. This is done using image recognition technology.

[1616] 2. Sentiment Analysis: Analyze the user's input text or voice data and calculate a sentiment score. Use a sentiment analysis library such as TextBlob.

[1617] 3. Menu generation: A generative AI generates a menu based on the emotion scores and recognized ingredient information. It uses OpenAI's GPT-3 model.

[1618] 4. Notification: The generated menu is sent to your smartphone via text message or voice guidance.

[1619] Specific examples

[1620] For example, if a user types, "I'm feeling very happy today!", the emotion engine analyzes this text and calculates a high emotion score. Next, the generative AI generates the optimal menu for the user based on this emotion score and the ingredient information recognized by the camera. The generated menu is displayed on the smartphone as "Today's dinner suggestion: steak and salad."

[1621] Prompt Sentence Examples

[1622] "Please suggest a menu for dinner tonight. Sentiment score is 0.8."

[1623] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1624] Step 1:

[1625] The server uses a camera to capture images of ingredients and seasonings. The input is the image captured by the camera, and the output is image data. This image data is used to identify the type and condition of the ingredients and seasonings.

[1626] Step 2:

[1627] The server uses image recognition technology to analyze the acquired image data and identify the types of ingredients and seasonings. The input is the image data acquired in step 1, and the output is a list of recognized ingredients and seasonings. This list is used to generate the subsequent menu.

[1628] Step 3:

[1629] The user inputs text or voice data that indicates their emotional state into the terminal. The input is the user's text or voice data, and the output is data that indicates their emotional state. This data is analyzed by the emotion engine.

[1630] Step 4:

[1631] The server uses an emotion engine to analyze the user's input data and calculate an emotion score. The input is the data indicating the emotional state obtained in step 3, and the output is an emotion score. This emotion score is used to optimize menu generation.

[1632] Step 5:

[1633] The server generates a menu based on the list of ingredients and seasonings recognized using generative AI and their emotion scores. The input is the list of ingredients and seasonings obtained in step 2 and the emotion scores obtained in step 4, and the output is the generated menu. This menu is then proposed to the user.

[1634] Step 6:

[1635] The server notifies the terminal of the generated menu. The input is the menu generated in step 5, and the output is a notification message to be displayed on the terminal. The notification is made by a text message or a voice guidance.

[1636] Step 7:

[1637] The user checks the notification message displayed on the terminal and decides whether to accept the proposed menu. The input is the menu notified in step 6, and the output is the user's selection. If the user accepts the menu, the meal is prepared based on that menu.

[1638] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1639] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1640] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1641] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1642] [Third embodiment]

[1643] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1644] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1645] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1646] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1647] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1648] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1649] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1650] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1651] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1652] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1653] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1654] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1655] "Example 1"

[1656] In one embodiment of the present invention, a camera installed in a refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera obtains information about ingredients by reading barcodes or QR codes. It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[1657] "Example 2"

[1658] The generative AI generates a menu based on the recognized ingredients and seasonings. The generative AI generates the optimal menu by taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in a family, it generates a menu for cooking meals for four people. It also estimates the family's favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[1659] "Example 3"

[1660] The generated menu is then suggested to the user. The suggestion method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you get home, a voice guide will guide you through that day's dinner menu, making it possible to make suggestions tailored to the user's lifestyle.

[1661] The processing flow of each embodiment will be described below.

[1662] "Example 1"

[1663] Step 1: A camera installed in the refrigerator or pantry recognizes ingredients and seasonings. This camera can recognize the type, quantity, expiration date, etc. of ingredients. For example, the camera can obtain information about ingredients by reading barcodes or QR codes. It is also possible to estimate the type and quantity of ingredients using image recognition technology.

[1664] "Example 2"

[1665] Step 1: Information about ingredients and seasonings obtained from the camera is sent to the generative AI.

[1666] Step 2: The generative AI generates the optimal menu, taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season. For example, if there are four people in the family, it generates a menu for cooking meals for four people. It also estimates the family's likely favorite meals from past menu history and generates a menu based on that. It is also possible to generate menus that prioritize the use of seasonal ingredients depending on the season.

[1667] "Example 3"

[1668] Step 1: Receive the menu generated by the generative AI.

[1669] Step 2: The generated menu is proposed to the user. The proposal method can be selected based on the user's situation and preferences, such as notification on a device such as a smartphone or tablet, or voice guidance. For example, when you wake up in the morning, a notification is sent to your smartphone suggesting that day's dinner menu. Or, when you return home, a voice guide will guide you through that day's dinner menu, making it possible to make proposals that suit the user's lifestyle.

[1670] Example 1

[1671] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1672] Conventional food management in refrigerators and pantries is often done manually, making it difficult to accurately grasp the type, quantity, and expiration date of ingredients. It is also common for ingredients to be overlooked due to expiration dates, resulting in waste. Furthermore, creating menus based on ingredient inventory information is time-consuming, creating a need for efficient food management.

[1673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1674] In this invention, the server includes a camera for recognizing ingredients and seasonings, a calculator for analyzing information on the recognized ingredients and seasonings, a storage device for saving the analysis results, and a display device for displaying the saved analysis results to the user. This allows the type, quantity, and expiration date of ingredients to be accurately determined, enabling efficient ingredient management.

[1675] The "photography device" is a device that captures images to recognize ingredients and seasonings.

[1676] The "computing device" is a device for analyzing information about recognized ingredients and seasonings.

[1677] A "storage device" is a device for saving analysis results.

[1678] A "display device" is a device for displaying the saved analysis results to a user.

[1679] "Image recognition technology" is a technology that detects and classifies specific objects and features from images.

[1680] A "barcode" is a code made up of a series of lines and spaces that contains information about ingredients and seasonings.

[1681] A "QR code" is a two-dimensional matrix code that contains information about ingredients and seasonings.

[1682] "Expiration date" is information indicating the date by which ingredients or seasonings can be safely consumed.

[1683] "Menu generation" is the process of suggesting dishes based on information about ingredients and seasonings.

[1684] MODE FOR CARRYING OUT THE INVENTION

[1685] This invention is a system that uses a camera installed in a refrigerator or pantry to recognize ingredients and seasonings, analyzes and stores that information, and displays it to the user. Specific embodiments of this system are described below.

[1686] server

[1687] The server receives image data sent from the camera installed in the refrigerator or pantry. A general server (e.g., cloud server or on-premise server) can be used as the hardware. HTTP requests are used to receive the image data.

[1688] The server preprocesses the received image data, including adjusting the image resolution and removing noise, using an image processing library such as OpenCV.

[1689] The server then uses image recognition technology to analyze the types, quantities, and expiration dates of ingredients and seasonings. Specifically, it uses machine learning libraries such as TensorFlow to classify ingredients in the image and uses object detection algorithms to estimate the amount of ingredients. It also uses the ZBar library to read barcodes and QR codes to obtain detailed information about ingredients.

[1690] The analysis results are converted to JSON format and stored in a database such as MySQL, allowing information such as the type, quantity, and expiration date of ingredients to be centrally managed.

[1691] Terminal

[1692] The device provides an application that allows users to check refrigerator and pantry inventory information. The hardware used can be a smartphone or tablet. The application runs on iOS or Android platforms.

[1693] The terminal retrieves inventory information from the server and displays it to the user. It parses the retrieved JSON data and displays the type, quantity, and expiration date of ingredients in a list format. This allows the user to easily check the inventory information of the refrigerator or pantry.

[1694] User

[1695] Users can check the inventory information of their refrigerator and pantry through the application. For example, a user can open the app and check the type, quantity, and expiration date of ingredients in the refrigerator.

[1696] Users can also create shopping lists based on inventory information. For example, users can select ingredients they are running low on and add them to the shopping list within the app. The list is saved on the device and can be edited as needed.

[1697] Examples and prompts

[1698] A specific example is obtaining information about ingredients in a refrigerator. A camera takes a picture of the ingredients in the refrigerator and sends the image data to a server. The server then uses an image recognition algorithm to analyze the type, quantity, and expiration date of the ingredients, and the analysis results are stored in a database. Users can then check this information through the app.

[1699] An example of a prompt sentence might be:

[1700] "I want to develop an application that retrieves information about ingredients in the refrigerator and displays inventory information. I would like to know how to build a system that sends images taken with a camera to a server and uses image recognition technology to analyze the type, quantity, and expiration date of ingredients."

[1701] In this way, by specifically explaining the system processing from the perspectives of the server, terminal, and user, the overall flow can be understood in detail.

[1702] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1703] Program processing flow

[1704] server

[1705] Step 1:

[1706] Receiving image data

[1707] The server receives image data transmitted from the image capturing device installed in the refrigerator or pantry.

[1708] Input: Image data sent from the imaging device

[1709] Output: Received image data

[1710] Specific operation: The camera periodically takes pictures of the inside of the refrigerator and sends them to the server via Wi-Fi. The server receives the HTTP request and saves the image data.

[1711] Step 2:

[1712] Image preprocessing

[1713] The server pre-processes the received image data.

[1714] Input: Received image data

[1715] Output: Preprocessed image data

[1716] Specific operation: The server adjusts the image resolution, removes noise, and adjusts the image contrast and brightness using OpenCV.

[1717] Step 3:

[1718] Barcode / QR code detection

[1719] The server detects barcodes and QR codes from the image.

[1720] Input: Preprocessed image data

[1721] Output: Information on detected barcodes and QR codes

[1722] What it does: The server uses the ZBar library to scan barcodes and QR codes in an image and extract the code's contents.

[1723] Step 4:

[1724] Database Matching

[1725] The server compares the extracted barcode or QR code information with a database.

[1726] Input: Information of the detected barcode or QR code

[1727] Output: Detailed information about ingredients retrieved from the database

[1728] Specific operation: The server sends a query to the MySQL database to obtain detailed information about the corresponding ingredients (type, quantity, expiration date, etc.).

[1729] Step 5:

[1730] Estimating the type and quantity of ingredients using image recognition

[1731] The server uses image recognition technology to estimate the type and quantity of ingredients.

[1732] Input: Preprocessed image data

[1733] Output: Estimated ingredients and quantities

[1734] How it works: The server uses a TensorFlow model to classify ingredients in the image and an object detection algorithm to estimate the amount of ingredients.

[1735] Step 6:

[1736] Saving analysis results

[1737] The server stores the analysis results in a database.

[1738] Input: Estimated ingredient type and quantity, detailed ingredient information retrieved from the database

[1739] Output: Analysis results stored in a database

[1740] Specific operation: The server converts the parsed results into JSON format and stores them in a MySQL database.

[1741] Terminal

[1742] Step 1:

[1743] Obtaining inventory information

[1744] The terminal obtains inventory information from the server.

[1745] Input: HTTP request to the server

[1746] Output: Stock information obtained from the server

[1747] What it does: When a user opens the app, the device sends an HTTP request to the server to get the latest inventory information.

[1748] Step 2:

[1749] Displaying inventory information

[1750] The terminal displays the acquired inventory information to the user.

[1751] Input: Inventory information obtained from the server

[1752] Output: Inventory information displayed to the user

[1753] Specific operation: The device parses the acquired JSON data and displays the type, quantity, and expiration date of the ingredients in a list format.

[1754] User

[1755] Step 1:

[1756] Check stock information

[1757] Users can check refrigerator and pantry inventory information through the app.

[1758] Input: Stock information displayed on the terminal

[1759] Output: User confirmation action

[1760] Specific operation: When a user opens the app, inventory information is displayed and they can check the type, quantity, and expiration date of ingredients.

[1761] Step 2:

[1762] Creating a shopping list

[1763] The user creates a shopping list based on inventory information.

[1764] Input: Stock information displayed on the terminal

[1765] Output: The created shopping list

[1766] How it works: Users select ingredients they're running low on and add them to a shopping list within the app. The list is saved on the device and can be edited as needed.

[1767] (Application example 1)

[1768] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1769] In modern households, managing the ingredients and seasonings stored in the refrigerator and pantry is complicated, and it is common for ingredients to pass their expiration date or run out of necessary ingredients. It is also difficult to keep track of ingredient inventory and plan menus efficiently. This results in food waste and increases household food costs. Furthermore, there is a need to reduce food waste by prioritizing the use of ingredients that are close to their expiration date.

[1770] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1771] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to the user, a means for managing ingredient inventory in real time, a means for notifying the user of ingredients approaching their expiration date, and a means for automatically ordering the necessary ingredients. This improves the efficiency of ingredient inventory management and enables priority use of ingredients approaching their expiration date. Furthermore, automatically ordering necessary ingredients can prevent ingredient shortages and reduce household food costs.

[1772] "Cameras for recognizing ingredients and seasonings" are cameras that are installed in refrigerators and pantries to recognize the type, quantity, and expiration date of ingredients and seasonings.

[1773] "Generative AI that generates menus based on recognized ingredients and seasonings" is artificial intelligence that automatically generates appropriate menus based on information about ingredients and seasonings recognized by the camera.

[1774] "Means for proposing the generated menu to the user" refers to means for notifying and proposing the menu generated by the generative AI to the user.

[1775] "Means for managing food inventory in real time" refers to a means for grasping and managing the current inventory status in real time based on the type and quantity of food ingredients recognized by the camera.

[1776] The "means for notifying the user of ingredients whose expiration date is approaching" is a means for monitoring the expiration date of ingredients and notifying the user of ingredients whose expiration date is approaching.

[1777] "Means for automatically ordering necessary ingredients" refers to a means for automatically ordering necessary ingredients from a food delivery service, etc., based on stock availability and expiration dates.

[1778] A system for implementing this invention includes a camera for recognizing ingredients and seasonings, a generative AI for generating menus based on the recognized ingredients and seasonings, a means for suggesting the generated menus to users, a means for managing ingredient inventory in real time, a means for notifying users of ingredients that are close to their expiration date, and a means for automatically ordering the necessary ingredients.

[1779] Hardware and Software Configuration

[1780] Hardware

[1781] Camera: A camera installed in the refrigerator or pantry to recognize the type, quantity, and expiration date of ingredients and seasonings.

[1782] Smartphone: A device that allows users to operate applications.

[1783] Server: A central processing unit for generative AI and data management.

[1784] software

[1785] OpenCV: Image processing library. Used to analyze images acquired from the camera and recognize the types and quantities of ingredients and seasonings.

[1786] Pyzbar: A barcode / QR code reading library. Used to read barcodes and QR codes on ingredients and seasonings.

[1787] Requests: An HTTP request library that we use to order the ingredients we need from a food delivery service.

[1788] Processing flow

[1789] Camera-based food recognition

[1790] The camera is installed in the refrigerator or pantry and periodically captures images. The captured images are analyzed using OpenCV to identify the type, quantity, and expiration date of ingredients and seasonings. If a barcode or QR code is included, the information is read using Pyzbar.

[1791] Menu generation using generative AI

[1792] The information on the recognized ingredients and seasonings is sent to a server, and a generative AI generates a menu based on this information, taking into account parameters such as the number of family members, past menu history, and the season.

[1793] Menu suggestions and inventory management

[1794] The generated menu is presented to the user via a smartphone application, and the availability of ingredients is managed in real time, with notifications sent to users when any ingredients are nearing their expiration date.

[1795] Automated Orders

[1796] If a required ingredient is in short supply, the server uses the Requests library to automatically place an order with a food delivery service, preventing shortages and reducing household food costs.

[1797] Specific examples

[1798] For example, if there are five tomatoes in the refrigerator and their expiration date is approaching in three days, the camera will recognize this and the generative AI will generate a menu using tomatoes. The generated menu will be suggested to the user through a smartphone application. The user will also be notified that the expiration date is approaching and can order more tomatoes if necessary.

[1799] Prompt Sentence Examples

[1800] "Please create an application that recognizes the type and amount of food in the refrigerator and notifies you when the food is nearing its expiration date. Also, please add a function that automatically orders the necessary ingredients from a food delivery service."

[1801] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1802] Step 1:

[1803] Cameras are installed in refrigerators and pantries to periodically capture images.

[1804] Input: Images of ingredients and condiments in your refrigerator or pantry.

[1805] Output: The captured image data.

[1806] Specific operation: The camera takes images at regular intervals and sends the image data to the server.

[1807] Step 2:

[1808] The server uses OpenCV to analyze the captured images and recognize the type, quantity, and expiration date of ingredients and seasonings.

[1809] Input: The captured image data.

[1810] Output: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[1811] Specific operation: The server uses the image processing library OpenCV to identify ingredients and seasonings in the image, and if a barcode or QR code is included, it uses Pyzbar to read that information.

[1812] Step 3:

[1813] The server uses generative AI to generate a menu based on the information on ingredients and seasonings recognized by the server.

[1814] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[1815] Output: The generated menu.

[1816] Specific operation: The server inputs information about ingredients and seasonings into the generative AI, which then generates a menu taking into account parameters such as the number of family members, past menu history, and the season.

[1817] Step 4:

[1818] The server proposes the generated menu to the user through a smartphone application.

[1819] Input: The generated menu.

[1820] Output: The menu suggested to the user.

[1821] Specific operation: The server sends the generated menu to the smartphone application, and the application notifies the user.

[1822] Step 5:

[1823] The server manages food inventory in real time and notifies customers of food items approaching their expiration date.

[1824] Input: Information on the type, quantity, and expiration date of recognized ingredients and seasonings.

[1825] Output: Real-time inventory status, notifications of ingredients approaching their expiration date.

[1826] Specific operation: The server stores information about ingredients and seasonings in a database, and if any ingredients are nearing their expiration date, it notifies the user via a smartphone application.

[1827] Step 6:

[1828] The server automatically orders the necessary ingredients from a food delivery service.

[1829] Input: Availability and expiration date information.

[1830] Output: An order from a food delivery service.

[1831] Specific operation: The server uses the Requests library to automatically order ingredients that are out of stock from a food delivery service.

[1832] Example 2

[1833] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1834] Conventional menu generation systems make it difficult for users to efficiently use ingredients and seasonings they have on hand, and also make it difficult to generate optimal menus that take into account parameters such as the number of family members, past menu history, and the season. Furthermore, they are unable to generate menus that take into account the expiration dates of ingredients, which leads to the problem of food waste.

[1835] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1836] In this invention, the server includes means for receiving information on ingredients and seasonings entered by the user and saving it in a database, means for sending prompts to the generative AI based on the saved information, and means for returning the generated menu to the user's terminal. This allows the user to efficiently use the ingredients and seasonings on hand and generate an optimal menu taking into account parameters such as the number of family members, past menu history, and the season. Furthermore, since the menu can be generated taking into account the expiration dates of ingredients, food waste can be reduced.

[1837] The "photography device" is a device for recognizing ingredients and seasonings.

[1838] "Generative AI" is AI that generates menus based on recognized ingredients and seasonings.

[1839] The "means for suggesting to the user" is a means for presenting the generated menu to the user.

[1840] The "means for saving in a database" is a means for receiving information on ingredients and seasonings entered by the user and saving it in a database.

[1841] The "means for sending a prompt" is a means for sending a prompt to the generative artificial intelligence based on the stored information.

[1842] The "means for returning" is a means for returning the generated menu to the user's terminal.

[1843] "Number of family members" is information about the number of family members to be taken into consideration when creating a menu.

[1844] "Past menu history" is history information of menus that have been generated in the past.

[1845] "Season" is information about the current season that is taken into consideration when generating a menu.

[1846] "Expiration date" is information indicating the expiration date of a food ingredient.

[1847] The present invention is a system that recognizes ingredients and seasonings and generates an optimal menu based on the ingredients and seasonings. A specific embodiment of this system will be described below.

[1848] Hardware and software used

[1849] Hardware: Server, terminal (PC, smartphone, tablet, etc.), imaging device

[1850] Software: Generative AI models (e.g., GPT-4), database management systems (e.g., MySQL), and user interfaces (e.g., web applications)

[1851] System configuration

[1852] 1. Imaging equipment

[1853] This is a device for recognizing ingredients and seasonings. The user takes a photo of the ingredients or seasonings at hand and sends that information to the system.

[1854] 2. Server

[1855] The server receives the information about ingredients and seasonings entered by the user and stores it in a database.

[1856] The server sends prompts to the generative AI model based on the stored information.

[1857] The server returns the generated menu to the user's terminal.

[1858] 3. Generative AI Models

[1859] The generative AI model generates the optimal menu based on prompts sent from the server.

[1860] The generative AI model takes into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[1861] 4. Database

[1862] The database stores information entered by the user, such as ingredients and seasonings, the number of family members, past menu history, and the season.

[1863] 5. User Interface

[1864] The user interface is a web application that allows users to input information about ingredients and seasonings.

[1865] The user interface presents the generated menu to the user.

[1866] Specific examples

[1867] Prompt Sentence Examples

[1868] "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu items include curry, stew, and stir-fry. The current season is autumn."

[1869] System Operation

[1870] 1. The user inputs ingredients and seasonings

[1871] The user uses the terminal to input the types and amounts of ingredients and seasonings currently on hand.

[1872] Example: "200g chicken, 1 carrot, 1 onion, 50ml soy sauce"

[1873] 2. The server receives the input data and stores it in the database

[1874] The server receives the information about ingredients and seasonings sent by the user and stores it in a database.

[1875] Example: Save as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce" in a MySQL database.

[1876] 3. The server sends a prompt to the generative AI model

[1877] The server generates and sends prompts to the generative AI model, including parameters entered by the user, such as information on ingredients and seasonings, the number of family members, past menu history, and the season.

[1878] Example: "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn."

[1879] 4. Generative AI model generates optimal menu

[1880] The generative AI model generates the optimal menu based on the prompts received, taking into account parameters such as the type and quantity of ingredients, expiration dates, number of family members, past menu history, and the season.

[1881] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[1882] 5. The server returns the generated menu to the user

[1883] The server returns the menu received from the generative AI model to the user's device.

[1884] Example: "Chicken and vegetable stew, salad made with seasonal fall ingredients."

[1885] 6. User checks the menu and starts cooking

[1886] The user checks the menu generated on the terminal and starts cooking.

[1887] Example: "Check out the recipe for boiled chicken and vegetables and start cooking."

[1888] In this way, the system uses a generative AI model to generate the optimal menu based on the ingredients and seasoning information entered by the user and provides it to the user.

[1889] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1890] Step 1:

[1891] The user inputs ingredients and seasonings

[1892] Input: The user uses a device (PC, smartphone, tablet, etc.) to input the types and quantities of ingredients and seasonings currently on hand.

[1893] Specific operation: The user enters information such as "200g of chicken, 1 carrot, 1 onion, 50ml of soy sauce" into an input form in a web application and clicks the submit button.

[1894] Output: The information about the ingredients and seasonings entered is sent to the server.

[1895] Step 2:

[1896] The server receives the input data and stores it in the database

[1897] Input: Ingredient and seasoning information sent by the user

[1898] Specific operation: The server receives the HTTP request, analyzes its contents, and stores it in a database (e.g., MySQL) as "200g chicken, 1 carrot, 1 onion, 50ml soy sauce."

[1899] Output: Information on ingredients and seasonings stored in the database

[1900] Step 3:

[1901] The server sends prompts to the generative AI model

[1902] Input: Parameters such as ingredients and seasoning information stored in the database, number of family members, past menu history, and seasons

[1903] Specific operation: The server retrieves the necessary information from the database and sends the prompt statement, "Generate a menu for a family of four using 200g of chicken, 1 carrot, 1 onion, and 50ml of soy sauce. Past menu history includes curry, stew, and stir-fry. The current season is autumn," to the generative AI model (e.g., GPT-4).

[1904] Output: The prompt sent to the generative AI model

[1905] Step 4:

[1906] Generative AI models generate optimal menus

[1907] Input: Prompt sent from the server

[1908] How it works: The generative AI model analyzes the prompt and generates a menu such as "Chicken and vegetable stew, and a salad made with seasonal autumn ingredients," taking into account parameters such as the type and quantity of ingredients, expiration date, number of family members, past menu history, and season.

[1909] Output: The generated menu

[1910] Step 5:

[1911] The server returns the generated menu to the user

[1912] Input: Menu received from the generative AI model

[1913] Specific operation: The server sends the generated menu to the user's device as an HTTP response and displays it on the web application.

[1914] Output: Menu displayed on the user's device

[1915] Step 6:

[1916] The user checks the menu and starts cooking

[1917] Input: The menu displayed on the user's device

[1918] Specific behavior: The user checks the recipe displayed on the web application for "Chicken and vegetable stew, salad using seasonal autumn ingredients" and starts cooking by following the necessary steps.

[1919] Output: Specific instructions and ingredients for the user to start cooking

[1920] (Application example 2)

[1921] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1922] Conventional systems that generate menus by recognizing ingredients and seasonings require users to manually purchase additional ingredients and seasonings required for the generated menu, which is inconvenient. Furthermore, there is a lack of a way to maximize the use of ingredients and seasonings that users already have and reduce waste. Furthermore, there is a need for a way to quickly obtain additional ingredients and seasonings based on the generated menu.

[1923] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1924] In this invention, the server includes a camera for recognizing ingredients and seasonings, a generative AI for generating a menu based on the recognized ingredients and seasonings, a means for suggesting the generated menu to the user, and a means for ordering additional ingredients and seasonings required for the generated menu. This allows the user to make the most of the ingredients and seasonings they have and reduce waste. Furthermore, additional ingredients and seasonings can be quickly obtained based on the generated menu, improving user convenience.

[1925] A "camera for recognizing ingredients and seasonings" is a device that captures images of ingredients and seasonings owned by the user and analyzes them as digital data.

[1926] "Generative AI" is an artificial intelligence system that automatically generates the optimal menu for the user based on recognized information about ingredients and seasonings.

[1927] "Means for proposing the generated menu to the user" refers to an interface or function for notifying and displaying the menu generated by the generative AI to the user.

[1928] "Means for ordering additional ingredients and condiments required for the generated menu" refers to a function that automatically identifies missing ingredients and condiments based on the generated menu and orders them through a food delivery service.

[1929] "Family size" is the number of people in the user's household that will be taken into account when generating the menu.

[1930] "Past menu history" is a record of menus previously created or consumed by a user, data that generative AI uses to learn about a user's preferences and tendencies.

[1931] "Season" refers to information about the current time or season that is taken into consideration when generating a menu, and is a parameter for giving priority to using ingredients that are in season.

[1932] "Expiration date" is information indicating the date by which ingredients and seasonings can be safely consumed, and is used to determine which ingredients to prioritize when creating a menu.

[1933] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1934] System configuration

[1935] The system consists of the following major components:

[1936] 1. Camera for recognizing ingredients and seasonings

[1937] 2. Generative AI

[1938] 3. A means of suggesting the generated menu to the user

[1939] 4. A way to order any additional ingredients or condiments needed for the generated menu.

[1940] Program processing

[1941] Camera for recognizing ingredients and seasonings

[1942] A camera is used to recognize ingredients and seasonings that the user owns. The camera takes a picture of the ingredients in the refrigerator or pantry, and the image is analyzed as digital data. Specifically, image processing libraries such as OpenCV are used to recognize ingredients and seasonings.

[1943] Generative AI

[1944] Based on th...

Claims

1. A means for recognizing ingredients and seasonings photographed by a camera; A means for collecting menu options previously selected by a user, the user's preferences, allergy information, and schedule information; a means for generating menu candidates using a prompt sentence instructing the user to generate a menu that can be cooked from the recognized ingredients and seasonings based on the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and a generation AI model; A means for notifying a user terminal of the generated menu candidates; A system including:

2. means for recognizing an emotion of the user by an emotion engine; 2. The system according to claim 1, wherein the generating means generates menu candidates using a prompt sentence instructing the system to generate a cookable menu from the recognized ingredients and seasonings based on the recognized emotions of the user, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model.

3. The collecting means further collects the number of family members, the past menu history, or the season; The system according to claim 1, wherein the generating means generates menu candidates using a prompt sentence instructing the system to generate a menu that can be cooked from the recognized ingredients and seasonings based on the collected information on the number of family members, past menu history, or season, the recognized ingredients and seasonings, previously selected menu candidates, the user's preferences, allergy information, and schedule information, and the generation AI model.

Images