System

Abstract

A system is provided.SOLUTION: A means for allowing a user to take a picture of a meal, a server means for receiving the taken picture, an image recognition means for analyzing the received picture and specifying the content of the meal, a means for updating preference data of the user based on the specified content of the meal, a means for allowing the user to input an address, a family structure, a monthly food budget, and a means for storing the input information, a means for collecting bargain sale information from neighboring stores, a means for generating an optimal menu based on the preference data, the family structure, the food budget, the nutrient balance, and the bargain sale information, means for displaying the generated menu and associated ingredient list to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Planning daily menus is a major burden, especially for those in charge of family meals. It is extremely difficult to prepare meals that satisfy the whole family while maintaining a balanced nutritional diet. Furthermore, it is necessary to provide high-quality meals while keeping ingredient costs down. In these circumstances, there is a need for a system that can automatically suggest menus and provide consistent support, right down to purchasing ingredients. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides the following means.

[0006] Provided is a system including: means for a user to take a photo of a meal; server means for receiving the photograph; image recognition means for analyzing the received photo and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; and means for displaying the generated menu and related ingredient list to the user.

[0007] This system allows users to simply take a photo of their meal and automatically receive appropriate menu suggestions. It also takes into account special offers, the user's preferences, family composition, and nutritional balance, making it cost-effective and enabling users to maintain a balanced diet. Users can also easily purchase ingredients for the suggested menus and select delivery options, streamlining daily meal preparation.

[0008] "Photos of food" are image data of food photographed by a user with a camera.

[0009] A "user" is an individual who uses this system to take photos of their meals and input various information.

[0010] A "terminal" is an electronic device, such as a smartphone or tablet, that a user uses to access the system.

[0011] "Server" is the central part of the system, a computer system that receives, processes, stores, and transmits data from users.

[0012] "Image recognition" is a technology that allows the server to analyze photos of food and identify its contents.

[0013] "Preference data" is data that records the user's eating preferences and patterns.

[0014] "Family structure" is information indicating the number and relationships of members in the user's household.

[0015] The "monthly food budget" is the amount of money the user plans to spend on meals each month.

[0016] "Retailer" refers to a store that sells ingredients and groceries, such as supermarkets and markets.

[0017] "Special sale information" is information about specific ingredients or products being offered at discounted prices at nearby stores.

[0018] A "menu" is a specific menu combination suggested based on the user's meal plan.

[0019] "Nutritionally balanced" refers to a diet that contains healthy and balanced nutrients.

[0020] The "ingredient list" is a list of ingredients required to create the proposed menu.

[0021] "Purchase options" are ways for users to obtain ingredients, including online ordering and delivery services. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The system for implementing this invention allows users to simply take photos of their meals, and the system analyzes the photos to suggest optimal menus. Furthermore, by inputting the user's address, family composition, and monthly food budget, the system suggests cost-effective ingredients taking into account sales information from nearby stores, and also provides the option to purchase the ingredients if necessary.

[0044] Overview of program processing

[0045] 1. Take and upload photos of your food

[0046] Users take photos of their meals with their smartphones or tablets and upload them to the app.

[0047] The terminal transmits this photo data to the server.

[0048] 2. Photo analysis and updating of preference data

[0049] The server analyzes the received photos using image recognition technology.

[0050] The server identifies the type of meal shown in the photo (e.g., pasta, salad, etc.) and stores this in the user's preference database.

[0051] 3. Enter your user information

[0052] Users enter their address, family composition, and monthly food budget through the app.

[0053] The terminal transmits this information to the server.

[0054] The server stores the received information in a database.

[0055] 4. Collecting information on special offers

[0056] The server uses the API to collect sale information from nearby stores based on the user's address.

[0057] Special sale information is updated in real time and stored in a database.

[0058] 5. Menu generation and suggestions

[0059] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0060] The proposed menu and a list of ingredients required for it are sent from the server to the terminal.

[0061] The terminal displays this to the user.

[0062] 6. Select your food purchase options and place your order

[0063] The user reviews the proposed menu and selects how to purchase the ingredients needed (e.g., order online for delivery).

[0064] The device sends the selection to the server.

[0065] Based on the order information, the server contacts the affiliated delivery service and arranges for the ingredients to be delivered.

[0066] The user receives the ingredients at home and cooks according to the provided menu.

[0067] Specific examples

[0068] For example, if a user uploads a photo of "grilled salmon," the server analyzes the photo and learns that salmon and grilled dishes are popular. If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a supermarket is having a sale on salmon, the server will suggest a menu using the sale items: "grilled salmon + vegetable salad + soup." This suggestion includes a list of ingredients that are on sale, and the user can also select the option to order the ingredients online and have them delivered to their home.

[0069] In this way, the system streamlines the daily task of deciding on menus and purchasing ingredients, which can be a heavy burden for users, and supports healthy and economical eating habits.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[0073] Step 2:

[0074] The device uploads photos to the app. The device sends the photos it has taken to the app, which then transfers the photo data to the server.

[0075] Step 3:

[0076] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[0077] Step 4:

[0078] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "grilled salmon," "salad," etc.).

[0079] Step 5:

[0080] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[0081] Step 6:

[0082] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[0083] Step 7:

[0084] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[0085] Step 8:

[0086] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores the sale information in a database.

[0087] Step 9:

[0088] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0089] Step 10:

[0090] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[0091] Step 11:

[0092] The user selects a grocery purchase option. The user selects their preferred method of grocery purchase from several options within the app (e.g., online ordering, delivery, select location).

[0093] Step 12:

[0094] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[0095] Step 13:

[0096] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[0097] Step 14:

[0098] The user receives the ingredients and prepares the meal according to the provided menu.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In many modern households, planning daily menus and efficiently purchasing ingredients according to the family's preferences and budget are a significant burden. Many people also find it difficult to shop economically by utilizing sales information. The purpose of this invention is to solve these problems and provide a system that supports an efficient and economical dietary lifestyle.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes: means for a user to take photos of meals; computer means for receiving the photos; image recognition means for analyzing the received photos and identifying the contents of the meals; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; database means for storing the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and associated ingredient list to the user; means for presenting the user with options for purchasing ingredients based on the displayed menu and ingredient list; and means for processing ingredient orders based on the options. This allows users to efficiently decide on their daily menu and purchase the necessary ingredients economically.

[0104] A "user" is someone who uses the system to take photos of their meals and input information such as their personal information and food budget.

[0105] "Means for taking photos of food" refers to functions or applications for taking photos of food using electronic devices such as smartphones or tablet devices.

[0106] "Computer means" refers to a server or computer system that receives the captured photograph data and performs subsequent analysis and data storage.

[0107] "Image recognition means" refers to software or algorithms that analyze received photos and identify the ingredients or dishes depicted.

[0108] "Preference data" refers to a database that records a user's food preferences and tendencies.

[0109] "Means for inputting address, family composition, and monthly food budget" is a function that provides an interface for the user to input their own address, family composition, and monthly food budget.

[0110] The "database means" is a system for storing and managing input information and analysis results as structured data.

[0111] "Sales information" refers to sale information and product price information obtained from nearby retailers.

[0112] "Means of collecting sales information" refers to APIs and web scraping technologies for collecting real-time sales information and product price information from nearby retailers.

[0113] "Means for generating menus" refers to algorithms or programs that generate optimal menus based on collected preference data, family composition, food budget, nutritional balance, and sales information.

[0114] The "means for displaying the ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's terminal.

[0115] The "means for presenting options for purchasing ingredients" is a function that presents options for the user to order ingredients online.

[0116] The "means for processing an order" refers to a system for communicating ingredient order information to a related service based on a user's selection and processing the order.

[0117] This invention is a system that helps users efficiently decide on daily menus and purchase ingredients economically. This system allows users to take photos of meals using a smartphone or tablet device, analyzes the photos, updates the user's preference data, and provides menu suggestions and ingredient purchasing support that take into account the user's address, family composition, and monthly food budget entered by the user.

[0118] Specifically, the following hardware and software are used.

[0119] 1. Take and upload a photo of your meal:

[0120] Users take photos of their meals using their smartphones or tablets and upload them to a dedicated app, which then sends the photos to a server.

[0121] 2. Photo analysis and preference data update:

[0122] The server temporarily stores the received photos in local storage. The server then analyzes the photos using image recognition technologies such as Google Cloud Vision API and Amazon Rekognition. The analysis results are obtained as dish names and ingredient information, such as "grilled salmon" or "pasta." The server updates the user's preference database based on these analysis results.

[0123] 3. Enter your user information:

[0124] Users enter their address, family composition, and monthly food budget through the app, and the device sends the information to the server, which then stores it in a database.

[0125] 4. Collecting special offers:

[0126] The server uses RapidAPI, Zomato API, etc. to collect sales information from nearby stores based on the user's address information. This sales information is updated in real time and stored in a database.

[0127] 5. Menu generation and suggestions:

[0128] The server runs a menu generation algorithm implemented in a programming language such as Python based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. This algorithm generates an optimal menu, such as "grilled salmon + vegetable salad + soup," and creates a list of ingredients needed for it. The server sends this information to the device, which displays it to the user.

[0129] 6. Select your food purchase options and order:

[0130] The user checks the proposed menu through the app and selects how to purchase the necessary ingredients (e.g., ordering online). The user's selection is sent from the device to the server, which then contacts the partner delivery service (e.g., Amazon Fresh or Rakuten Seiyu Net Supermarket) and processes the order. The user receives the ingredients at home and cooks them according to the provided menu.

[0131] Specific examples

[0132] For example, if a user uploads a photo of "grilled salmon," the server will analyze it using the Google Cloud Vision API and get the results "salmon," "grilled," and "lemon." If the user enters their address in Minato Ward, Tokyo, a family structure of a couple with two children, and a monthly food budget of 30,000 yen, the server will use RapidAPI to obtain sales information from a nearby supermarket, collecting information such as "salmon on sale for 500 yen." Based on this information, the server will generate a menu of "grilled salmon + vegetable salad + consommé soup" and suggest it to the user. The user selects online ordering, and the server places an order for the ingredients with Amazon Fresh.

[0133] Prompt Sentence Examples

[0134] Simply take a photo of the food you've eaten and upload it to the app. Then, enter your address, family size, and monthly food budget. The system will suggest the best meal plan and give you the option to order the ingredients you need online.

[0135] As described above, the present invention enables users to make daily menu decisions and purchase ingredients more efficiently, and supports healthy and economical eating habits.

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] Step 1:

[0138] The user takes a photo of their meal using a smartphone or tablet device. The input is the photo data taken. The user uploads the photo data to the app, and the device sends it to the server. The output is the photo data sent to the server. Specifically, this involves taking a photo of "grilled salmon" using the device's camera function and pressing the "upload photo" button on the app.

[0139] Step 2:

[0140] The server temporarily stores the received photo data in local storage. The input is the photo data sent from the device. The server then sends this photo data to the image recognition API. The output is the analysis results. Specifically, the server calls the Google Cloud Vision API and obtains the analysis results "salmon," "grilled," and "lemon."

[0141] Step 3:

[0142] The server adds the obtained analysis results to the user's preference database and updates it. The input is the analysis results obtained from the image recognition API. The database is updated with new preference data. The output is the updated preference database. Specifically, this includes the action of adding the ingredient information "salmon" to the user's preference database.

[0143] Step 4:

[0144] The user enters their address, family composition, and monthly food budget via the app. The input is the address (e.g., Minato-ku, Tokyo), family composition (couple and two children), and monthly food budget (e.g., 30,000 yen) manually entered by the user. The device sends this information to the server. The output is the user information stored on the server. Specifically, this involves filling in the information in the app's input form and pressing the "Submit" button.

[0145] Step 5:

[0146] The server uses API to collect sale information from nearby stores based on the user's address information. The input is the user's address information and the store's API. The server obtains the sale information through the API and stores it in a database in real time. The output is the collected sale information. Specifically, this includes operations such as using RapidAPI to obtain information such as "Salmon is on sale for 500 yen."

[0147] Step 6:

[0148] The server runs an algorithm that generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. The input is a user database and a special sale information database. The algorithm generates a menu and a list of necessary ingredients. The output is the generated menu and ingredient list. Specifically, it includes the operation of generating a menu such as "grilled salmon + vegetable salad + soup" using an algorithm implemented in Python.

[0149] Step 7:

[0150] The server sends the generated menu and ingredient list to the terminal, which then displays it to the user. The input is the generated menu and ingredient list. The output is the menu and ingredient list displayed to the user. Specifically, this includes an action that displays the menu "Grilled salmon + vegetable salad + soup" on the app screen.

[0151] Step 8:

[0152] The user reviews the proposed menu through the app and selects the food options they wish to purchase. The input is the displayed menu and food list. The device sends the user's selections to the server. The output is the selections sent to the server. Specifically, this includes selecting "Order online" in the app and pressing the "Confirm selection" button.

[0153] Step 9:

[0154] Based on the selected order information, the server contacts the partner delivery service and processes the order. The input is the user's order information. The server sends the order details to the delivery service and arranges for the ingredients to be delivered to the user's address. The output is the progress of the delivery procedure. Specifically, this includes the actions of sending the order details to Amazon Fresh and receiving an order confirmation email.

[0155] This allows the system to streamline users' menu planning and ingredient purchasing, supporting healthy and economical eating habits.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Conventional menu suggestion systems do not adequately consider the user's preferences, budget, or family structure, and also lack the functionality to generate economical menus by utilizing sales information. As a result, users are unable to reduce the burden of planning their daily menus, and it is difficult to purchase ingredients efficiently and economically. Furthermore, they do not provide a means for users to quickly obtain the ingredients needed based on the suggested menus, so they lack the completeness of a comprehensive service.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; and means for linking with a delivery platform based on the ingredient list so that ingredients can be obtained quickly. This allows ingredients to be purchased efficiently and economically based on the proposed menu, and enables ingredients to be obtained more quickly.

[0161] "Means for users to take photos of their meals" refers to a function that allows users to easily take photos of their meals using devices such as smartphones or tablets and save them as digital data.

[0162] The "server means for receiving the photographs taken" is a server function for receiving, storing, and processing the photograph data sent by the user from the terminal via the network.

[0163] "Image recognition means for analyzing the received photograph and identifying the contents of the meal" is a function that uses image recognition technology to analyze the received photograph of the meal and identify the ingredients and dishes contained in the photograph.

[0164] The "means for updating the user's preference data based on the identified meal content" is a function that stores the meal content identified by image recognition in the user's preference database and keeps the user's preference information up to date.

[0165] "Means for users to input their address, family composition, and monthly food budget" refers to a function that allows users to input their address, number of family members and composition, and monthly food budget through an application.

[0166] The "means for saving the input information" is a function that saves information entered by the user, such as address, family composition, and monthly food budget, on a server for use in future plans and proposals.

[0167] "Means for collecting sales information from nearby stores" is a function that automatically collects sales information from nearby stores based on the user's address information via an API or the like.

[0168] "Means for generating optimal menus based on user preference data, family composition, food budget, nutritional balance, and sale information" is a function that automatically generates healthy and economical daily menus based on the user's individual information and collected sale information.

[0169] "Means for displaying the generated menu and related ingredient list to the user" is a function that displays the generated menu and the ingredient list required for it on a terminal used by the user (such as a smartphone or tablet).

[0170] "Means for linking with a delivery platform to quickly obtain ingredients based on the ingredient list" refers to a function that links with a delivery service to easily order ingredients online using the generated ingredient list.

[0171] This invention provides a system that allows users to easily take photos of their meals, analyzes the photos, and suggests optimal menus. The system considers the user's address, family composition, and monthly food budget, suggests optimal ingredients based on sales information from nearby stores, and, if necessary, provides the option to purchase the ingredients via delivery.

[0172] System configuration and hardware / software used

[0173] The system mainly consists of the following components:

[0174] 1. How to take photos of your food:

[0175] This function allows users to take photos of their meals using devices such as smartphones and tablets. The photo data is temporarily stored on the device.

[0176] 2. Server means:

[0177] This is a cloud server that receives and processes photographed photo data via the network. This server has built-in database and image analysis functions.

[0178] 3. Image Recognition Methods:

[0179] Using image analysis technology, the photos taken on the cloud are analyzed to identify ingredients and dishes, using TensorFlow and AWS Rekognition as the means for this.

[0180] 4. Preference data update method:

[0181] This function updates the user's preference database based on the food content identified by image recognition. This allows the system to accumulate user preferences and provide more accurate menu suggestions.

[0182] 5. User information input method:

[0183] It provides an interface for users to input their address, family composition, and monthly food budget. This information is sent to the server through the application and stored.

[0184] 6. How to collect special offers:

[0185] An API is used to gather sales information from nearby stores based on the user's address information. Google Places API or an API specifically for sales information can be used.

[0186] 7. Menu Generation Method:

[0187] Using a generative AI model, the system generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and collected sales information. This method uses a generative AI model such as GPT-3.

[0188] 8. Menu and ingredient list display method:

[0189] This function displays the generated menu and list of ingredients on the screen of a smartphone or tablet, allowing users to check the menu and prepare for ingredient purchases.

[0190] 9. Delivery Platform Collaboration Methods:

[0191] This function allows you to quickly obtain ingredients by linking with delivery services based on your ingredient list. This function uses delivery service APIs such as the Uber Eats API.

[0192] Specific examples

[0193] For example, if a user takes a photo of "grilled salmon" and uploads it to an app, the app sends the photo to a cloud server, which uses AWS Rekognition to analyze the photo and identify it as salmon. The server then updates the user's preference database with this information.

[0194] If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server will use the API to collect information about nearby sales based on the user's address. For example, if information is collected about a special sale on salmon at a specific store, the server will use a generative AI model (GPT-3) to generate an optimal menu that includes the special sale salmon. The generated menu is "grilled salmon + vegetable salad + soup," and a list of the necessary ingredients is also displayed.

[0195] Users can then order the ingredients online through a delivery platform and have them delivered quickly to their homes, allowing them to eat healthily and economically without any hassle.

[0196] Prompt Sentence Examples

[0197] As an example of a prompt, enter the following text:

[0198] Prompt: Suggest the best meal plan based on the given ingredients, family size, and budget. Image: "Grilled salmon", Family size: "Couple and two children", Monthly food budget: "30,000 yen", Local sale information: "Salmon is on sale"

[0199] This prompt allows the generative AI model to suggest optimal menus that take into account special sales information, allowing users to efficiently purchase ingredients based on these suggestions.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] The user takes a photo of the meal.

[0203] Users take photos of their meals using devices such as smartphones or tablets, and save them as high-resolution image files using the device's camera function.

[0204] Step 2:

[0205] The device sends the photograph to the server.

[0206] The user selects a photo they have taken and uploads it to the cloud server through the application. The input photo file (image data) is sent by the application to the server via the network. The server receives this data and temporarily stores it.

[0207] Step 3:

[0208] The server analyzes the received photos and identifies the contents of the meal.

[0209] The server analyzes the received image data using image analysis tools such as TensorFlow and AWS Rekognition. The input is image data, and the output is data on identified ingredients and dishes. Specific operations include running an algorithm to analyze the image and automatically recognizing objects and ingredients in the photo.

[0210] Step 4:

[0211] The server updates the user's preference data based on the identified meal content.

[0212] The server uses the identified meal details as input to update the user's preference database, adding the analysis results to the database and storing the user's preferences and past history.

[0213] Step 5:

[0214] The user enters their address, family composition, and monthly food budget.

[0215] A user uses the application to input information such as address, family composition, monthly food budget, etc. This input data is sent from the application to the server and stored on the server. Specific operations include data input via the user interface and communication processing.

[0216] Step 6:

[0217] The server collects sale information from nearby stores.

[0218] Using the user's address information as input, the server collects sales information from nearby stores through an API (e.g., Google Places API). The output is sales information data from the stores, which is automatically updated based on the user's address. Specific operations include API calls and data retrieval processes.

[0219] Step 7:

[0220] The server generates the optimal menu.

[0221] The server uses a generative AI model (e.g., GPT-3) to generate an optimal menu using the user's preference data, family composition, food budget, nutritional balance, and sale information as input. The output is a suggested menu and a list of ingredients. Specific operations include generating and inputting prompts to the generative AI model and analyzing the results.

[0222] Step 8:

[0223] The generated menu and ingredient list are displayed to the user.

[0224] The server sends the generated menu and ingredient list to the terminal, which displays it on the screen of the user's smartphone or tablet. Specific operations include sending data and displaying it using a GUI (graphical user interface).

[0225] Step 9:

[0226] The user orders ingredients from the delivery platform based on the ingredient list.

[0227] The user checks the displayed list of ingredients and orders them through the delivery platform. The input is the list of ingredients and order information, and the output is request information sent to the delivery platform. Specific operations include confirming the order and processing the order using the delivery API.

[0228] Step 10:

[0229] The server works with the delivery platform to arrange for the delivery of ingredients.

[0230] Based on the order information from the delivery platform, the server immediately arranges delivery. The output includes order confirmation and delivery arrangement information from the delivery platform. Specific operations include calling the delivery API and updating the order status.

[0231] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0232] In a system embodying this invention, a user takes photos of their meals, and the system analyzes the photos to suggest optimal menus. The system also suggests value-for-money ingredients based on sales information from nearby stores by inputting the user's address, family composition, and monthly food budget, and provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[0233] Overview of program processing

[0234] 1. Take and upload photos of your food

[0235] The user takes a photo of their meal using a smartphone or tablet.

[0236] The device uploads this photo data to the app, which then transfers it to the server.

[0237] 2. Photo analysis and updating of preference data

[0238] The server analyzes the received photos using image recognition technology.

[0239] Based on the analysis results, the server updates the user's preference data.

[0240] 3. Enter and save user information

[0241] Users enter their address, family composition, and monthly food budget through the app.

[0242] The device sends this information to the server, which stores it in a database.

[0243] 4. Emotion Recognition and Emotion Data Update

[0244] The server uses the photograph and input information to recognize the user's emotions using an emotion engine.

[0245] The emotion engine updates emotion data based on user input and feedback, learning emotional trends over a long period of time.

[0246] 5. Collecting information on special offers

[0247] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[0248] 6. Menu generation and suggestions

[0249] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data.

[0250] The menu contents are sent to the terminal, which displays them to the user.

[0251] 7. Select your food purchase options and place your order

[0252] The user checks the displayed menu and selects how to purchase the necessary ingredients.

[0253] The device sends the selection to the server.

[0254] The server contacts the affiliated delivery service based on the order information and processes the delivery of the ingredients.

[0255] 8. Specific Examples

[0256] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user enters a family of a couple with two children and a monthly food budget of 50,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[0257] If the emotion engine detects that the user has been feeling stressed recently, it will adjust the menu to include ingredients with relaxing properties (e.g., herbs or teas). These suggestions are presented to the user along with an ingredient list that reflects sales. The user also has the option to order the items online and have them delivered to their home.

[0258] In this way, the system will help users make menu decisions and purchase ingredients more efficiently, and will also be able to suggest dietary habits that take the user's emotions into consideration, supporting a healthier and more satisfying lifestyle.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[0262] Step 2:

[0263] The device uploads the photo to the app. The device sends the photo it has taken to the app, which then transfers it to the server.

[0264] Step 3:

[0265] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[0266] Step 4:

[0267] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "sushi," "pasta," etc.).

[0268] Step 5:

[0269] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[0270] Step 6:

[0271] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[0272] Step 7:

[0273] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[0274] Step 8:

[0275] The server recognizes emotions. The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information.

[0276] Step 9:

[0277] The server updates the emotion data. The emotion engine updates the emotion database based on user input and feedback, learning the user's emotional tendencies over a long period of time.

[0278] Step 10:

[0279] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores it in a database.

[0280] Step 11:

[0281] The server generates a menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. The server also adjusts the menu based on the emotional data, taking into account the user's emotional state.

[0282] Step 12:

[0283] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[0284] Step 13:

[0285] The user selects an ingredient purchasing option. The user reviews the suggested meal plan and selects how to purchase the ingredients needed (e.g., online order, delivery, or select a location).

[0286] Step 14:

[0287] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[0288] Step 15:

[0289] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[0290] Step 16:

[0291] The user receives the ingredients and prepares the meal according to the provided menu.

[0292] Example 2

[0293] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] Conventional menu suggestion systems have had problems in that it is difficult to propose menus that take into account the user's preferences and emotions, and it is also difficult to efficiently suggest ingredients using sale information and carry out purchasing procedures. Furthermore, they lack the functionality to adjust menus based on the user's emotions, which limits their ability to support an efficient and satisfying dietary lifestyle.

[0295] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0296] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient lists to the user; emotion engine means for recognizing the user's emotions using the photo and the input information; and means for adjusting the menu based on the user's emotion data. This makes it possible to propose an optimal menu that comprehensively takes into account the user's preferences, emotions, and sales information, thereby streamlining the purchase of necessary ingredients.

[0297] "User" refers to an individual who uses the system to take photos of their meals and receives menu suggestions and support for purchasing ingredients.

[0298] "Photos of meals" refer to image data of meals taken by a user using a camera function.

[0299] "Server means" refers to a device or system that receives photos of meals uploaded by users and then analyzes, stores, and processes the data.

[0300] "Image recognition means" refers to the technology or algorithm used to analyze received photos of food and identify their contents.

[0301] "Preference data" refers to data that indicates a user's eating preferences and tendencies.

[0302] "Address, family composition, monthly food budget" refers to information about an individual's residence, the number and composition of family members, and monthly food budget information that a user inputs into the system.

[0303] "Storage means" refers to the technology or device for storing input information and analysis results in a database, etc.

[0304] "Special sale information" refers to information about sales at special prices offered by nearby retailers.

[0305] "Menu" refers to suggestions that show the combinations and contents of meals that a user should eat.

[0306] "Emotion engine means" refers to a technology or device for recognizing a user's emotions and performing processing based on that data.

[0307] "Purchase options" refers to options that allow a user to purchase ingredients needed for a suggested menu.

[0308] "Home delivery service" refers to a service that delivers purchased ingredients to a location specified by the user.

[0309] The system embodying this invention allows users to take photos of their meals, analyzes the photos, and proposes optimal menus. By inputting the user's address, family composition, and monthly food budget, the system suggests value-for-money ingredients taking into account sales information from nearby stores, and also provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[0310] Hardware and Software Configuration

[0311] This system uses the following hardware and software:

[0312] Smartphones and tablets: devices that allow users to take photos of their meals and enter and display information through an application.

[0313] Cloud server: A server for analyzing, storing, and processing received photo data and input information. The following software is also used:

[0314] Image recognition libraries (e.g., TensorFlow, OpenCV): Analyze photos of food and identify the contents of the food.

[0315] Database management system (e.g., MySQL, PostgreSQL): Stores user input information and analysis results.

[0316] Sentiment analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API): Recognizes user emotions and updates / stores emotion data.

[0317] Special Offer Information Collection API: Collects special offer information from nearby retailers (using retailer APIs and scraping technology).

[0318] System Operation

[0319] 1. Take and upload a photo of your meal:

[0320] Users take photos of their meals using the camera on their smartphone or tablet device and upload them to a dedicated app, which then sends the photo data to a cloud server.

[0321] 2. Photo analysis and preference data update:

[0322] The server analyzes the received photo data using image recognition technology, and based on the analysis results, the user's preference data is recorded and updated in the database.

[0323] 3. Enter and save your user information:

[0324] Users enter their address, family composition, and monthly food budget through the app, and the device sends this information to the server, which stores it in a database.

[0325] 4. Emotion Recognition and Emotion Data Update:

[0326] The server uses an emotion engine to recognize the user's emotions based on the photos and input information received, and the emotion data is recorded and updated in the database.

[0327] 5. Collecting special offers:

[0328] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[0329] 6. Menu generation and suggestions:

[0330] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[0331] 7. Select your food purchase options and order:

[0332] The user checks the displayed menu and selects how to purchase the necessary ingredients, and the terminal sends the details to the server, which then contacts an affiliated delivery service based on the order information and processes the delivery of the ingredients.

[0333] Specific examples

[0334] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user inputs a family structure of a couple with two children and a monthly food budget of ¥50,000, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server might suggest a menu of "sushi + miso soup + salad." If the emotion engine detects that the user has recently been feeling stressed, the server might adjust the menu to include ingredients with a relaxing effect (e.g., herbs or tea). These suggestions are presented to the user along with a list of ingredients that reflect the sale information, and the user can also select the option to order ingredients online and have them delivered to their home.

[0335] Prompt Sentence Examples

[0336] "I want you to upload photos of your meals and get suggestions for the perfect menu."

[0337] "Our family consists of a couple and two children, and our monthly food budget is 50,000 yen. We live in central Tokyo. Please tell me what menu we'd like to have based on this information."

[0338] "I've been feeling stressed lately. Can you suggest a meal that includes relaxing ingredients?"

[0339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0340] Step 1:

[0341] The user takes a photo of the meal using the camera on their smartphone or tablet.

[0342] Input: The food item to be photographed.

[0343] What it does: Open the camera app, focus on your food and press the shutter button.

[0344] Output: Photo data of the meal.

[0345] Step 2:

[0346] The device uploads the captured photo data to a dedicated app and sends this data to a cloud server.

[0347] Input: The captured photo data.

[0348] How it works: Retrieves photo data from the device's internal storage and sends a POST request to the server via the REST API.

[0349] Output: Photo data uploaded to the server.

[0350] Step 3:

[0351] The server analyzes the received photo data using image recognition technology to identify the ingredients and the name of the dish.

[0352] Input: Photo data uploaded to the server.

[0353] How it works: Using image recognition libraries (e.g. TensorFlow, OpenCV), it inputs photo data and extracts ingredients and dish names.

[0354] Output: Extracted ingredients and dish names.

[0355] Step 4:

[0356] The server updates the user's preference data based on the analysis results.

[0357] Input: Extracted ingredient and dish name data.

[0358] What it does: Converts the analysis results into SQL queries and inserts or updates the preference data into a database.

[0359] Output: Updated preference data.

[0360] Step 5:

[0361] Users enter their address, family composition, and monthly food budget via the app, and the device sends this information to the server, which stores it in a database.

[0362] Input: address, family composition, monthly food budget.

[0363] How it works: A user enters information into a form in your app and presses the submit button. The device sends the form data as a POST request, and the server executes an SQL query to store it in the database.

[0364] Output: Address, family composition, and monthly food budget information stored in the database.

[0365] Step 6:

[0366] The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information, and updates the emotion data.

[0367] Input: Uploaded photo data and input information.

[0368] How it works: Sends photo data or text data to an emotion analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API), retrieves the emotion analysis results, and inserts or updates the results into a database as an SQL query.

[0369] Output: Updated emotion data.

[0370] Step 7:

[0371] Based on the user's address information, the server uses APIs and web scraping technology to collect special sale information from nearby stores and stores it in a database.

[0372] Input: User's address information.

[0373] How it works: The server accesses the retailer's API to retrieve sales information in JSON format, and optionally extracts information using web scraping techniques.

[0374] Output: Sale information stored in a database.

[0375] Step 8:

[0376] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[0377] Input: preference data, family composition, food budget, nutritional balance, sale information, emotional data.

[0378] How it works: It runs an algorithm that retrieves data by performing multiple database queries, then uses that data to calculate the optimal menu. The resulting menu data is sent in JSON format to the device, which parses it and displays it in the UI.

[0379] Output: Menu information displayed to the user.

[0380] Step 9:

[0381] The user checks the displayed menu and selects how to purchase the necessary ingredients. The terminal then sends the details to the server, which then contacts an affiliated delivery service based on the order information and arranges for the ingredients to be delivered.

[0382] Input: User selection of purchase option.

[0383] How it works: The user selects a purchase method using buttons or checkboxes within the app and presses the "Purchase" button. The device posts the selection to the server as a JSON-formatted request, and the server sends the order information to the delivery service's API and begins the delivery process.

[0384] Output: Order information sent to the delivery service.

[0385] (Application example 2)

[0386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] In today's busy lifestyles, it is difficult for users to select appropriate meals, maintain nutritional balance, and make budget-friendly purchases. It is even more difficult to select meals that reflect the user's emotional state. To address these issues, an effective system is needed that proposes optimal menus that take into account the user's preferences and emotional state, and supports the user in making those purchases.

[0388] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; means for adjusting the menu based on the user's emotions using an emotion engine that recognizes the user's emotions; and means for inputting prompt sentences to recognize emotions and reflect them in the menu generation. This enables optimal meal suggestions based on the user's preferences and emotional state and efficient ingredient purchasing.

[0389] "Means for users to take photos of their meals" refers to devices or applications that allow users to take photos of their meals and save and transmit them as data.

[0390] "Server means" means a system for receiving, processing, storing and transmitting data sent by a user.

[0391] "Image recognition means" refers to algorithms or software that analyzes received photo data and identifies the contents of the meal.

[0392] The "means for updating preference data" is a system that records the user's food preferences based on the analyzed meal contents and updates the database.

[0393] "Means for entering address, family composition, and monthly food budget" refers to an interface that allows users to enter and send this information using an application or device.

[0394] "Means for storing information" refers to a mechanism for storing input information and analysis results in a database.

[0395] The "means for collecting sale information" is a system that acquires and stores sale information provided by nearby stores.

[0396] The "means for generating optimal menus" refers to algorithms or programs that suggest optimal meal combinations for users based on their preference data, family composition, food budget, nutritional balance, and special sale information.

[0397] The "means for displaying the generated menu and related ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's device.

[0398] The "means for adjusting menus based on the user's emotions using an emotion engine" is a system that uses an engine that analyzes the user's emotions to make meal suggestions based on the user's emotional state.

[0399] "Means for recognizing emotions and reflecting them in menu generation by inputting prompt sentences" is a mechanism that analyzes emotions based on prompt sentences input by the user in words or sentences and reflects them in menu suggestions.

[0400] The system for realizing this invention starts with the user taking a photo of their meal using a smartphone or tablet device and uploading the data to a server. This device has an application with photo and upload functions installed.

[0401] The server uses image recognition technology such as the TensorFlow model to analyze the received photo data. The analysis results are reflected in the user's preference database. User information is entered and saved via the application, with address, family composition, and food budget entered, and the information is saved in the database by the server.

[0402] Emotion recognition is achieved by analyzing user-provided text using natural language processing, for example by using the Hugging Face Transformers library to analyze emotion prompts entered by the user in the application, and the results of this analysis are stored in a user emotion database.

[0403] The server then uses the user's address and other required information to collect special offers from nearby retailers and delivery services via API, which is also stored in a database.

[0404] The server uses an algorithm to generate the optimal menu, based on a comprehensive analysis of the user's preferences, family composition, food budget, nutritional balance, sales information, and emotional data. The generated menu is then displayed on the user's device along with a list of related ingredients.

[0405] Emotion recognition and menu adjustment based on user-provided prompts involves emotion analysis, where the user inputs a specific prompt into the application, such as "I've been feeling tired lately, so please suggest a meal that will help me relax." Based on this information, the server generates and suggests a menu using ingredients that have a relaxing effect.

[0406] For example, if a user takes a photo of "sushi" and inputs their family composition (couple, two children) and monthly food budget of 50,000 yen, the server will use this information to gather sales information from nearby stores. For example, if there is a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[0407] Additionally, if a user inputs "I've been feeling stressed lately," the emotion engine will adjust the menu to include relaxing herbs and teas. These suggestions are provided to the user along with an ingredient list that reflects special offers, and the user can order them online and have them delivered to their home.

[0408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0409] Step 1:

[0410] The user takes a photo of their meal using a smartphone or tablet, which temporarily stores the photo data within the app and prompts the user to confirm upload.

[0411] Input: Food photo

[0412] Output: Temporarily saved photo data

[0413] Step 2:

[0414] Users upload photos of their meals via the application to the server, which receives the photos and stores them in a database.

[0415] Input: Temporarily saved photo data

[0416] Output: Photo data stored on the server

[0417] Step 3:

[0418] The server analyzes the received photo data using image recognition techniques, such as TensorFlow, to identify the contents of the meal.

[0419] Input: Photo data stored on the server

[0420] Output: Identified meal content data

[0421] Step 4:

[0422] The server updates the user's preference data based on the meal content data. Specifically, it tags the meal content and compares it with previous preference data to reflect the user's preference trends in the database.

[0423] Input: Identified meal content data

[0424] Output: Updated preference data

[0425] Step 5:

[0426] The user enters their address, family composition, and monthly food budget through the application, and the device sends this information to the server, which stores it in a database.

[0427] Input: Address, family composition, monthly food budget

[0428] Output: User information data stored on the server

[0429] Step 6:

[0430] The server uses APIs to collect special sale information from nearby stores and delivery services, and the collected special sale information is stored in a database.

[0431] Input: User Address

[0432] Output: Collected sale information data

[0433] Step 7:

[0434] The user inputs a prompt sentence into the application. For example, a prompt sentence such as "I've been feeling tired lately, so please suggest a meal that will help me relax." The device then sends this prompt sentence to the server.

[0435] Input: prompt statement

[0436] Output: Prompt data sent to the server

[0437] Step 8:

[0438] The server analyzes the prompt sentence using natural language processing to recognize the user's emotions, and generates emotion data using libraries such as Hugging Face's Transformers library.

[0439] Input: Prompt data sent to the server

[0440] Output: Generated emotion data

[0441] Step 9:

[0442] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. Using a generative AI model, it suggests optimal meal combinations.

[0443] Input: User preference data, family structure, food budget, nutritional balance, sale information, emotional data

[0444] Output: Generated menu data

[0445] Step 10:

[0446] The generated menu and related ingredient list are displayed on the user's terminal, allowing the user to confirm the menu and check the ingredient list.

[0447] Input: Generated menu data

[0448] Output: Menu and ingredient list data displayed on the user's device

[0449] Step 11:

[0450] The user selects an option to purchase based on the displayed ingredient list, and the selected option is transmitted from the terminal to the server.

[0451] Input: User's choice option

[0452] Output: Purchase option data sent to the server

[0453] Step 12:

[0454] The server processes the order and, based on the purchase options selected, contacts the partner delivery service and arranges for the delivery of the ingredients.

[0455] Input: Purchase option data sent to the server

[0456] Output: Order data sent to the delivery service

[0457] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0459] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0460] [Second embodiment]

[0461] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0462] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0463] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0464] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0465] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0466] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0467] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0468] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0471] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0472] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0473] The system for implementing this invention allows users to simply take photos of their meals, and the system analyzes the photos to suggest optimal menus. Furthermore, by inputting the user's address, family composition, and monthly food budget, the system suggests cost-effective ingredients taking into account sales information from nearby stores, and also provides the option to purchase the ingredients if necessary.

[0474] Overview of program processing

[0475] 1. Take and upload photos of your food

[0476] Users take photos of their meals with their smartphones or tablets and upload them to the app.

[0477] The terminal transmits this photo data to the server.

[0478] 2. Photo analysis and updating of preference data

[0479] The server analyzes the received photos using image recognition technology.

[0480] The server identifies the type of meal shown in the photo (e.g., pasta, salad, etc.) and stores this in the user's preference database.

[0481] 3. Enter your user information

[0482] Users enter their address, family composition, and monthly food budget through the app.

[0483] The terminal transmits this information to the server.

[0484] The server stores the received information in a database.

[0485] 4. Collecting information on special offers

[0486] The server uses the API to collect sale information from nearby stores based on the user's address.

[0487] Special sale information is updated in real time and stored in a database.

[0488] 5. Menu generation and suggestions

[0489] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0490] The proposed menu and a list of ingredients required for it are sent from the server to the terminal.

[0491] The terminal displays this to the user.

[0492] 6. Select your food purchase options and place your order

[0493] The user reviews the proposed menu and selects how to purchase the ingredients needed (e.g., order online for delivery).

[0494] The device sends the selection to the server.

[0495] Based on the order information, the server contacts the affiliated delivery service and arranges for the ingredients to be delivered.

[0496] The user receives the ingredients at home and cooks according to the provided menu.

[0497] Specific examples

[0498] For example, if a user uploads a photo of "grilled salmon," the server analyzes the photo and learns that salmon and grilled dishes are popular. If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a supermarket is having a sale on salmon, the server will suggest a menu using the sale items: "grilled salmon + vegetable salad + soup." This suggestion includes a list of ingredients that are on sale, and the user can also select the option to order the ingredients online and have them delivered to their home.

[0499] In this way, the system streamlines the daily task of deciding on menus and purchasing ingredients, which can be a heavy burden for users, and supports healthy and economical eating habits.

[0500] The processing flow will be explained below.

[0501] Step 1:

[0502] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[0503] Step 2:

[0504] The device uploads photos to the app. The device sends the photos it has taken to the app, which then transfers the photo data to the server.

[0505] Step 3:

[0506] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[0507] Step 4:

[0508] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "grilled salmon," "salad," etc.).

[0509] Step 5:

[0510] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[0511] Step 6:

[0512] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[0513] Step 7:

[0514] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[0515] Step 8:

[0516] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores the sale information in a database.

[0517] Step 9:

[0518] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0519] Step 10:

[0520] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[0521] Step 11:

[0522] The user selects a grocery purchase option. The user selects their preferred method of grocery purchase from several options within the app (e.g., online ordering, delivery, select location).

[0523] Step 12:

[0524] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[0525] Step 13:

[0526] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[0527] Step 14:

[0528] The user receives the ingredients and prepares the meal according to the provided menu.

[0529] Example 1

[0530] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] In many modern households, planning daily menus and efficiently purchasing ingredients according to the family's preferences and budget are a significant burden. Many people also find it difficult to shop economically by utilizing sales information. The purpose of this invention is to solve these problems and provide a system that supports an efficient and economical dietary lifestyle.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0533] In this invention, the server includes: means for a user to take photos of meals; computer means for receiving the photos; image recognition means for analyzing the received photos and identifying the contents of the meals; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; database means for storing the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and associated ingredient list to the user; means for presenting the user with options for purchasing ingredients based on the displayed menu and ingredient list; and means for processing ingredient orders based on the options. This allows users to efficiently decide on their daily menu and purchase the necessary ingredients economically.

[0534] A "user" is someone who uses the system to take photos of their meals and input information such as their personal information and food budget.

[0535] "Means for taking photos of food" refers to functions or applications for taking photos of food using electronic devices such as smartphones or tablet devices.

[0536] "Computer means" refers to a server or computer system that receives the captured photograph data and performs subsequent analysis and data storage.

[0537] "Image recognition means" refers to software or algorithms that analyze received photos and identify the ingredients or dishes depicted.

[0538] "Preference data" refers to a database that records a user's food preferences and tendencies.

[0539] "Means for inputting address, family composition, and monthly food budget" is a function that provides an interface for the user to input their own address, family composition, and monthly food budget.

[0540] The "database means" is a system for storing and managing input information and analysis results as structured data.

[0541] "Sales information" refers to sale information and product price information obtained from nearby retailers.

[0542] "Means of collecting sales information" refers to APIs and web scraping technologies for collecting real-time sales information and product price information from nearby retailers.

[0543] "Means for generating menus" refers to algorithms or programs that generate optimal menus based on collected preference data, family composition, food budget, nutritional balance, and sales information.

[0544] The "means for displaying the ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's terminal.

[0545] The "means for presenting options for purchasing ingredients" is a function that presents options for the user to order ingredients online.

[0546] The "means for processing an order" refers to a system for communicating ingredient order information to a related service based on a user's selection and processing the order.

[0547] This invention is a system that helps users efficiently decide on daily menus and purchase ingredients economically. This system allows users to take photos of meals using a smartphone or tablet device, analyzes the photos, updates the user's preference data, and provides menu suggestions and ingredient purchasing support that take into account the user's address, family composition, and monthly food budget entered by the user.

[0548] Specifically, the following hardware and software are used.

[0549] 1. Take and upload a photo of your meal:

[0550] Users take photos of their meals using their smartphones or tablets and upload them to a dedicated app, which then sends the photos to a server.

[0551] 2. Photo analysis and preference data update:

[0552] The server temporarily stores the received photos in local storage. The server then analyzes the photos using image recognition technologies such as Google Cloud Vision API and Amazon Rekognition. The analysis results are obtained as dish names and ingredient information, such as "grilled salmon" or "pasta." The server updates the user's preference database based on these analysis results.

[0553] 3. Enter your user information:

[0554] Users enter their address, family composition, and monthly food budget through the app, and the device sends the information to the server, which then stores it in a database.

[0555] 4. Collecting special offers:

[0556] The server uses RapidAPI, Zomato API, etc. to collect sales information from nearby stores based on the user's address information. This sales information is updated in real time and stored in a database.

[0557] 5. Menu generation and suggestions:

[0558] The server runs a menu generation algorithm implemented in a programming language such as Python based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. This algorithm generates an optimal menu, such as "grilled salmon + vegetable salad + soup," and creates a list of ingredients needed for it. The server sends this information to the device, which displays it to the user.

[0559] 6. Select your food purchase options and order:

[0560] The user checks the proposed menu through the app and selects how to purchase the necessary ingredients (e.g., ordering online). The user's selection is sent from the device to the server, which then contacts the partner delivery service (e.g., Amazon Fresh or Rakuten Seiyu Net Supermarket) and processes the order. The user receives the ingredients at home and cooks them according to the provided menu.

[0561] Specific examples

[0562] For example, if a user uploads a photo of "grilled salmon," the server will analyze it using the Google Cloud Vision API and get the results "salmon," "grilled," and "lemon." If the user enters their address in Minato Ward, Tokyo, a family structure of a couple with two children, and a monthly food budget of 30,000 yen, the server will use RapidAPI to obtain sales information from a nearby supermarket, collecting information such as "salmon on sale for 500 yen." Based on this information, the server will generate a menu of "grilled salmon + vegetable salad + consommé soup" and suggest it to the user. The user selects online ordering, and the server places an order for the ingredients with Amazon Fresh.

[0563] Prompt Sentence Examples

[0564] Simply take a photo of the food you've eaten and upload it to the app. Then, enter your address, family size, and monthly food budget. The system will suggest the best meal plan and give you the option to order the ingredients you need online.

[0565] As described above, the present invention enables users to make daily menu decisions and purchase ingredients more efficiently, and supports healthy and economical eating habits.

[0566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0567] Step 1:

[0568] The user takes a photo of their meal using a smartphone or tablet device. The input is the photo data taken. The user uploads the photo data to the app, and the device sends it to the server. The output is the photo data sent to the server. Specifically, this involves taking a photo of "grilled salmon" using the device's camera function and pressing the "upload photo" button on the app.

[0569] Step 2:

[0570] The server temporarily stores the received photo data in local storage. The input is the photo data sent from the device. The server then sends this photo data to the image recognition API. The output is the analysis results. Specifically, the server calls the Google Cloud Vision API and obtains the analysis results "salmon," "grilled," and "lemon."

[0571] Step 3:

[0572] The server adds the obtained analysis results to the user's preference database and updates it. The input is the analysis results obtained from the image recognition API. The database is updated with new preference data. The output is the updated preference database. Specifically, this includes the action of adding the ingredient information "salmon" to the user's preference database.

[0573] Step 4:

[0574] The user enters their address, family composition, and monthly food budget via the app. The input is the address (e.g., Minato-ku, Tokyo), family composition (couple and two children), and monthly food budget (e.g., 30,000 yen) manually entered by the user. The device sends this information to the server. The output is the user information stored on the server. Specifically, this involves filling in the information in the app's input form and pressing the "Submit" button.

[0575] Step 5:

[0576] The server uses API to collect sale information from nearby stores based on the user's address information. The input is the user's address information and the store's API. The server obtains the sale information through the API and stores it in a database in real time. The output is the collected sale information. Specifically, this includes operations such as using RapidAPI to obtain information such as "Salmon is on sale for 500 yen."

[0577] Step 6:

[0578] The server runs an algorithm that generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. The input is a user database and a special sale information database. The algorithm generates a menu and a list of necessary ingredients. The output is the generated menu and ingredient list. Specifically, it includes the operation of generating a menu such as "grilled salmon + vegetable salad + soup" using an algorithm implemented in Python.

[0579] Step 7:

[0580] The server sends the generated menu and ingredient list to the terminal, which then displays it to the user. The input is the generated menu and ingredient list. The output is the menu and ingredient list displayed to the user. Specifically, this includes an action that displays the menu "Grilled salmon + vegetable salad + soup" on the app screen.

[0581] Step 8:

[0582] The user reviews the proposed menu through the app and selects the food options they wish to purchase. The input is the displayed menu and food list. The device sends the user's selections to the server. The output is the selections sent to the server. Specifically, this includes selecting "Order online" in the app and pressing the "Confirm selection" button.

[0583] Step 9:

[0584] Based on the selected order information, the server contacts the partner delivery service and processes the order. The input is the user's order information. The server sends the order details to the delivery service and arranges for the ingredients to be delivered to the user's address. The output is the progress of the delivery procedure. Specifically, this includes the actions of sending the order details to Amazon Fresh and receiving an order confirmation email.

[0585] This allows the system to streamline users' menu planning and ingredient purchasing, supporting healthy and economical eating habits.

[0586] (Application example 1)

[0587] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] Conventional menu suggestion systems do not adequately consider the user's preferences, budget, or family structure, and also lack the functionality to generate economical menus by utilizing sales information. As a result, users are unable to reduce the burden of planning their daily menus, and it is difficult to purchase ingredients efficiently and economically. Furthermore, they do not provide a means for users to quickly obtain the ingredients needed based on the suggested menus, so they lack the completeness of a comprehensive service.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0590] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; and means for linking with a delivery platform based on the ingredient list so that ingredients can be obtained quickly. This allows ingredients to be purchased efficiently and economically based on the proposed menu, and enables ingredients to be obtained more quickly.

[0591] "Means for users to take photos of their meals" refers to a function that allows users to easily take photos of their meals using devices such as smartphones or tablets and save them as digital data.

[0592] The "server means for receiving the photographs taken" is a server function for receiving, storing, and processing the photograph data sent by the user from the terminal via the network.

[0593] "Image recognition means for analyzing the received photograph and identifying the contents of the meal" is a function that uses image recognition technology to analyze the received photograph of the meal and identify the ingredients and dishes contained in the photograph.

[0594] The "means for updating the user's preference data based on the identified meal content" is a function that stores the meal content identified by image recognition in the user's preference database and keeps the user's preference information up to date.

[0595] "Means for users to input their address, family composition, and monthly food budget" refers to a function that allows users to input their address, number of family members and composition, and monthly food budget through an application.

[0596] The "means for saving the input information" is a function that saves information entered by the user, such as address, family composition, and monthly food budget, on a server for use in future plans and proposals.

[0597] "Means for collecting sales information from nearby stores" is a function that automatically collects sales information from nearby stores based on the user's address information via an API or the like.

[0598] "Means for generating optimal menus based on user preference data, family composition, food budget, nutritional balance, and sale information" is a function that automatically generates healthy and economical daily menus based on the user's individual information and collected sale information.

[0599] "Means for displaying the generated menu and related ingredient list to the user" is a function that displays the generated menu and the ingredient list required for it on a terminal used by the user (such as a smartphone or tablet).

[0600] "Means for linking with a delivery platform to quickly obtain ingredients based on the ingredient list" refers to a function that links with a delivery service to easily order ingredients online using the generated ingredient list.

[0601] This invention provides a system that allows users to easily take photos of their meals, analyzes the photos, and suggests optimal menus. The system considers the user's address, family composition, and monthly food budget, suggests optimal ingredients based on sales information from nearby stores, and, if necessary, provides the option to purchase the ingredients via delivery.

[0602] System configuration and hardware / software used

[0603] The system mainly consists of the following components:

[0604] 1. How to take photos of your food:

[0605] This function allows users to take photos of their meals using devices such as smartphones and tablets. The photo data is temporarily stored on the device.

[0606] 2. Server means:

[0607] This is a cloud server that receives and processes photographed photo data via the network. This server has built-in database and image analysis functions.

[0608] 3. Image Recognition Methods:

[0609] Using image analysis technology, the photos taken on the cloud are analyzed to identify ingredients and dishes, using TensorFlow and AWS Rekognition as the means for this.

[0610] 4. Preference data update method:

[0611] This function updates the user's preference database based on the food content identified by image recognition. This allows the system to accumulate user preferences and provide more accurate menu suggestions.

[0612] 5. User information input method:

[0613] It provides an interface for users to input their address, family composition, and monthly food budget. This information is sent to the server through the application and stored.

[0614] 6. How to collect special offers:

[0615] An API is used to gather sales information from nearby stores based on the user's address information. Google Places API or an API specifically for sales information can be used.

[0616] 7. Menu Generation Method:

[0617] Using a generative AI model, the system generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and collected sales information. This method uses a generative AI model such as GPT-3.

[0618] 8. Menu and ingredient list display method:

[0619] This function displays the generated menu and list of ingredients on the screen of a smartphone or tablet, allowing users to check the menu and prepare for ingredient purchases.

[0620] 9. Delivery Platform Collaboration Methods:

[0621] This function allows you to quickly obtain ingredients by linking with delivery services based on your ingredient list. This function uses delivery service APIs such as the Uber Eats API.

[0622] Specific examples

[0623] For example, if a user takes a photo of "grilled salmon" and uploads it to an app, the app sends the photo to a cloud server, which uses AWS Rekognition to analyze the photo and identify it as salmon. The server then updates the user's preference database with this information.

[0624] If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server will use the API to collect information about nearby sales based on the user's address. For example, if information is collected about a special sale on salmon at a specific store, the server will use a generative AI model (GPT-3) to generate an optimal menu that includes the special sale salmon. The generated menu is "grilled salmon + vegetable salad + soup," and a list of the necessary ingredients is also displayed.

[0625] Users can then order the ingredients online through a delivery platform and have them delivered quickly to their homes, allowing them to eat healthily and economically without any hassle.

[0626] Prompt Sentence Examples

[0627] As an example of a prompt, enter the following text:

[0628] Prompt: Suggest the best meal plan based on the given ingredients, family size, and budget. Image: "Grilled salmon", Family size: "Couple and two children", Monthly food budget: "30,000 yen", Local sale information: "Salmon is on sale"

[0629] This prompt allows the generative AI model to suggest optimal menus that take into account special sales information, allowing users to efficiently purchase ingredients based on these suggestions.

[0630] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0631] Step 1:

[0632] The user takes a photo of the meal.

[0633] Users take photos of their meals using devices such as smartphones or tablets, and save them as high-resolution image files using the device's camera function.

[0634] Step 2:

[0635] The device sends the photograph to the server.

[0636] The user selects a photo they have taken and uploads it to the cloud server through the application. The input photo file (image data) is sent by the application to the server via the network. The server receives this data and temporarily stores it.

[0637] Step 3:

[0638] The server analyzes the received photos and identifies the contents of the meal.

[0639] The server analyzes the received image data using image analysis tools such as TensorFlow and AWS Rekognition. The input is image data, and the output is data on identified ingredients and dishes. Specific operations include running an algorithm to analyze the image and automatically recognizing objects and ingredients in the photo.

[0640] Step 4:

[0641] The server updates the user's preference data based on the identified meal content.

[0642] The server uses the identified meal details as input to update the user's preference database, adding the analysis results to the database and storing the user's preferences and past history.

[0643] Step 5:

[0644] The user enters their address, family composition, and monthly food budget.

[0645] A user uses the application to input information such as address, family composition, monthly food budget, etc. This input data is sent from the application to the server and stored on the server. Specific operations include data input via the user interface and communication processing.

[0646] Step 6:

[0647] The server collects sale information from nearby stores.

[0648] Using the user's address information as input, the server collects sales information from nearby stores through an API (e.g., Google Places API). The output is sales information data from the stores, which is automatically updated based on the user's address. Specific operations include API calls and data retrieval processes.

[0649] Step 7:

[0650] The server generates the optimal menu.

[0651] The server uses a generative AI model (e.g., GPT-3) to generate an optimal menu using the user's preference data, family composition, food budget, nutritional balance, and sale information as input. The output is a suggested menu and a list of ingredients. Specific operations include generating and inputting prompts to the generative AI model and analyzing the results.

[0652] Step 8:

[0653] The generated menu and ingredient list are displayed to the user.

[0654] The server sends the generated menu and ingredient list to the terminal, which displays it on the screen of the user's smartphone or tablet. Specific operations include sending data and displaying it using a GUI (graphical user interface).

[0655] Step 9:

[0656] The user orders ingredients from the delivery platform based on the ingredient list.

[0657] The user checks the displayed list of ingredients and orders them through the delivery platform. The input is the list of ingredients and order information, and the output is request information sent to the delivery platform. Specific operations include confirming the order and processing the order using the delivery API.

[0658] Step 10:

[0659] The server works with the delivery platform to arrange for the delivery of ingredients.

[0660] Based on the order information from the delivery platform, the server immediately arranges delivery. The output includes order confirmation and delivery arrangement information from the delivery platform. Specific operations include calling the delivery API and updating the order status.

[0661] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0662] In a system embodying this invention, a user takes photos of their meals, and the system analyzes the photos to suggest optimal menus. The system also suggests value-for-money ingredients based on sales information from nearby stores by inputting the user's address, family composition, and monthly food budget, and provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[0663] Overview of program processing

[0664] 1. Take and upload photos of your food

[0665] The user takes a photo of their meal using a smartphone or tablet.

[0666] The device uploads this photo data to the app, which then transfers it to the server.

[0667] 2. Photo analysis and updating of preference data

[0668] The server analyzes the received photos using image recognition technology.

[0669] Based on the analysis results, the server updates the user's preference data.

[0670] 3. Enter and save user information

[0671] Users enter their address, family composition, and monthly food budget through the app.

[0672] The device sends this information to the server, which stores it in a database.

[0673] 4. Emotion Recognition and Emotion Data Update

[0674] The server uses the photograph and input information to recognize the user's emotions using an emotion engine.

[0675] The emotion engine updates emotion data based on user input and feedback, learning emotional trends over a long period of time.

[0676] 5. Collecting information on special offers

[0677] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[0678] 6. Menu generation and suggestions

[0679] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data.

[0680] The menu contents are sent to the terminal, which displays them to the user.

[0681] 7. Select your food purchase options and place your order

[0682] The user checks the displayed menu and selects how to purchase the necessary ingredients.

[0683] The device sends the selection to the server.

[0684] The server contacts the affiliated delivery service based on the order information and processes the delivery of the ingredients.

[0685] 8. Specific Examples

[0686] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user enters a family of a couple with two children and a monthly food budget of 50,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[0687] If the emotion engine detects that the user has been feeling stressed recently, it will adjust the menu to include ingredients with relaxing properties (e.g., herbs or teas). These suggestions are presented to the user along with an ingredient list that reflects sales. The user also has the option to order the items online and have them delivered to their home.

[0688] In this way, the system will help users make menu decisions and purchase ingredients more efficiently, and will also be able to suggest dietary habits that take the user's emotions into consideration, supporting a healthier and more satisfying lifestyle.

[0689] The processing flow will be explained below.

[0690] Step 1:

[0691] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[0692] Step 2:

[0693] The device uploads the photo to the app. The device sends the photo it has taken to the app, which then transfers it to the server.

[0694] Step 3:

[0695] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[0696] Step 4:

[0697] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "sushi," "pasta," etc.).

[0698] Step 5:

[0699] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[0700] Step 6:

[0701] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[0702] Step 7:

[0703] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[0704] Step 8:

[0705] The server recognizes emotions. The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information.

[0706] Step 9:

[0707] The server updates the emotion data. The emotion engine updates the emotion database based on user input and feedback, learning the user's emotional tendencies over a long period of time.

[0708] Step 10:

[0709] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores it in a database.

[0710] Step 11:

[0711] The server generates a menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. The server also adjusts the menu based on the emotional data, taking into account the user's emotional state.

[0712] Step 12:

[0713] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[0714] Step 13:

[0715] The user selects an ingredient purchasing option. The user reviews the suggested meal plan and selects how to purchase the ingredients needed (e.g., online order, delivery, or select a location).

[0716] Step 14:

[0717] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[0718] Step 15:

[0719] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[0720] Step 16:

[0721] The user receives the ingredients and prepares the meal according to the provided menu.

[0722] Example 2

[0723] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Conventional menu suggestion systems have had problems in that it is difficult to propose menus that take into account the user's preferences and emotions, and it is also difficult to efficiently suggest ingredients using sale information and carry out purchasing procedures. Furthermore, they lack the functionality to adjust menus based on the user's emotions, which limits their ability to support an efficient and satisfying dietary lifestyle.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0726] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient lists to the user; emotion engine means for recognizing the user's emotions using the photo and the input information; and means for adjusting the menu based on the user's emotion data. This makes it possible to propose an optimal menu that comprehensively takes into account the user's preferences, emotions, and sales information, thereby streamlining the purchase of necessary ingredients.

[0727] "User" refers to an individual who uses the system to take photos of their meals and receives menu suggestions and support for purchasing ingredients.

[0728] "Photos of meals" refer to image data of meals taken by a user using a camera function.

[0729] "Server means" refers to a device or system that receives photos of meals uploaded by users and then analyzes, stores, and processes the data.

[0730] "Image recognition means" refers to the technology or algorithm used to analyze received photos of food and identify their contents.

[0731] "Preference data" refers to data that indicates a user's eating preferences and tendencies.

[0732] "Address, family composition, monthly food budget" refers to information about an individual's residence, the number and composition of family members, and monthly food budget information that a user inputs into the system.

[0733] "Storage means" refers to the technology or device for storing input information and analysis results in a database, etc.

[0734] "Special sale information" refers to information about sales at special prices offered by nearby retailers.

[0735] "Menu" refers to suggestions that show the combinations and contents of meals that a user should eat.

[0736] "Emotion engine means" refers to a technology or device for recognizing a user's emotions and performing processing based on that data.

[0737] "Purchase options" refers to options that allow a user to purchase ingredients needed for a suggested menu.

[0738] "Home delivery service" refers to a service that delivers purchased ingredients to a location specified by the user.

[0739] The system embodying this invention allows users to take photos of their meals, analyzes the photos, and proposes optimal menus. By inputting the user's address, family composition, and monthly food budget, the system suggests value-for-money ingredients taking into account sales information from nearby stores, and also provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[0740] Hardware and Software Configuration

[0741] This system uses the following hardware and software:

[0742] Smartphones and tablets: devices that allow users to take photos of their meals and enter and display information through an application.

[0743] Cloud server: A server for analyzing, storing, and processing received photo data and input information. The following software is also used:

[0744] Image recognition libraries (e.g., TensorFlow, OpenCV): Analyze photos of food and identify the contents of the food.

[0745] Database management system (e.g., MySQL, PostgreSQL): Stores user input information and analysis results.

[0746] Sentiment analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API): Recognizes user emotions and updates / stores emotion data.

[0747] Special Offer Information Collection API: Collects special offer information from nearby retailers (using retailer APIs and scraping technology).

[0748] System Operation

[0749] 1. Take and upload a photo of your meal:

[0750] Users take photos of their meals using the camera on their smartphone or tablet device and upload them to a dedicated app, which then sends the photo data to a cloud server.

[0751] 2. Photo analysis and preference data update:

[0752] The server analyzes the received photo data using image recognition technology, and based on the analysis results, the user's preference data is recorded and updated in the database.

[0753] 3. Enter and save your user information:

[0754] Users enter their address, family composition, and monthly food budget through the app, and the device sends this information to the server, which stores it in a database.

[0755] 4. Emotion Recognition and Emotion Data Update:

[0756] The server uses an emotion engine to recognize the user's emotions based on the photos and input information received, and the emotion data is recorded and updated in the database.

[0757] 5. Collecting special offers:

[0758] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[0759] 6. Menu generation and suggestions:

[0760] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[0761] 7. Select your food purchase options and order:

[0762] The user checks the displayed menu and selects how to purchase the necessary ingredients, and the terminal sends the details to the server, which then contacts an affiliated delivery service based on the order information and processes the delivery of the ingredients.

[0763] Specific examples

[0764] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user inputs a family structure of a couple with two children and a monthly food budget of ¥50,000, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server might suggest a menu of "sushi + miso soup + salad." If the emotion engine detects that the user has recently been feeling stressed, the server might adjust the menu to include ingredients with a relaxing effect (e.g., herbs or tea). These suggestions are presented to the user along with a list of ingredients that reflect the sale information, and the user can also select the option to order ingredients online and have them delivered to their home.

[0765] Prompt Sentence Examples

[0766] "I want you to upload photos of your meals and get suggestions for the perfect menu."

[0767] "Our family consists of a couple and two children, and our monthly food budget is 50,000 yen. We live in central Tokyo. Please tell me what menu we'd like to have based on this information."

[0768] "I've been feeling stressed lately. Can you suggest a meal that includes relaxing ingredients?"

[0769] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0770] Step 1:

[0771] The user takes a photo of the meal using the camera on their smartphone or tablet.

[0772] Input: The food item to be photographed.

[0773] What it does: Open the camera app, focus on your food and press the shutter button.

[0774] Output: Photo data of the meal.

[0775] Step 2:

[0776] The device uploads the captured photo data to a dedicated app and sends this data to a cloud server.

[0777] Input: The captured photo data.

[0778] How it works: Retrieves photo data from the device's internal storage and sends a POST request to the server via the REST API.

[0779] Output: Photo data uploaded to the server.

[0780] Step 3:

[0781] The server analyzes the received photo data using image recognition technology to identify the ingredients and the name of the dish.

[0782] Input: Photo data uploaded to the server.

[0783] How it works: Using image recognition libraries (e.g. TensorFlow, OpenCV), it inputs photo data and extracts ingredients and dish names.

[0784] Output: Extracted ingredients and dish names.

[0785] Step 4:

[0786] The server updates the user's preference data based on the analysis results.

[0787] Input: Extracted ingredient and dish name data.

[0788] What it does: Converts the analysis results into SQL queries and inserts or updates the preference data into a database.

[0789] Output: Updated preference data.

[0790] Step 5:

[0791] Users enter their address, family composition, and monthly food budget via the app, and the device sends this information to the server, which stores it in a database.

[0792] Input: address, family composition, monthly food budget.

[0793] How it works: A user enters information into a form in your app and presses the submit button. The device sends the form data as a POST request, and the server executes an SQL query to store it in the database.

[0794] Output: Address, family composition, and monthly food budget information stored in the database.

[0795] Step 6:

[0796] The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information, and updates the emotion data.

[0797] Input: Uploaded photo data and input information.

[0798] How it works: Sends photo data or text data to an emotion analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API), retrieves the emotion analysis results, and inserts or updates the results into a database as an SQL query.

[0799] Output: Updated emotion data.

[0800] Step 7:

[0801] Based on the user's address information, the server uses APIs and web scraping technology to collect special sale information from nearby stores and stores it in a database.

[0802] Input: User's address information.

[0803] How it works: The server accesses the retailer's API to retrieve sales information in JSON format, and optionally extracts information using web scraping techniques.

[0804] Output: Sale information stored in a database.

[0805] Step 8:

[0806] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[0807] Input: preference data, family composition, food budget, nutritional balance, sale information, emotional data.

[0808] How it works: It runs an algorithm that retrieves data by performing multiple database queries, then uses that data to calculate the optimal menu. The resulting menu data is sent in JSON format to the device, which parses it and displays it in the UI.

[0809] Output: Menu information displayed to the user.

[0810] Step 9:

[0811] The user checks the displayed menu and selects how to purchase the necessary ingredients. The terminal then sends the details to the server, which then contacts an affiliated delivery service based on the order information and arranges for the ingredients to be delivered.

[0812] Input: User selection of purchase option.

[0813] How it works: The user selects a purchase method using buttons or checkboxes within the app and presses the "Purchase" button. The device posts the selection to the server as a JSON-formatted request, and the server sends the order information to the delivery service's API and begins the delivery process.

[0814] Output: Order information sent to the delivery service.

[0815] (Application example 2)

[0816] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] In today's busy lifestyles, it is difficult for users to select appropriate meals, maintain nutritional balance, and make budget-friendly purchases. It is even more difficult to select meals that reflect the user's emotional state. To address these issues, an effective system is needed that proposes optimal menus that take into account the user's preferences and emotional state, and supports the user in making those purchases.

[0818] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; means for adjusting the menu based on the user's emotions using an emotion engine that recognizes the user's emotions; and means for inputting prompt sentences to recognize emotions and reflect them in the menu generation. This enables optimal meal suggestions based on the user's preferences and emotional state and efficient ingredient purchasing.

[0819] "Means for users to take photos of their meals" refers to devices or applications that allow users to take photos of their meals and save and transmit them as data.

[0820] "Server means" means a system for receiving, processing, storing and transmitting data sent by a user.

[0821] "Image recognition means" refers to algorithms or software that analyzes received photo data and identifies the contents of the meal.

[0822] The "means for updating preference data" is a system that records the user's food preferences based on the analyzed meal contents and updates the database.

[0823] "Means for entering address, family composition, and monthly food budget" refers to an interface that allows users to enter and send this information using an application or device.

[0824] "Means for storing information" refers to a mechanism for storing input information and analysis results in a database.

[0825] The "means for collecting sale information" is a system that acquires and stores sale information provided by nearby stores.

[0826] The "means for generating optimal menus" refers to algorithms or programs that suggest optimal meal combinations for users based on their preference data, family composition, food budget, nutritional balance, and special sale information.

[0827] The "means for displaying the generated menu and related ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's device.

[0828] The "means for adjusting menus based on the user's emotions using an emotion engine" is a system that uses an engine that analyzes the user's emotions to make meal suggestions based on the user's emotional state.

[0829] "Means for recognizing emotions and reflecting them in menu generation by inputting prompt sentences" is a mechanism that analyzes emotions based on prompt sentences input by the user in words or sentences and reflects them in menu suggestions.

[0830] The system for realizing this invention starts with the user taking a photo of their meal using a smartphone or tablet device and uploading the data to a server. This device has an application with photo and upload functions installed.

[0831] The server uses image recognition technology such as the TensorFlow model to analyze the received photo data. The analysis results are reflected in the user's preference database. User information is entered and saved via the application, with address, family composition, and food budget entered, and the information is saved in the database by the server.

[0832] Emotion recognition is achieved by analyzing user-provided text using natural language processing, for example by using the Hugging Face Transformers library to analyze emotion prompts entered by the user in the application, and the results of this analysis are stored in a user emotion database.

[0833] The server then uses the user's address and other required information to collect special offers from nearby retailers and delivery services via API, which is also stored in a database.

[0834] The server uses an algorithm to generate the optimal menu, based on a comprehensive analysis of the user's preferences, family composition, food budget, nutritional balance, sales information, and emotional data. The generated menu is then displayed on the user's device along with a list of related ingredients.

[0835] Emotion recognition and menu adjustment based on user-provided prompts involves emotion analysis, where the user inputs a specific prompt into the application, such as "I've been feeling tired lately, so please suggest a meal that will help me relax." Based on this information, the server generates and suggests a menu using ingredients that have a relaxing effect.

[0836] For example, if a user takes a photo of "sushi" and inputs their family composition (couple, two children) and monthly food budget of 50,000 yen, the server will use this information to gather sales information from nearby stores. For example, if there is a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[0837] Additionally, if a user inputs "I've been feeling stressed lately," the emotion engine will adjust the menu to include relaxing herbs and teas. These suggestions are provided to the user along with an ingredient list that reflects special offers, and the user can order them online and have them delivered to their home.

[0838] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0839] Step 1:

[0840] The user takes a photo of their meal using a smartphone or tablet, which temporarily stores the photo data within the app and prompts the user to confirm upload.

[0841] Input: Food photo

[0842] Output: Temporarily saved photo data

[0843] Step 2:

[0844] Users upload photos of their meals via the application to the server, which receives the photos and stores them in a database.

[0845] Input: Temporarily saved photo data

[0846] Output: Photo data stored on the server

[0847] Step 3:

[0848] The server analyzes the received photo data using image recognition techniques, such as TensorFlow, to identify the contents of the meal.

[0849] Input: Photo data stored on the server

[0850] Output: Identified meal content data

[0851] Step 4:

[0852] The server updates the user's preference data based on the meal content data. Specifically, it tags the meal content and compares it with previous preference data to reflect the user's preference trends in the database.

[0853] Input: Identified meal content data

[0854] Output: Updated preference data

[0855] Step 5:

[0856] The user enters their address, family composition, and monthly food budget through the application, and the device sends this information to the server, which stores it in a database.

[0857] Input: Address, family composition, monthly food budget

[0858] Output: User information data stored on the server

[0859] Step 6:

[0860] The server uses APIs to collect special sale information from nearby stores and delivery services, and the collected special sale information is stored in a database.

[0861] Input: User Address

[0862] Output: Collected sale information data

[0863] Step 7:

[0864] The user inputs a prompt sentence into the application. For example, a prompt sentence such as "I've been feeling tired lately, so please suggest a meal that will help me relax." The device then sends this prompt sentence to the server.

[0865] Input: prompt statement

[0866] Output: Prompt data sent to the server

[0867] Step 8:

[0868] The server analyzes the prompt sentence using natural language processing to recognize the user's emotions, and generates emotion data using libraries such as Hugging Face's Transformers library.

[0869] Input: Prompt data sent to the server

[0870] Output: Generated emotion data

[0871] Step 9:

[0872] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. Using a generative AI model, it suggests optimal meal combinations.

[0873] Input: User preference data, family structure, food budget, nutritional balance, sale information, emotional data

[0874] Output: Generated menu data

[0875] Step 10:

[0876] The generated menu and related ingredient list are displayed on the user's terminal, allowing the user to confirm the menu and check the ingredient list.

[0877] Input: Generated menu data

[0878] Output: Menu and ingredient list data displayed on the user's device

[0879] Step 11:

[0880] The user selects an option to purchase based on the displayed ingredient list, and the selected option is transmitted from the terminal to the server.

[0881] Input: User's choice option

[0882] Output: Purchase option data sent to the server

[0883] Step 12:

[0884] The server processes the order and, based on the purchase options selected, contacts the partner delivery service and arranges for the delivery of the ingredients.

[0885] Input: Purchase option data sent to the server

[0886] Output: Order data sent to the delivery service

[0887] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0888] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0889] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0890] [Third embodiment]

[0891] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0892] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0893] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0894] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0895] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0896] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0897] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0898] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0899] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0900] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0901] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0902] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0903] The system for implementing this invention allows users to simply take photos of their meals, and the system analyzes the photos to suggest optimal menus. Furthermore, by inputting the user's address, family composition, and monthly food budget, the system suggests cost-effective ingredients taking into account sales information from nearby stores, and also provides the option to purchase the ingredients if necessary.

[0904] Overview of program processing

[0905] 1. Take and upload photos of your food

[0906] Users take photos of their meals with their smartphones or tablets and upload them to the app.

[0907] The terminal transmits this photo data to the server.

[0908] 2. Photo analysis and updating of preference data

[0909] The server analyzes the received photos using image recognition technology.

[0910] The server identifies the type of meal shown in the photo (e.g., pasta, salad, etc.) and stores this in the user's preference database.

[0911] 3. Enter your user information

[0912] Users enter their address, family composition, and monthly food budget through the app.

[0913] The terminal transmits this information to the server.

[0914] The server stores the received information in a database.

[0915] 4. Collecting information on special offers

[0916] The server uses the API to collect sale information from nearby stores based on the user's address.

[0917] Special sale information is updated in real time and stored in a database.

[0918] 5. Menu generation and suggestions

[0919] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0920] The proposed menu and a list of ingredients required for it are sent from the server to the terminal.

[0921] The terminal displays this to the user.

[0922] 6. Select your food purchase options and place your order

[0923] The user reviews the proposed menu and selects how to purchase the ingredients needed (e.g., order online for delivery).

[0924] The device sends the selection to the server.

[0925] Based on the order information, the server contacts the affiliated delivery service and arranges for the ingredients to be delivered.

[0926] The user receives the ingredients at home and cooks according to the provided menu.

[0927] Specific examples

[0928] For example, if a user uploads a photo of "grilled salmon," the server analyzes the photo and learns that salmon and grilled dishes are popular. If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a supermarket is having a sale on salmon, the server will suggest a menu using the sale items: "grilled salmon + vegetable salad + soup." This suggestion includes a list of ingredients that are on sale, and the user can also select the option to order the ingredients online and have them delivered to their home.

[0929] In this way, the system streamlines the daily task of deciding on menus and purchasing ingredients, which can be a heavy burden for users, and supports healthy and economical eating habits.

[0930] The processing flow will be explained below.

[0931] Step 1:

[0932] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[0933] Step 2:

[0934] The device uploads photos to the app. The device sends the photos it has taken to the app, which then transfers the photo data to the server.

[0935] Step 3:

[0936] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[0937] Step 4:

[0938] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "grilled salmon," "salad," etc.).

[0939] Step 5:

[0940] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[0941] Step 6:

[0942] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[0943] Step 7:

[0944] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[0945] Step 8:

[0946] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores the sale information in a database.

[0947] Step 9:

[0948] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[0949] Step 10:

[0950] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[0951] Step 11:

[0952] The user selects a grocery purchase option. The user selects their preferred method of grocery purchase from several options within the app (e.g., online ordering, delivery, select location).

[0953] Step 12:

[0954] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[0955] Step 13:

[0956] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[0957] Step 14:

[0958] The user receives the ingredients and prepares the meal according to the provided menu.

[0959] Example 1

[0960] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0961] In many modern households, planning daily menus and efficiently purchasing ingredients according to the family's preferences and budget are a significant burden. Many people also find it difficult to shop economically by utilizing sales information. The purpose of this invention is to solve these problems and provide a system that supports an efficient and economical dietary lifestyle.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0963] In this invention, the server includes: means for a user to take photos of meals; computer means for receiving the photos; image recognition means for analyzing the received photos and identifying the contents of the meals; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; database means for storing the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and associated ingredient list to the user; means for presenting the user with options for purchasing ingredients based on the displayed menu and ingredient list; and means for processing ingredient orders based on the options. This allows users to efficiently decide on their daily menu and purchase the necessary ingredients economically.

[0964] A "user" is someone who uses the system to take photos of their meals and input information such as their personal information and food budget.

[0965] "Means for taking photos of food" refers to functions or applications for taking photos of food using electronic devices such as smartphones or tablet devices.

[0966] "Computer means" refers to a server or computer system that receives the captured photograph data and performs subsequent analysis and data storage.

[0967] "Image recognition means" refers to software or algorithms that analyze received photos and identify the ingredients or dishes depicted.

[0968] "Preference data" refers to a database that records a user's food preferences and tendencies.

[0969] "Means for inputting address, family composition, and monthly food budget" is a function that provides an interface for the user to input their own address, family composition, and monthly food budget.

[0970] The "database means" is a system for storing and managing input information and analysis results as structured data.

[0971] "Sales information" refers to sale information and product price information obtained from nearby retailers.

[0972] "Means of collecting sales information" refers to APIs and web scraping technologies for collecting real-time sales information and product price information from nearby retailers.

[0973] "Means for generating menus" refers to algorithms or programs that generate optimal menus based on collected preference data, family composition, food budget, nutritional balance, and sales information.

[0974] The "means for displaying the ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's terminal.

[0975] The "means for presenting options for purchasing ingredients" is a function that presents options for the user to order ingredients online.

[0976] The "means for processing an order" refers to a system for communicating ingredient order information to a related service based on a user's selection and processing the order.

[0977] This invention is a system that helps users efficiently decide on daily menus and purchase ingredients economically. This system allows users to take photos of meals using a smartphone or tablet device, analyzes the photos, updates the user's preference data, and provides menu suggestions and ingredient purchasing support that take into account the user's address, family composition, and monthly food budget entered by the user.

[0978] Specifically, the following hardware and software are used.

[0979] 1. Take and upload a photo of your meal:

[0980] Users take photos of their meals using their smartphones or tablets and upload them to a dedicated app, which then sends the photos to a server.

[0981] 2. Photo analysis and preference data update:

[0982] The server temporarily stores the received photos in local storage. The server then analyzes the photos using image recognition technologies such as Google Cloud Vision API and Amazon Rekognition. The analysis results are obtained as dish names and ingredient information, such as "grilled salmon" or "pasta." The server updates the user's preference database based on these analysis results.

[0983] 3. Enter your user information:

[0984] Users enter their address, family composition, and monthly food budget through the app, and the device sends the information to the server, which then stores it in a database.

[0985] 4. Collecting special offers:

[0986] The server uses RapidAPI, Zomato API, etc. to collect sales information from nearby stores based on the user's address information. This sales information is updated in real time and stored in a database.

[0987] 5. Menu generation and suggestions:

[0988] The server runs a menu generation algorithm implemented in a programming language such as Python based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. This algorithm generates an optimal menu, such as "grilled salmon + vegetable salad + soup," and creates a list of ingredients needed for it. The server sends this information to the device, which displays it to the user.

[0989] 6. Select your food purchase options and order:

[0990] The user checks the proposed menu through the app and selects how to purchase the necessary ingredients (e.g., ordering online). The user's selection is sent from the device to the server, which then contacts the partner delivery service (e.g., Amazon Fresh or Rakuten Seiyu Net Supermarket) and processes the order. The user receives the ingredients at home and cooks them according to the provided menu.

[0991] Specific examples

[0992] For example, if a user uploads a photo of "grilled salmon," the server will analyze it using the Google Cloud Vision API and get the results "salmon," "grilled," and "lemon." If the user enters their address in Minato Ward, Tokyo, a family structure of a couple with two children, and a monthly food budget of 30,000 yen, the server will use RapidAPI to obtain sales information from a nearby supermarket, collecting information such as "salmon on sale for 500 yen." Based on this information, the server will generate a menu of "grilled salmon + vegetable salad + consommé soup" and suggest it to the user. The user selects online ordering, and the server places an order for the ingredients with Amazon Fresh.

[0993] Prompt Sentence Examples

[0994] Simply take a photo of the food you've eaten and upload it to the app. Then, enter your address, family size, and monthly food budget. The system will suggest the best meal plan and give you the option to order the ingredients you need online.

[0995] As described above, the present invention enables users to make daily menu decisions and purchase ingredients more efficiently, and supports healthy and economical eating habits.

[0996] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0997] Step 1:

[0998] The user takes a photo of their meal using a smartphone or tablet device. The input is the photo data taken. The user uploads the photo data to the app, and the device sends it to the server. The output is the photo data sent to the server. Specifically, this involves taking a photo of "grilled salmon" using the device's camera function and pressing the "upload photo" button on the app.

[0999] Step 2:

[1000] The server temporarily stores the received photo data in local storage. The input is the photo data sent from the device. The server then sends this photo data to the image recognition API. The output is the analysis results. Specifically, the server calls the Google Cloud Vision API and obtains the analysis results "salmon," "grilled," and "lemon."

[1001] Step 3:

[1002] The server adds the obtained analysis results to the user's preference database and updates it. The input is the analysis results obtained from the image recognition API. The database is updated with new preference data. The output is the updated preference database. Specifically, this includes the action of adding the ingredient information "salmon" to the user's preference database.

[1003] Step 4:

[1004] The user enters their address, family composition, and monthly food budget via the app. The input is the address (e.g., Minato-ku, Tokyo), family composition (couple and two children), and monthly food budget (e.g., 30,000 yen) manually entered by the user. The device sends this information to the server. The output is the user information stored on the server. Specifically, this involves filling in the information in the app's input form and pressing the "Submit" button.

[1005] Step 5:

[1006] The server uses API to collect sale information from nearby stores based on the user's address information. The input is the user's address information and the store's API. The server obtains the sale information through the API and stores it in a database in real time. The output is the collected sale information. Specifically, this includes operations such as using RapidAPI to obtain information such as "Salmon is on sale for 500 yen."

[1007] Step 6:

[1008] The server runs an algorithm that generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. The input is a user database and a special sale information database. The algorithm generates a menu and a list of necessary ingredients. The output is the generated menu and ingredient list. Specifically, it includes the operation of generating a menu such as "grilled salmon + vegetable salad + soup" using an algorithm implemented in Python.

[1009] Step 7:

[1010] The server sends the generated menu and ingredient list to the terminal, which then displays it to the user. The input is the generated menu and ingredient list. The output is the menu and ingredient list displayed to the user. Specifically, this includes an action that displays the menu "Grilled salmon + vegetable salad + soup" on the app screen.

[1011] Step 8:

[1012] The user reviews the proposed menu through the app and selects the food options they wish to purchase. The input is the displayed menu and food list. The device sends the user's selections to the server. The output is the selections sent to the server. Specifically, this includes selecting "Order online" in the app and pressing the "Confirm selection" button.

[1013] Step 9:

[1014] Based on the selected order information, the server contacts the partner delivery service and processes the order. The input is the user's order information. The server sends the order details to the delivery service and arranges for the ingredients to be delivered to the user's address. The output is the progress of the delivery procedure. Specifically, this includes the actions of sending the order details to Amazon Fresh and receiving an order confirmation email.

[1015] This allows the system to streamline users' menu planning and ingredient purchasing, supporting healthy and economical eating habits.

[1016] (Application example 1)

[1017] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] Conventional menu suggestion systems do not adequately consider the user's preferences, budget, or family structure, and also lack the functionality to generate economical menus by utilizing sales information. As a result, users are unable to reduce the burden of planning their daily menus, and it is difficult to purchase ingredients efficiently and economically. Furthermore, they do not provide a means for users to quickly obtain the ingredients needed based on the suggested menus, so they lack the completeness of a comprehensive service.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1020] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; and means for linking with a delivery platform based on the ingredient list so that ingredients can be obtained quickly. This allows ingredients to be purchased efficiently and economically based on the proposed menu, and enables ingredients to be obtained more quickly.

[1021] "Means for users to take photos of their meals" refers to a function that allows users to easily take photos of their meals using devices such as smartphones or tablets and save them as digital data.

[1022] The "server means for receiving the photographs taken" is a server function for receiving, storing, and processing the photograph data sent by the user from the terminal via the network.

[1023] "Image recognition means for analyzing the received photograph and identifying the contents of the meal" is a function that uses image recognition technology to analyze the received photograph of the meal and identify the ingredients and dishes contained in the photograph.

[1024] The "means for updating the user's preference data based on the identified meal content" is a function that stores the meal content identified by image recognition in the user's preference database and keeps the user's preference information up to date.

[1025] "Means for users to input their address, family composition, and monthly food budget" refers to a function that allows users to input their address, number of family members and composition, and monthly food budget through an application.

[1026] The "means for saving the input information" is a function that saves information entered by the user, such as address, family composition, and monthly food budget, on a server for use in future plans and proposals.

[1027] "Means for collecting sales information from nearby stores" is a function that automatically collects sales information from nearby stores based on the user's address information via an API or the like.

[1028] "Means for generating optimal menus based on user preference data, family composition, food budget, nutritional balance, and sale information" is a function that automatically generates healthy and economical daily menus based on the user's individual information and collected sale information.

[1029] "Means for displaying the generated menu and related ingredient list to the user" is a function that displays the generated menu and the ingredient list required for it on a terminal used by the user (such as a smartphone or tablet).

[1030] "Means for linking with a delivery platform to quickly obtain ingredients based on the ingredient list" refers to a function that links with a delivery service to easily order ingredients online using the generated ingredient list.

[1031] This invention provides a system that allows users to easily take photos of their meals, analyzes the photos, and suggests optimal menus. The system considers the user's address, family composition, and monthly food budget, suggests optimal ingredients based on sales information from nearby stores, and, if necessary, provides the option to purchase the ingredients via delivery.

[1032] System configuration and hardware / software used

[1033] The system mainly consists of the following components:

[1034] 1. How to take photos of your food:

[1035] This function allows users to take photos of their meals using devices such as smartphones and tablets. The photo data is temporarily stored on the device.

[1036] 2. Server means:

[1037] This is a cloud server that receives and processes photographed photo data via the network. This server has built-in database and image analysis functions.

[1038] 3. Image Recognition Methods:

[1039] Using image analysis technology, the photos taken on the cloud are analyzed to identify ingredients and dishes, using TensorFlow and AWS Rekognition as the means for this.

[1040] 4. Preference data update method:

[1041] This function updates the user's preference database based on the food content identified by image recognition. This allows the system to accumulate user preferences and provide more accurate menu suggestions.

[1042] 5. User information input method:

[1043] It provides an interface for users to input their address, family composition, and monthly food budget. This information is sent to the server through the application and stored.

[1044] 6. How to collect special offers:

[1045] An API is used to gather sales information from nearby stores based on the user's address information. Google Places API or an API specifically for sales information can be used.

[1046] 7. Menu Generation Method:

[1047] Using a generative AI model, the system generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and collected sales information. This method uses a generative AI model such as GPT-3.

[1048] 8. Menu and ingredient list display method:

[1049] This function displays the generated menu and list of ingredients on the screen of a smartphone or tablet, allowing users to check the menu and prepare for ingredient purchases.

[1050] 9. Delivery Platform Collaboration Methods:

[1051] This function allows you to quickly obtain ingredients by linking with delivery services based on your ingredient list. This function uses delivery service APIs such as the Uber Eats API.

[1052] Specific examples

[1053] For example, if a user takes a photo of "grilled salmon" and uploads it to an app, the app sends the photo to a cloud server, which uses AWS Rekognition to analyze the photo and identify it as salmon. The server then updates the user's preference database with this information.

[1054] If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server will use the API to collect information about nearby sales based on the user's address. For example, if information is collected about a special sale on salmon at a specific store, the server will use a generative AI model (GPT-3) to generate an optimal menu that includes the special sale salmon. The generated menu is "grilled salmon + vegetable salad + soup," and a list of the necessary ingredients is also displayed.

[1055] Users can then order the ingredients online through a delivery platform and have them delivered quickly to their homes, allowing them to eat healthily and economically without any hassle.

[1056] Prompt Sentence Examples

[1057] As an example of a prompt, enter the following text:

[1058] Prompt: Suggest the best meal plan based on the given ingredients, family size, and budget. Image: "Grilled salmon", Family size: "Couple and two children", Monthly food budget: "30,000 yen", Local sale information: "Salmon is on sale"

[1059] This prompt allows the generative AI model to suggest optimal menus that take into account special sales information, allowing users to efficiently purchase ingredients based on these suggestions.

[1060] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1061] Step 1:

[1062] The user takes a photo of the meal.

[1063] Users take photos of their meals using devices such as smartphones or tablets, and save them as high-resolution image files using the device's camera function.

[1064] Step 2:

[1065] The device sends the photograph to the server.

[1066] The user selects a photo they have taken and uploads it to the cloud server through the application. The input photo file (image data) is sent by the application to the server via the network. The server receives this data and temporarily stores it.

[1067] Step 3:

[1068] The server analyzes the received photos and identifies the contents of the meal.

[1069] The server analyzes the received image data using image analysis tools such as TensorFlow and AWS Rekognition. The input is image data, and the output is data on identified ingredients and dishes. Specific operations include running an algorithm to analyze the image and automatically recognizing objects and ingredients in the photo.

[1070] Step 4:

[1071] The server updates the user's preference data based on the identified meal content.

[1072] The server uses the identified meal details as input to update the user's preference database, adding the analysis results to the database and storing the user's preferences and past history.

[1073] Step 5:

[1074] The user enters their address, family composition, and monthly food budget.

[1075] A user uses the application to input information such as address, family composition, monthly food budget, etc. This input data is sent from the application to the server and stored on the server. Specific operations include data input via the user interface and communication processing.

[1076] Step 6:

[1077] The server collects sale information from nearby stores.

[1078] Using the user's address information as input, the server collects sales information from nearby stores through an API (e.g., Google Places API). The output is sales information data from the stores, which is automatically updated based on the user's address. Specific operations include API calls and data retrieval processes.

[1079] Step 7:

[1080] The server generates the optimal menu.

[1081] The server uses a generative AI model (e.g., GPT-3) to generate an optimal menu using the user's preference data, family composition, food budget, nutritional balance, and sale information as input. The output is a suggested menu and a list of ingredients. Specific operations include generating and inputting prompts to the generative AI model and analyzing the results.

[1082] Step 8:

[1083] The generated menu and ingredient list are displayed to the user.

[1084] The server sends the generated menu and ingredient list to the terminal, which displays it on the screen of the user's smartphone or tablet. Specific operations include sending data and displaying it using a GUI (graphical user interface).

[1085] Step 9:

[1086] The user orders ingredients from the delivery platform based on the ingredient list.

[1087] The user checks the displayed list of ingredients and orders them through the delivery platform. The input is the list of ingredients and order information, and the output is request information sent to the delivery platform. Specific operations include confirming the order and processing the order using the delivery API.

[1088] Step 10:

[1089] The server works with the delivery platform to arrange for the delivery of ingredients.

[1090] Based on the order information from the delivery platform, the server immediately arranges delivery. The output includes order confirmation and delivery arrangement information from the delivery platform. Specific operations include calling the delivery API and updating the order status.

[1091] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1092] In a system embodying this invention, a user takes photos of their meals, and the system analyzes the photos to suggest optimal menus. The system also suggests value-for-money ingredients based on sales information from nearby stores by inputting the user's address, family composition, and monthly food budget, and provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[1093] Overview of program processing

[1094] 1. Take and upload photos of your food

[1095] The user takes a photo of their meal using a smartphone or tablet.

[1096] The device uploads this photo data to the app, which then transfers it to the server.

[1097] 2. Photo analysis and updating of preference data

[1098] The server analyzes the received photos using image recognition technology.

[1099] Based on the analysis results, the server updates the user's preference data.

[1100] 3. Enter and save user information

[1101] Users enter their address, family composition, and monthly food budget through the app.

[1102] The device sends this information to the server, which stores it in a database.

[1103] 4. Emotion Recognition and Emotion Data Update

[1104] The server uses the photograph and input information to recognize the user's emotions using an emotion engine.

[1105] The emotion engine updates emotion data based on user input and feedback, learning emotional trends over a long period of time.

[1106] 5. Collecting information on special offers

[1107] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[1108] 6. Menu generation and suggestions

[1109] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data.

[1110] The menu contents are sent to the terminal, which displays them to the user.

[1111] 7. Select your food purchase options and place your order

[1112] The user checks the displayed menu and selects how to purchase the necessary ingredients.

[1113] The device sends the selection to the server.

[1114] The server contacts the affiliated delivery service based on the order information and processes the delivery of the ingredients.

[1115] 8. Specific Examples

[1116] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user enters a family of a couple with two children and a monthly food budget of 50,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[1117] If the emotion engine detects that the user has been feeling stressed recently, it will adjust the menu to include ingredients with relaxing properties (e.g., herbs or teas). These suggestions are presented to the user along with an ingredient list that reflects sales. The user also has the option to order the items online and have them delivered to their home.

[1118] In this way, the system will help users make menu decisions and purchase ingredients more efficiently, and will also be able to suggest dietary habits that take the user's emotions into consideration, supporting a healthier and more satisfying lifestyle.

[1119] The processing flow will be explained below.

[1120] Step 1:

[1121] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[1122] Step 2:

[1123] The device uploads the photo to the app. The device sends the photo it has taken to the app, which then transfers it to the server.

[1124] Step 3:

[1125] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[1126] Step 4:

[1127] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "sushi," "pasta," etc.).

[1128] Step 5:

[1129] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[1130] Step 6:

[1131] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[1132] Step 7:

[1133] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[1134] Step 8:

[1135] The server recognizes emotions. The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information.

[1136] Step 9:

[1137] The server updates the emotion data. The emotion engine updates the emotion database based on user input and feedback, learning the user's emotional tendencies over a long period of time.

[1138] Step 10:

[1139] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores it in a database.

[1140] Step 11:

[1141] The server generates a menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. The server also adjusts the menu based on the emotional data, taking into account the user's emotional state.

[1142] Step 12:

[1143] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[1144] Step 13:

[1145] The user selects an ingredient purchasing option. The user reviews the suggested meal plan and selects how to purchase the ingredients needed (e.g., online order, delivery, or select a location).

[1146] Step 14:

[1147] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[1148] Step 15:

[1149] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[1150] Step 16:

[1151] The user receives the ingredients and prepares the meal according to the provided menu.

[1152] Example 2

[1153] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] Conventional menu suggestion systems have had problems in that it is difficult to propose menus that take into account the user's preferences and emotions, and it is also difficult to efficiently suggest ingredients using sale information and carry out purchasing procedures. Furthermore, they lack the functionality to adjust menus based on the user's emotions, which limits their ability to support an efficient and satisfying dietary lifestyle.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1156] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient lists to the user; emotion engine means for recognizing the user's emotions using the photo and the input information; and means for adjusting the menu based on the user's emotion data. This makes it possible to propose an optimal menu that comprehensively takes into account the user's preferences, emotions, and sales information, thereby streamlining the purchase of necessary ingredients.

[1157] "User" refers to an individual who uses the system to take photos of their meals and receives menu suggestions and support for purchasing ingredients.

[1158] "Photos of meals" refer to image data of meals taken by a user using a camera function.

[1159] "Server means" refers to a device or system that receives photos of meals uploaded by users and then analyzes, stores, and processes the data.

[1160] "Image recognition means" refers to the technology or algorithm used to analyze received photos of food and identify their contents.

[1161] "Preference data" refers to data that indicates a user's eating preferences and tendencies.

[1162] "Address, family composition, monthly food budget" refers to information about an individual's residence, the number and composition of family members, and monthly food budget information that a user inputs into the system.

[1163] "Storage means" refers to the technology or device for storing input information and analysis results in a database, etc.

[1164] "Special sale information" refers to information about sales at special prices offered by nearby retailers.

[1165] "Menu" refers to suggestions that show the combinations and contents of meals that a user should eat.

[1166] "Emotion engine means" refers to a technology or device for recognizing a user's emotions and performing processing based on that data.

[1167] "Purchase options" refers to options that allow a user to purchase ingredients needed for a suggested menu.

[1168] "Home delivery service" refers to a service that delivers purchased ingredients to a location specified by the user.

[1169] The system embodying this invention allows users to take photos of their meals, analyzes the photos, and proposes optimal menus. By inputting the user's address, family composition, and monthly food budget, the system suggests value-for-money ingredients taking into account sales information from nearby stores, and also provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[1170] Hardware and Software Configuration

[1171] This system uses the following hardware and software:

[1172] Smartphones and tablets: devices that allow users to take photos of their meals and enter and display information through an application.

[1173] Cloud server: A server for analyzing, storing, and processing received photo data and input information. The following software is also used:

[1174] Image recognition libraries (e.g., TensorFlow, OpenCV): Analyze photos of food and identify the contents of the food.

[1175] Database management system (e.g., MySQL, PostgreSQL): Stores user input information and analysis results.

[1176] Sentiment analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API): Recognizes user emotions and updates / stores emotion data.

[1177] Special Offer Information Collection API: Collects special offer information from nearby retailers (using retailer APIs and scraping technology).

[1178] System Operation

[1179] 1. Take and upload a photo of your meal:

[1180] Users take photos of their meals using the camera on their smartphone or tablet device and upload them to a dedicated app, which then sends the photo data to a cloud server.

[1181] 2. Photo analysis and preference data update:

[1182] The server analyzes the received photo data using image recognition technology, and based on the analysis results, the user's preference data is recorded and updated in the database.

[1183] 3. Enter and save your user information:

[1184] Users enter their address, family composition, and monthly food budget through the app, and the device sends this information to the server, which stores it in a database.

[1185] 4. Emotion Recognition and Emotion Data Update:

[1186] The server uses an emotion engine to recognize the user's emotions based on the photos and input information received, and the emotion data is recorded and updated in the database.

[1187] 5. Collecting special offers:

[1188] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[1189] 6. Menu generation and suggestions:

[1190] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[1191] 7. Select your food purchase options and order:

[1192] The user checks the displayed menu and selects how to purchase the necessary ingredients, and the terminal sends the details to the server, which then contacts an affiliated delivery service based on the order information and processes the delivery of the ingredients.

[1193] Specific examples

[1194] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user inputs a family structure of a couple with two children and a monthly food budget of ¥50,000, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server might suggest a menu of "sushi + miso soup + salad." If the emotion engine detects that the user has recently been feeling stressed, the server might adjust the menu to include ingredients with a relaxing effect (e.g., herbs or tea). These suggestions are presented to the user along with a list of ingredients that reflect the sale information, and the user can also select the option to order ingredients online and have them delivered to their home.

[1195] Prompt Sentence Examples

[1196] "I want you to upload photos of your meals and get suggestions for the perfect menu."

[1197] "Our family consists of a couple and two children, and our monthly food budget is 50,000 yen. We live in central Tokyo. Please tell me what menu we'd like to have based on this information."

[1198] "I've been feeling stressed lately. Can you suggest a meal that includes relaxing ingredients?"

[1199] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1200] Step 1:

[1201] The user takes a photo of the meal using the camera on their smartphone or tablet.

[1202] Input: The food item to be photographed.

[1203] What it does: Open the camera app, focus on your food and press the shutter button.

[1204] Output: Photo data of the meal.

[1205] Step 2:

[1206] The device uploads the captured photo data to a dedicated app and sends this data to a cloud server.

[1207] Input: The captured photo data.

[1208] How it works: Retrieves photo data from the device's internal storage and sends a POST request to the server via the REST API.

[1209] Output: Photo data uploaded to the server.

[1210] Step 3:

[1211] The server analyzes the received photo data using image recognition technology to identify the ingredients and the name of the dish.

[1212] Input: Photo data uploaded to the server.

[1213] How it works: Using image recognition libraries (e.g. TensorFlow, OpenCV), it inputs photo data and extracts ingredients and dish names.

[1214] Output: Extracted ingredients and dish names.

[1215] Step 4:

[1216] The server updates the user's preference data based on the analysis results.

[1217] Input: Extracted ingredient and dish name data.

[1218] What it does: Converts the analysis results into SQL queries and inserts or updates the preference data into a database.

[1219] Output: Updated preference data.

[1220] Step 5:

[1221] Users enter their address, family composition, and monthly food budget via the app, and the device sends this information to the server, which stores it in a database.

[1222] Input: address, family composition, monthly food budget.

[1223] How it works: A user enters information into a form in your app and presses the submit button. The device sends the form data as a POST request, and the server executes an SQL query to store it in the database.

[1224] Output: Address, family composition, and monthly food budget information stored in the database.

[1225] Step 6:

[1226] The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information, and updates the emotion data.

[1227] Input: Uploaded photo data and input information.

[1228] How it works: Sends photo data or text data to an emotion analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API), retrieves the emotion analysis results, and inserts or updates the results into a database as an SQL query.

[1229] Output: Updated emotion data.

[1230] Step 7:

[1231] Based on the user's address information, the server uses APIs and web scraping technology to collect special sale information from nearby stores and stores it in a database.

[1232] Input: User's address information.

[1233] How it works: The server accesses the retailer's API to retrieve sales information in JSON format, and optionally extracts information using web scraping techniques.

[1234] Output: Sale information stored in a database.

[1235] Step 8:

[1236] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[1237] Input: preference data, family composition, food budget, nutritional balance, sale information, emotional data.

[1238] How it works: It runs an algorithm that retrieves data by performing multiple database queries, then uses that data to calculate the optimal menu. The resulting menu data is sent in JSON format to the device, which parses it and displays it in the UI.

[1239] Output: Menu information displayed to the user.

[1240] Step 9:

[1241] The user checks the displayed menu and selects how to purchase the necessary ingredients. The terminal then sends the details to the server, which then contacts an affiliated delivery service based on the order information and arranges for the ingredients to be delivered.

[1242] Input: User selection of purchase option.

[1243] How it works: The user selects a purchase method using buttons or checkboxes within the app and presses the "Purchase" button. The device posts the selection to the server as a JSON-formatted request, and the server sends the order information to the delivery service's API and begins the delivery process.

[1244] Output: Order information sent to the delivery service.

[1245] (Application example 2)

[1246] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] In today's busy lifestyles, it is difficult for users to select appropriate meals, maintain nutritional balance, and make budget-friendly purchases. It is even more difficult to select meals that reflect the user's emotional state. To address these issues, an effective system is needed that proposes optimal menus that take into account the user's preferences and emotional state, and supports the user in making those purchases.

[1248] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; means for adjusting the menu based on the user's emotions using an emotion engine that recognizes the user's emotions; and means for inputting prompt sentences to recognize emotions and reflect them in the menu generation. This enables optimal meal suggestions based on the user's preferences and emotional state and efficient ingredient purchasing.

[1249] "Means for users to take photos of their meals" refers to devices or applications that allow users to take photos of their meals and save and transmit them as data.

[1250] "Server means" means a system for receiving, processing, storing and transmitting data sent by a user.

[1251] "Image recognition means" refers to algorithms or software that analyzes received photo data and identifies the contents of the meal.

[1252] The "means for updating preference data" is a system that records the user's food preferences based on the analyzed meal contents and updates the database.

[1253] "Means for entering address, family composition, and monthly food budget" refers to an interface that allows users to enter and send this information using an application or device.

[1254] "Means for storing information" refers to a mechanism for storing input information and analysis results in a database.

[1255] The "means for collecting sale information" is a system that acquires and stores sale information provided by nearby stores.

[1256] The "means for generating optimal menus" refers to algorithms or programs that suggest optimal meal combinations for users based on their preference data, family composition, food budget, nutritional balance, and special sale information.

[1257] The "means for displaying the generated menu and related ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's device.

[1258] The "means for adjusting menus based on the user's emotions using an emotion engine" is a system that uses an engine that analyzes the user's emotions to make meal suggestions based on the user's emotional state.

[1259] "Means for recognizing emotions and reflecting them in menu generation by inputting prompt sentences" is a mechanism that analyzes emotions based on prompt sentences input by the user in words or sentences and reflects them in menu suggestions.

[1260] The system for realizing this invention starts with the user taking a photo of their meal using a smartphone or tablet device and uploading the data to a server. This device has an application with photo and upload functions installed.

[1261] The server uses image recognition technology such as the TensorFlow model to analyze the received photo data. The analysis results are reflected in the user's preference database. User information is entered and saved via the application, with address, family composition, and food budget entered, and the information is saved in the database by the server.

[1262] Emotion recognition is achieved by analyzing user-provided text using natural language processing, for example by using the Hugging Face Transformers library to analyze emotion prompts entered by the user in the application, and the results of this analysis are stored in a user emotion database.

[1263] The server then uses the user's address and other required information to collect special offers from nearby retailers and delivery services via API, which is also stored in a database.

[1264] The server uses an algorithm to generate the optimal menu, based on a comprehensive analysis of the user's preferences, family composition, food budget, nutritional balance, sales information, and emotional data. The generated menu is then displayed on the user's device along with a list of related ingredients.

[1265] Emotion recognition and menu adjustment based on user-provided prompts involves emotion analysis, where the user inputs a specific prompt into the application, such as "I've been feeling tired lately, so please suggest a meal that will help me relax." Based on this information, the server generates and suggests a menu using ingredients that have a relaxing effect.

[1266] For example, if a user takes a photo of "sushi" and inputs their family composition (couple, two children) and monthly food budget of 50,000 yen, the server will use this information to gather sales information from nearby stores. For example, if there is a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[1267] Additionally, if a user inputs "I've been feeling stressed lately," the emotion engine will adjust the menu to include relaxing herbs and teas. These suggestions are provided to the user along with an ingredient list that reflects special offers, and the user can order them online and have them delivered to their home.

[1268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1269] Step 1:

[1270] The user takes a photo of their meal using a smartphone or tablet, which temporarily stores the photo data within the app and prompts the user to confirm upload.

[1271] Input: Food photo

[1272] Output: Temporarily saved photo data

[1273] Step 2:

[1274] Users upload photos of their meals via the application to the server, which receives the photos and stores them in a database.

[1275] Input: Temporarily saved photo data

[1276] Output: Photo data stored on the server

[1277] Step 3:

[1278] The server analyzes the received photo data using image recognition techniques, such as TensorFlow, to identify the contents of the meal.

[1279] Input: Photo data stored on the server

[1280] Output: Identified meal content data

[1281] Step 4:

[1282] The server updates the user's preference data based on the meal content data. Specifically, it tags the meal content and compares it with previous preference data to reflect the user's preference trends in the database.

[1283] Input: Identified meal content data

[1284] Output: Updated preference data

[1285] Step 5:

[1286] The user enters their address, family composition, and monthly food budget through the application, and the device sends this information to the server, which stores it in a database.

[1287] Input: Address, family composition, monthly food budget

[1288] Output: User information data stored on the server

[1289] Step 6:

[1290] The server uses APIs to collect special sale information from nearby stores and delivery services, and the collected special sale information is stored in a database.

[1291] Input: User Address

[1292] Output: Collected sale information data

[1293] Step 7:

[1294] The user inputs a prompt sentence into the application. For example, a prompt sentence such as "I've been feeling tired lately, so please suggest a meal that will help me relax." The device then sends this prompt sentence to the server.

[1295] Input: prompt statement

[1296] Output: Prompt data sent to the server

[1297] Step 8:

[1298] The server analyzes the prompt sentence using natural language processing to recognize the user's emotions, and generates emotion data using libraries such as Hugging Face's Transformers library.

[1299] Input: Prompt data sent to the server

[1300] Output: Generated emotion data

[1301] Step 9:

[1302] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. Using a generative AI model, it suggests optimal meal combinations.

[1303] Input: User preference data, family structure, food budget, nutritional balance, sale information, emotional data

[1304] Output: Generated menu data

[1305] Step 10:

[1306] The generated menu and related ingredient list are displayed on the user's terminal, allowing the user to confirm the menu and check the ingredient list.

[1307] Input: Generated menu data

[1308] Output: Menu and ingredient list data displayed on the user's device

[1309] Step 11:

[1310] The user selects an option to purchase based on the displayed ingredient list, and the selected option is transmitted from the terminal to the server.

[1311] Input: User's choice option

[1312] Output: Purchase option data sent to the server

[1313] Step 12:

[1314] The server processes the order and, based on the purchase options selected, contacts the partner delivery service and arranges for the delivery of the ingredients.

[1315] Input: Purchase option data sent to the server

[1316] Output: Order data sent to the delivery service

[1317] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1318] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1319] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1320] [Fourth embodiment]

[1321] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1322] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1323] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1324] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1325] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1326] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1327] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1328] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1329] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1330] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1331] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1332] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1333] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1334] The system for implementing this invention allows users to simply take photos of their meals, and the system analyzes the photos to suggest optimal menus. Furthermore, by inputting the user's address, family composition, and monthly food budget, the system suggests cost-effective ingredients taking into account sales information from nearby stores, and also provides the option to purchase the ingredients if necessary.

[1335] Overview of program processing

[1336] 1. Take and upload photos of your food

[1337] Users take photos of their meals with their smartphones or tablets and upload them to the app.

[1338] The terminal transmits this photo data to the server.

[1339] 2. Photo analysis and updating of preference data

[1340] The server analyzes the received photos using image recognition technology.

[1341] The server identifies the type of meal shown in the photo (e.g., pasta, salad, etc.) and stores this in the user's preference database.

[1342] 3. Enter your user information

[1343] Users enter their address, family composition, and monthly food budget through the app.

[1344] The terminal transmits this information to the server.

[1345] The server stores the received information in a database.

[1346] 4. Collecting information on special offers

[1347] The server uses the API to collect sale information from nearby stores based on the user's address.

[1348] Special sale information is updated in real time and stored in a database.

[1349] 5. Menu generation and suggestions

[1350] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[1351] The proposed menu and a list of ingredients required for it are sent from the server to the terminal.

[1352] The terminal displays this to the user.

[1353] 6. Select your food purchase options and place your order

[1354] The user reviews the proposed menu and selects how to purchase the ingredients needed (e.g., order online for delivery).

[1355] The device sends the selection to the server.

[1356] Based on the order information, the server contacts the affiliated delivery service and arranges for the ingredients to be delivered.

[1357] The user receives the ingredients at home and cooks according to the provided menu.

[1358] Specific examples

[1359] For example, if a user uploads a photo of "grilled salmon," the server analyzes the photo and learns that salmon and grilled dishes are popular. If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a supermarket is having a sale on salmon, the server will suggest a menu using the sale items: "grilled salmon + vegetable salad + soup." This suggestion includes a list of ingredients that are on sale, and the user can also select the option to order the ingredients online and have them delivered to their home.

[1360] In this way, the system streamlines the daily task of deciding on menus and purchasing ingredients, which can be a heavy burden for users, and supports healthy and economical eating habits.

[1361] The processing flow will be explained below.

[1362] Step 1:

[1363] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[1364] Step 2:

[1365] The device uploads photos to the app. The device sends the photos it has taken to the app, which then transfers the photo data to the server.

[1366] Step 3:

[1367] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[1368] Step 4:

[1369] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "grilled salmon," "salad," etc.).

[1370] Step 5:

[1371] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[1372] Step 6:

[1373] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[1374] Step 7:

[1375] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[1376] Step 8:

[1377] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores the sale information in a database.

[1378] Step 9:

[1379] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and special sale information.

[1380] Step 10:

[1381] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[1382] Step 11:

[1383] The user selects a grocery purchase option. The user selects their preferred method of grocery purchase from several options within the app (e.g., online ordering, delivery, select location).

[1384] Step 12:

[1385] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[1386] Step 13:

[1387] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[1388] Step 14:

[1389] The user receives the ingredients and prepares the meal according to the provided menu.

[1390] Example 1

[1391] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1392] In many modern households, planning daily menus and efficiently purchasing ingredients according to the family's preferences and budget are a significant burden. Many people also find it difficult to shop economically by utilizing sales information. The purpose of this invention is to solve these problems and provide a system that supports an efficient and economical dietary lifestyle.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1394] In this invention, the server includes: means for a user to take photos of meals; computer means for receiving the photos; image recognition means for analyzing the received photos and identifying the contents of the meals; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; database means for storing the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and associated ingredient list to the user; means for presenting the user with options for purchasing ingredients based on the displayed menu and ingredient list; and means for processing ingredient orders based on the options. This allows users to efficiently decide on their daily menu and purchase the necessary ingredients economically.

[1395] A "user" is someone who uses the system to take photos of their meals and input information such as their personal information and food budget.

[1396] "Means for taking photos of food" refers to functions or applications for taking photos of food using electronic devices such as smartphones or tablet devices.

[1397] "Computer means" refers to a server or computer system that receives the captured photograph data and performs subsequent analysis and data storage.

[1398] "Image recognition means" refers to software or algorithms that analyze received photos and identify the ingredients or dishes depicted.

[1399] "Preference data" refers to a database that records a user's food preferences and tendencies.

[1400] "Means for inputting address, family composition, and monthly food budget" is a function that provides an interface for the user to input their own address, family composition, and monthly food budget.

[1401] The "database means" is a system for storing and managing input information and analysis results as structured data.

[1402] "Sales information" refers to sale information and product price information obtained from nearby retailers.

[1403] "Means of collecting sales information" refers to APIs and web scraping technologies for collecting real-time sales information and product price information from nearby retailers.

[1404] "Means for generating menus" refers to algorithms or programs that generate optimal menus based on collected preference data, family composition, food budget, nutritional balance, and sales information.

[1405] The "means for displaying the ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's terminal.

[1406] The "means for presenting options for purchasing ingredients" is a function that presents options for the user to order ingredients online.

[1407] The "means for processing an order" refers to a system for communicating ingredient order information to a related service based on a user's selection and processing the order.

[1408] This invention is a system that helps users efficiently decide on daily menus and purchase ingredients economically. This system allows users to take photos of meals using a smartphone or tablet device, analyzes the photos, updates the user's preference data, and provides menu suggestions and ingredient purchasing support that take into account the user's address, family composition, and monthly food budget entered by the user.

[1409] Specifically, the following hardware and software are used.

[1410] 1. Take and upload a photo of your meal:

[1411] Users take photos of their meals using their smartphones or tablets and upload them to a dedicated app, which then sends the photos to a server.

[1412] 2. Photo analysis and preference data update:

[1413] The server temporarily stores the received photos in local storage. The server then analyzes the photos using image recognition technologies such as Google Cloud Vision API and Amazon Rekognition. The analysis results are obtained as dish names and ingredient information, such as "grilled salmon" or "pasta." The server updates the user's preference database based on these analysis results.

[1414] 3. Enter your user information:

[1415] Users enter their address, family composition, and monthly food budget through the app, and the device sends the information to the server, which then stores it in a database.

[1416] 4. Collecting special offers:

[1417] The server uses RapidAPI, Zomato API, etc. to collect sales information from nearby stores based on the user's address information. This sales information is updated in real time and stored in a database.

[1418] 5. Menu generation and suggestions:

[1419] The server runs a menu generation algorithm implemented in a programming language such as Python based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. This algorithm generates an optimal menu, such as "grilled salmon + vegetable salad + soup," and creates a list of ingredients needed for it. The server sends this information to the device, which displays it to the user.

[1420] 6. Select your food purchase options and order:

[1421] The user checks the proposed menu through the app and selects how to purchase the necessary ingredients (e.g., ordering online). The user's selection is sent from the device to the server, which then contacts the partner delivery service (e.g., Amazon Fresh or Rakuten Seiyu Net Supermarket) and processes the order. The user receives the ingredients at home and cooks them according to the provided menu.

[1422] Specific examples

[1423] For example, if a user uploads a photo of "grilled salmon," the server will analyze it using the Google Cloud Vision API and get the results "salmon," "grilled," and "lemon." If the user enters their address in Minato Ward, Tokyo, a family structure of a couple with two children, and a monthly food budget of 30,000 yen, the server will use RapidAPI to obtain sales information from a nearby supermarket, collecting information such as "salmon on sale for 500 yen." Based on this information, the server will generate a menu of "grilled salmon + vegetable salad + consommé soup" and suggest it to the user. The user selects online ordering, and the server places an order for the ingredients with Amazon Fresh.

[1424] Prompt Sentence Examples

[1425] Simply take a photo of the food you've eaten and upload it to the app. Then, enter your address, family size, and monthly food budget. The system will suggest the best meal plan and give you the option to order the ingredients you need online.

[1426] As described above, the present invention enables users to make daily menu decisions and purchase ingredients more efficiently, and supports healthy and economical eating habits.

[1427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1428] Step 1:

[1429] The user takes a photo of their meal using a smartphone or tablet device. The input is the photo data taken. The user uploads the photo data to the app, and the device sends it to the server. The output is the photo data sent to the server. Specifically, this involves taking a photo of "grilled salmon" using the device's camera function and pressing the "upload photo" button on the app.

[1430] Step 2:

[1431] The server temporarily stores the received photo data in local storage. The input is the photo data sent from the device. The server then sends this photo data to the image recognition API. The output is the analysis results. Specifically, the server calls the Google Cloud Vision API and obtains the analysis results "salmon," "grilled," and "lemon."

[1432] Step 3:

[1433] The server adds the obtained analysis results to the user's preference database and updates it. The input is the analysis results obtained from the image recognition API. The database is updated with new preference data. The output is the updated preference database. Specifically, this includes the action of adding the ingredient information "salmon" to the user's preference database.

[1434] Step 4:

[1435] The user enters their address, family composition, and monthly food budget via the app. The input is the address (e.g., Minato-ku, Tokyo), family composition (couple and two children), and monthly food budget (e.g., 30,000 yen) manually entered by the user. The device sends this information to the server. The output is the user information stored on the server. Specifically, this involves filling in the information in the app's input form and pressing the "Submit" button.

[1436] Step 5:

[1437] The server uses API to collect sale information from nearby stores based on the user's address information. The input is the user's address information and the store's API. The server obtains the sale information through the API and stores it in a database in real time. The output is the collected sale information. Specifically, this includes operations such as using RapidAPI to obtain information such as "Salmon is on sale for 500 yen."

[1438] Step 6:

[1439] The server runs an algorithm that generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and special sale information. The input is a user database and a special sale information database. The algorithm generates a menu and a list of necessary ingredients. The output is the generated menu and ingredient list. Specifically, it includes the operation of generating a menu such as "grilled salmon + vegetable salad + soup" using an algorithm implemented in Python.

[1440] Step 7:

[1441] The server sends the generated menu and ingredient list to the terminal, which then displays it to the user. The input is the generated menu and ingredient list. The output is the menu and ingredient list displayed to the user. Specifically, this includes an action that displays the menu "Grilled salmon + vegetable salad + soup" on the app screen.

[1442] Step 8:

[1443] The user reviews the proposed menu through the app and selects the food options they wish to purchase. The input is the displayed menu and food list. The device sends the user's selections to the server. The output is the selections sent to the server. Specifically, this includes selecting "Order online" in the app and pressing the "Confirm selection" button.

[1444] Step 9:

[1445] Based on the selected order information, the server contacts the partner delivery service and processes the order. The input is the user's order information. The server sends the order details to the delivery service and arranges for the ingredients to be delivered to the user's address. The output is the progress of the delivery procedure. Specifically, this includes the actions of sending the order details to Amazon Fresh and receiving an order confirmation email.

[1446] This allows the system to streamline users' menu planning and ingredient purchasing, supporting healthy and economical eating habits.

[1447] (Application example 1)

[1448] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1449] Conventional menu suggestion systems do not adequately consider the user's preferences, budget, or family structure, and also lack the functionality to generate economical menus by utilizing sales information. As a result, users are unable to reduce the burden of planning their daily menus, and it is difficult to purchase ingredients efficiently and economically. Furthermore, they do not provide a means for users to quickly obtain the ingredients needed based on the suggested menus, so they lack the completeness of a comprehensive service.

[1450] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1451] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; and means for linking with a delivery platform based on the ingredient list so that ingredients can be obtained quickly. This allows ingredients to be purchased efficiently and economically based on the proposed menu, and enables ingredients to be obtained more quickly.

[1452] "Means for users to take photos of their meals" refers to a function that allows users to easily take photos of their meals using devices such as smartphones or tablets and save them as digital data.

[1453] The "server means for receiving the photographs taken" is a server function for receiving, storing, and processing the photograph data sent by the user from the terminal via the network.

[1454] "Image recognition means for analyzing the received photograph and identifying the contents of the meal" is a function that uses image recognition technology to analyze the received photograph of the meal and identify the ingredients and dishes contained in the photograph.

[1455] The "means for updating the user's preference data based on the identified meal content" is a function that stores the meal content identified by image recognition in the user's preference database and keeps the user's preference information up to date.

[1456] "Means for users to input their address, family composition, and monthly food budget" refers to a function that allows users to input their address, number of family members and composition, and monthly food budget through an application.

[1457] The "means for saving the input information" is a function that saves information entered by the user, such as address, family composition, and monthly food budget, on a server for use in future plans and proposals.

[1458] "Means for collecting sales information from nearby stores" is a function that automatically collects sales information from nearby stores based on the user's address information via an API or the like.

[1459] "Means for generating optimal menus based on user preference data, family composition, food budget, nutritional balance, and sale information" is a function that automatically generates healthy and economical daily menus based on the user's individual information and collected sale information.

[1460] "Means for displaying the generated menu and related ingredient list to the user" is a function that displays the generated menu and the ingredient list required for it on a terminal used by the user (such as a smartphone or tablet).

[1461] "Means for linking with a delivery platform to quickly obtain ingredients based on the ingredient list" refers to a function that links with a delivery service to easily order ingredients online using the generated ingredient list.

[1462] This invention provides a system that allows users to easily take photos of their meals, analyzes the photos, and suggests optimal menus. The system considers the user's address, family composition, and monthly food budget, suggests optimal ingredients based on sales information from nearby stores, and, if necessary, provides the option to purchase the ingredients via delivery.

[1463] System configuration and hardware / software used

[1464] The system mainly consists of the following components:

[1465] 1. How to take photos of your food:

[1466] This function allows users to take photos of their meals using devices such as smartphones and tablets. The photo data is temporarily stored on the device.

[1467] 2. Server means:

[1468] This is a cloud server that receives and processes photographed photo data via the network. This server has built-in database and image analysis functions.

[1469] 3. Image Recognition Methods:

[1470] Using image analysis technology, the photos taken on the cloud are analyzed to identify ingredients and dishes, using TensorFlow and AWS Rekognition as the means for this.

[1471] 4. Preference data update method:

[1472] This function updates the user's preference database based on the food content identified by image recognition. This allows the system to accumulate user preferences and provide more accurate menu suggestions.

[1473] 5. User information input method:

[1474] It provides an interface for users to input their address, family composition, and monthly food budget. This information is sent to the server through the application and stored.

[1475] 6. How to collect special offers:

[1476] An API is used to gather sales information from nearby stores based on the user's address information. Google Places API or an API specifically for sales information can be used.

[1477] 7. Menu Generation Method:

[1478] Using a generative AI model, the system generates optimal menus based on the user's preference data, family composition, food budget, nutritional balance, and collected sales information. This method uses a generative AI model such as GPT-3.

[1479] 8. Menu and ingredient list display method:

[1480] This function displays the generated menu and list of ingredients on the screen of a smartphone or tablet, allowing users to check the menu and prepare for ingredient purchases.

[1481] 9. Delivery Platform Collaboration Methods:

[1482] This function allows you to quickly obtain ingredients by linking with delivery services based on your ingredient list. This function uses delivery service APIs such as the Uber Eats API.

[1483] Specific examples

[1484] For example, if a user takes a photo of "grilled salmon" and uploads it to an app, the app sends the photo to a cloud server, which uses AWS Rekognition to analyze the photo and identify it as salmon. The server then updates the user's preference database with this information.

[1485] If a user enters a family of a couple with two children and a monthly food budget of 30,000 yen, the server will use the API to collect information about nearby sales based on the user's address. For example, if information is collected about a special sale on salmon at a specific store, the server will use a generative AI model (GPT-3) to generate an optimal menu that includes the special sale salmon. The generated menu is "grilled salmon + vegetable salad + soup," and a list of the necessary ingredients is also displayed.

[1486] Users can then order the ingredients online through a delivery platform and have them delivered quickly to their homes, allowing them to eat healthily and economically without any hassle.

[1487] Prompt Sentence Examples

[1488] As an example of a prompt, enter the following text:

[1489] Prompt: Suggest the best meal plan based on the given ingredients, family size, and budget. Image: "Grilled salmon", Family size: "Couple and two children", Monthly food budget: "30,000 yen", Local sale information: "Salmon is on sale"

[1490] This prompt allows the generative AI model to suggest optimal menus that take into account special sales information, allowing users to efficiently purchase ingredients based on these suggestions.

[1491] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1492] Step 1:

[1493] The user takes a photo of the meal.

[1494] Users take photos of their meals using devices such as smartphones or tablets, and save them as high-resolution image files using the device's camera function.

[1495] Step 2:

[1496] The device sends the photograph to the server.

[1497] The user selects a photo they have taken and uploads it to the cloud server through the application. The input photo file (image data) is sent by the application to the server via the network. The server receives this data and temporarily stores it.

[1498] Step 3:

[1499] The server analyzes the received photos and identifies the contents of the meal.

[1500] The server analyzes the received image data using image analysis tools such as TensorFlow and AWS Rekognition. The input is image data, and the output is data on identified ingredients and dishes. Specific operations include running an algorithm to analyze the image and automatically recognizing objects and ingredients in the photo.

[1501] Step 4:

[1502] The server updates the user's preference data based on the identified meal content.

[1503] The server uses the identified meal details as input to update the user's preference database, adding the analysis results to the database and storing the user's preferences and past history.

[1504] Step 5:

[1505] The user enters their address, family composition, and monthly food budget.

[1506] A user uses the application to input information such as address, family composition, monthly food budget, etc. This input data is sent from the application to the server and stored on the server. Specific operations include data input via the user interface and communication processing.

[1507] Step 6:

[1508] The server collects sale information from nearby stores.

[1509] Using the user's address information as input, the server collects sales information from nearby stores through an API (e.g., Google Places API). The output is sales information data from the stores, which is automatically updated based on the user's address. Specific operations include API calls and data retrieval processes.

[1510] Step 7:

[1511] The server generates the optimal menu.

[1512] The server uses a generative AI model (e.g., GPT-3) to generate an optimal menu using the user's preference data, family composition, food budget, nutritional balance, and sale information as input. The output is a suggested menu and a list of ingredients. Specific operations include generating and inputting prompts to the generative AI model and analyzing the results.

[1513] Step 8:

[1514] The generated menu and ingredient list are displayed to the user.

[1515] The server sends the generated menu and ingredient list to the terminal, which displays it on the screen of the user's smartphone or tablet. Specific operations include sending data and displaying it using a GUI (graphical user interface).

[1516] Step 9:

[1517] The user orders ingredients from the delivery platform based on the ingredient list.

[1518] The user checks the displayed list of ingredients and orders them through the delivery platform. The input is the list of ingredients and order information, and the output is request information sent to the delivery platform. Specific operations include confirming the order and processing the order using the delivery API.

[1519] Step 10:

[1520] The server works with the delivery platform to arrange for the delivery of ingredients.

[1521] Based on the order information from the delivery platform, the server immediately arranges delivery. The output includes order confirmation and delivery arrangement information from the delivery platform. Specific operations include calling the delivery API and updating the order status.

[1522] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1523] In a system embodying this invention, a user takes photos of their meals, and the system analyzes the photos to suggest optimal menus. The system also suggests value-for-money ingredients based on sales information from nearby stores by inputting the user's address, family composition, and monthly food budget, and provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[1524] Overview of program processing

[1525] 1. Take and upload photos of your food

[1526] The user takes a photo of their meal using a smartphone or tablet.

[1527] The device uploads this photo data to the app, which then transfers it to the server.

[1528] 2. Photo analysis and updating of preference data

[1529] The server analyzes the received photos using image recognition technology.

[1530] Based on the analysis results, the server updates the user's preference data.

[1531] 3. Enter and save user information

[1532] Users enter their address, family composition, and monthly food budget through the app.

[1533] The device sends this information to the server, which stores it in a database.

[1534] 4. Emotion Recognition and Emotion Data Update

[1535] The server uses the photograph and input information to recognize the user's emotions using an emotion engine.

[1536] The emotion engine updates emotion data based on user input and feedback, learning emotional trends over a long period of time.

[1537] 5. Collecting information on special offers

[1538] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[1539] 6. Menu generation and suggestions

[1540] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data.

[1541] The menu contents are sent to the terminal, which displays them to the user.

[1542] 7. Select your food purchase options and place your order

[1543] The user checks the displayed menu and selects how to purchase the necessary ingredients.

[1544] The device sends the selection to the server.

[1545] The server contacts the affiliated delivery service based on the order information and processes the delivery of the ingredients.

[1546] 8. Specific Examples

[1547] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user enters a family of a couple with two children and a monthly food budget of 50,000 yen, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[1548] If the emotion engine detects that the user has been feeling stressed recently, it will adjust the menu to include ingredients with relaxing properties (e.g., herbs or teas). These suggestions are presented to the user along with an ingredient list that reflects sales. The user also has the option to order the items online and have them delivered to their home.

[1549] In this way, the system will help users make menu decisions and purchase ingredients more efficiently, and will also be able to suggest dietary habits that take the user's emotions into consideration, supporting a healthier and more satisfying lifestyle.

[1550] The processing flow will be explained below.

[1551] Step 1:

[1552] The user takes a photo of the meal. The user takes a photo of the meal using a smartphone or tablet device.

[1553] Step 2:

[1554] The device uploads the photo to the app. The device sends the photo it has taken to the app, which then transfers it to the server.

[1555] Step 3:

[1556] The server receives the photos. The server receives the uploaded photo data and temporarily stores it in a database.

[1557] Step 4:

[1558] The server analyzes the photo. Using image recognition technology, the server analyzes the content of the photo and identifies specific meal items (e.g., "sushi," "pasta," etc.).

[1559] Step 5:

[1560] The server updates the preference data. The server updates the user's preference database based on the analysis results and adds new meal items.

[1561] Step 6:

[1562] The user enters their address, family composition, and monthly food budget. The user enters their address, family composition, and monthly food budget into a form within the app, and the device sends this information to the server.

[1563] Step 7:

[1564] The server stores the user information. The server stores the address, family composition, and food budget information it receives in a database.

[1565] Step 8:

[1566] The server recognizes emotions. The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information.

[1567] Step 9:

[1568] The server updates the emotion data. The emotion engine updates the emotion database based on user input and feedback, learning the user's emotional tendencies over a long period of time.

[1569] Step 10:

[1570] The server collects sale information. Using the API, the server collects sale information from nearby stores in real time based on the user's address and stores it in a database.

[1571] Step 11:

[1572] The server generates a menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. The server also adjusts the menu based on the emotional data, taking into account the user's emotional state.

[1573] Step 12:

[1574] The terminal displays the menu to the user. The server sends the generated menu and a list of necessary ingredients to the terminal, which then displays this information to the user.

[1575] Step 13:

[1576] The user selects an ingredient purchasing option. The user reviews the suggested meal plan and selects how to purchase the ingredients needed (e.g., online order, delivery, or select a location).

[1577] Step 14:

[1578] The terminal sends the user's selection to the server. The terminal sends the user's selected purchase option to the server.

[1579] Step 15:

[1580] The server processes the order and, based on the purchase options received, contacts the partner delivery service to begin the process of delivering the ingredients.

[1581] Step 16:

[1582] The user receives the ingredients and prepares the meal according to the provided menu.

[1583] Example 2

[1584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] Conventional menu suggestion systems have had problems in that it is difficult to propose menus that take into account the user's preferences and emotions, and it is also difficult to efficiently suggest ingredients using sale information and carry out purchasing procedures. Furthermore, they lack the functionality to adjust menus based on the user's emotions, which limits their ability to support an efficient and satisfying dietary lifestyle.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1587] In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; means for the user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient lists to the user; emotion engine means for recognizing the user's emotions using the photo and the input information; and means for adjusting the menu based on the user's emotion data. This makes it possible to propose an optimal menu that comprehensively takes into account the user's preferences, emotions, and sales information, thereby streamlining the purchase of necessary ingredients.

[1588] "User" refers to an individual who uses the system to take photos of their meals and receives menu suggestions and support for purchasing ingredients.

[1589] "Photos of meals" refer to image data of meals taken by a user using a camera function.

[1590] "Server means" refers to a device or system that receives photos of meals uploaded by users and then analyzes, stores, and processes the data.

[1591] "Image recognition means" refers to the technology or algorithm used to analyze received photos of food and identify their contents.

[1592] "Preference data" refers to data that indicates a user's eating preferences and tendencies.

[1593] "Address, family composition, monthly food budget" refers to information about an individual's residence, the number and composition of family members, and monthly food budget information that a user inputs into the system.

[1594] "Storage means" refers to the technology or device for storing input information and analysis results in a database, etc.

[1595] "Special sale information" refers to information about sales at special prices offered by nearby retailers.

[1596] "Menu" refers to suggestions that show the combinations and contents of meals that a user should eat.

[1597] "Emotion engine means" refers to a technology or device for recognizing a user's emotions and performing processing based on that data.

[1598] "Purchase options" refers to options that allow a user to purchase ingredients needed for a suggested menu.

[1599] "Home delivery service" refers to a service that delivers purchased ingredients to a location specified by the user.

[1600] The system embodying this invention allows users to take photos of their meals, analyzes the photos, and proposes optimal menus. By inputting the user's address, family composition, and monthly food budget, the system suggests value-for-money ingredients taking into account sales information from nearby stores, and also provides the option to purchase those ingredients if necessary. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can also adjust menus based on the user's emotions.

[1601] Hardware and Software Configuration

[1602] This system uses the following hardware and software:

[1603] Smartphones and tablets: devices that allow users to take photos of their meals and enter and display information through an application.

[1604] Cloud server: A server for analyzing, storing, and processing received photo data and input information. The following software is also used:

[1605] Image recognition libraries (e.g., TensorFlow, OpenCV): Analyze photos of food and identify the contents of the food.

[1606] Database management system (e.g., MySQL, PostgreSQL): Stores user input information and analysis results.

[1607] Sentiment analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API): Recognizes user emotions and updates / stores emotion data.

[1608] Special Offer Information Collection API: Collects special offer information from nearby retailers (using retailer APIs and scraping technology).

[1609] System Operation

[1610] 1. Take and upload a photo of your meal:

[1611] Users take photos of their meals using the camera on their smartphone or tablet device and upload them to a dedicated app, which then sends the photo data to a cloud server.

[1612] 2. Photo analysis and preference data update:

[1613] The server analyzes the received photo data using image recognition technology, and based on the analysis results, the user's preference data is recorded and updated in the database.

[1614] 3. Enter and save your user information:

[1615] Users enter their address, family composition, and monthly food budget through the app, and the device sends this information to the server, which stores it in a database.

[1616] 4. Emotion Recognition and Emotion Data Update:

[1617] The server uses an emotion engine to recognize the user's emotions based on the photos and input information received, and the emotion data is recorded and updated in the database.

[1618] 5. Collecting special offers:

[1619] The server uses the API to collect sales information from nearby stores based on the user's address and stores it in a database.

[1620] 6. Menu generation and suggestions:

[1621] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[1622] 7. Select your food purchase options and order:

[1623] The user checks the displayed menu and selects how to purchase the necessary ingredients, and the terminal sends the details to the server, which then contacts an affiliated delivery service based on the order information and processes the delivery of the ingredients.

[1624] Specific examples

[1625] For example, if a user uploads a photo of "sushi," the server analyzes the photo and learns that sushi is a favorite. If a user inputs a family structure of a couple with two children and a monthly food budget of ¥50,000, the server uses that information to gather information about sales at nearby stores. For example, if a nearby supermarket is having a sale on fresh fish, the server might suggest a menu of "sushi + miso soup + salad." If the emotion engine detects that the user has recently been feeling stressed, the server might adjust the menu to include ingredients with a relaxing effect (e.g., herbs or tea). These suggestions are presented to the user along with a list of ingredients that reflect the sale information, and the user can also select the option to order ingredients online and have them delivered to their home.

[1626] Prompt Sentence Examples

[1627] "I want you to upload photos of your meals and get suggestions for the perfect menu."

[1628] "Our family consists of a couple and two children, and our monthly food budget is 50,000 yen. We live in central Tokyo. Please tell me what menu we'd like to have based on this information."

[1629] "I've been feeling stressed lately. Can you suggest a meal that includes relaxing ingredients?"

[1630] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1631] Step 1:

[1632] The user takes a photo of the meal using the camera on their smartphone or tablet.

[1633] Input: The food item to be photographed.

[1634] What it does: Open the camera app, focus on your food and press the shutter button.

[1635] Output: Photo data of the meal.

[1636] Step 2:

[1637] The device uploads the captured photo data to a dedicated app and sends this data to a cloud server.

[1638] Input: The captured photo data.

[1639] How it works: Retrieves photo data from the device's internal storage and sends a POST request to the server via the REST API.

[1640] Output: Photo data uploaded to the server.

[1641] Step 3:

[1642] The server analyzes the received photo data using image recognition technology to identify the ingredients and the name of the dish.

[1643] Input: Photo data uploaded to the server.

[1644] How it works: Using image recognition libraries (e.g. TensorFlow, OpenCV), it inputs photo data and extracts ingredients and dish names.

[1645] Output: Extracted ingredients and dish names.

[1646] Step 4:

[1647] The server updates the user's preference data based on the analysis results.

[1648] Input: Extracted ingredient and dish name data.

[1649] What it does: Converts the analysis results into SQL queries and inserts or updates the preference data into a database.

[1650] Output: Updated preference data.

[1651] Step 5:

[1652] Users enter their address, family composition, and monthly food budget via the app, and the device sends this information to the server, which stores it in a database.

[1653] Input: address, family composition, monthly food budget.

[1654] How it works: A user enters information into a form in your app and presses the submit button. The device sends the form data as a POST request, and the server executes an SQL query to store it in the database.

[1655] Output: Address, family composition, and monthly food budget information stored in the database.

[1656] Step 6:

[1657] The server uses an emotion engine to recognize the user's emotions based on the uploaded photos and input information, and updates the emotion data.

[1658] Input: Uploaded photo data and input information.

[1659] How it works: Sends photo data or text data to an emotion analysis API (e.g., Microsoft Azure Emotion API, Google Cloud Natural Language API), retrieves the emotion analysis results, and inserts or updates the results into a database as an SQL query.

[1660] Output: Updated emotion data.

[1661] Step 7:

[1662] Based on the user's address information, the server uses APIs and web scraping technology to collect special sale information from nearby stores and stores it in a database.

[1663] Input: User's address information.

[1664] How it works: The server accesses the retailer's API to retrieve sales information in JSON format, and optionally extracts information using web scraping techniques.

[1665] Output: Sale information stored in a database.

[1666] Step 8:

[1667] The server generates an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data, and sends the generated menu to the terminal, which then displays it to the user.

[1668] Input: preference data, family composition, food budget, nutritional balance, sale information, emotional data.

[1669] How it works: It runs an algorithm that retrieves data by performing multiple database queries, then uses that data to calculate the optimal menu. The resulting menu data is sent in JSON format to the device, which parses it and displays it in the UI.

[1670] Output: Menu information displayed to the user.

[1671] Step 9:

[1672] The user checks the displayed menu and selects how to purchase the necessary ingredients. The terminal then sends the details to the server, which then contacts an affiliated delivery service based on the order information and arranges for the ingredients to be delivered.

[1673] Input: User selection of purchase option.

[1674] How it works: The user selects a purchase method using buttons or checkboxes within the app and presses the "Purchase" button. The device posts the selection to the server as a JSON-formatted request, and the server sends the order information to the delivery service's API and begins the delivery process.

[1675] Output: Order information sent to the delivery service.

[1676] (Application example 2)

[1677] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1678] In today's busy lifestyles, it is difficult for users to select appropriate meals, maintain nutritional balance, and make budget-friendly purchases. It is even more difficult to select meals that reflect the user's emotional state. To address these issues, an effective system is needed that proposes optimal menus that take into account the user's preferences and emotional state, and supports the user in making those purchases.

[1679] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a meal; server means for receiving the taken photo; image recognition means for analyzing the received photo and identifying the meal contents; means for updating the user's preference data based on the identified meal contents; means for a user to input their address, family composition, and monthly food budget; means for saving the input information; means for collecting sales information from nearby stores; means for generating an optimal menu based on the user's preference data, family composition, food budget, nutritional balance, and sales information; means for displaying the generated menu and related ingredient list to the user; means for adjusting the menu based on the user's emotions using an emotion engine that recognizes the user's emotions; and means for inputting prompt sentences to recognize emotions and reflect them in the menu generation. This enables optimal meal suggestions based on the user's preferences and emotional state and efficient ingredient purchasing.

[1680] "Means for users to take photos of their meals" refers to devices or applications that allow users to take photos of their meals and save and transmit them as data.

[1681] "Server means" means a system for receiving, processing, storing and transmitting data sent by a user.

[1682] "Image recognition means" refers to algorithms or software that analyzes received photo data and identifies the contents of the meal.

[1683] The "means for updating preference data" is a system that records the user's food preferences based on the analyzed meal contents and updates the database.

[1684] "Means for entering address, family composition, and monthly food budget" refers to an interface that allows users to enter and send this information using an application or device.

[1685] "Means for storing information" refers to a mechanism for storing input information and analysis results in a database.

[1686] The "means for collecting sale information" is a system that acquires and stores sale information provided by nearby stores.

[1687] The "means for generating optimal menus" refers to algorithms or programs that suggest optimal meal combinations for users based on their preference data, family composition, food budget, nutritional balance, and special sale information.

[1688] The "means for displaying the generated menu and related ingredient list to the user" is an interface for displaying the generated menu and the ingredient list required for it on the user's device.

[1689] The "means for adjusting menus based on the user's emotions using an emotion engine" is a system that uses an engine that analyzes the user's emotions to make meal suggestions based on the user's emotional state.

[1690] "Means for recognizing emotions and reflecting them in menu generation by inputting prompt sentences" is a mechanism that analyzes emotions based on prompt sentences input by the user in words or sentences and reflects them in menu suggestions.

[1691] The system for realizing this invention starts with the user taking a photo of their meal using a smartphone or tablet device and uploading the data to a server. This device has an application with photo and upload functions installed.

[1692] The server uses image recognition technology such as the TensorFlow model to analyze the received photo data. The analysis results are reflected in the user's preference database. User information is entered and saved via the application, with address, family composition, and food budget entered, and the information is saved in the database by the server.

[1693] Emotion recognition is achieved by analyzing user-provided text using natural language processing, for example by using the Hugging Face Transformers library to analyze emotion prompts entered by the user in the application, and the results of this analysis are stored in a user emotion database.

[1694] The server then uses the user's address and other required information to collect special offers from nearby retailers and delivery services via API, which is also stored in a database.

[1695] The server uses an algorithm to generate the optimal menu, based on a comprehensive analysis of the user's preferences, family composition, food budget, nutritional balance, sales information, and emotional data. The generated menu is then displayed on the user's device along with a list of related ingredients.

[1696] Emotion recognition and menu adjustment based on user-provided prompts involves emotion analysis, where the user inputs a specific prompt into the application, such as "I've been feeling tired lately, so please suggest a meal that will help me relax." Based on this information, the server generates and suggests a menu using ingredients that have a relaxing effect.

[1697] For example, if a user takes a photo of "sushi" and inputs their family composition (couple, two children) and monthly food budget of 50,000 yen, the server will use this information to gather sales information from nearby stores. For example, if there is a sale on fresh fish, the server will suggest a menu of "sushi + miso soup + salad."

[1698] Additionally, if a user inputs "I've been feeling stressed lately," the emotion engine will adjust the menu to include relaxing herbs and teas. These suggestions are provided to the user along with an ingredient list that reflects special offers, and the user can order them online and have them delivered to their home.

[1699] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1700] Step 1:

[1701] The user takes a photo of their meal using a smartphone or tablet, which temporarily stores the photo data within the app and prompts the user to confirm upload.

[1702] Input: Food photo

[1703] Output: Temporarily saved photo data

[1704] Step 2:

[1705] Users upload photos of their meals via the application to the server, which receives the photos and stores them in a database.

[1706] Input: Temporarily saved photo data

[1707] Output: Photo data stored on the server

[1708] Step 3:

[1709] The server analyzes the received photo data using image recognition techniques, such as TensorFlow, to identify the contents of the meal.

[1710] Input: Photo data stored on the server

[1711] Output: Identified meal content data

[1712] Step 4:

[1713] The server updates the user's preference data based on the meal content data. Specifically, it tags the meal content and compares it with previous preference data to reflect the user's preference trends in the database.

[1714] Input: Identified meal content data

[1715] Output: Updated preference data

[1716] Step 5:

[1717] The user enters their address, family composition, and monthly food budget through the application, and the device sends this information to the server, which stores it in a database.

[1718] Input: Address, family composition, monthly food budget

[1719] Output: User information data stored on the server

[1720] Step 6:

[1721] The server uses APIs to collect special sale information from nearby stores and delivery services, and the collected special sale information is stored in a database.

[1722] Input: User Address

[1723] Output: Collected sale information data

[1724] Step 7:

[1725] The user inputs a prompt sentence into the application. For example, a prompt sentence such as "I've been feeling tired lately, so please suggest a meal that will help me relax." The device then sends this prompt sentence to the server.

[1726] Input: prompt statement

[1727] Output: Prompt data sent to the server

[1728] Step 8:

[1729] The server analyzes the prompt sentence using natural language processing to recognize the user's emotions, and generates emotion data using libraries such as Hugging Face's Transformers library.

[1730] Input: Prompt data sent to the server

[1731] Output: Generated emotion data

[1732] Step 9:

[1733] The server generates the optimal menu based on the user's preference data, family composition, food budget, nutritional balance, sale information, and emotional data. Using a generative AI model, it suggests optimal meal combinations.

[1734] Input: User preference data, family structure, food budget, nutritional balance, sale information, emotional data

[1735] Output: Generated menu data

[1736] Step 10:

[1737] The generated menu and related ingredient list are displayed on the user's terminal, allowing the user to confirm the menu and check the ingredient list.

[1738] Input: Generated menu data

[1739] Output: Menu and ingredient list data displayed on the user's device

[1740] Step 11:

[1741] The user selects an option to purchase based on the displayed ingredient list, and the selected option is transmitted from the terminal to the server.

[1742] Input: User's choice option

[1743] Output: Purchase option data sent to the server

[1744] Step 12:

[1745] The server processes the order and, based on the purchase options selected, contacts the partner delivery service and arranges for the delivery of the ingredients.

[1746] Input: Purchase option data sent to the server

[1747] Output: Order data sent to the delivery service

[1748] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1749] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1750] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1751] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1752] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1753] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1754] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1755] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1756] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1757] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1758] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1759] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1760] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1761] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1762] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1763] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1764] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1765] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1766] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1767] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1768] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1769] The following is further disclosed regarding the above embodiment.

[1770] (Claim 1)

[1771] a means for a user to take a photo of a meal;

[1772] a server means for receiving the photographs;

[1773] an image recognition means for analyzing the received photograph and identifying the contents of the meal;

[1774] means for updating the user's preference data based on the identified meal contents;

[1775] A way for users to input their address, family composition, and monthly food budget;

[1776] means for storing the input information;

[1777] A means of collecting sales information from nearby retailers,

[1778] A means for generating an optimal menu based on user preference data, family structure, food budget, nutritional balance, and special sale information;

[1779] means for displaying the generated menu and associated ingredient list to a user;

[1780] A system including:

[1781] (Claim 2)

[1782] a means for a user to select an option for purchasing ingredients based on the displayed ingredient list;

[1783] means for processing the order based on the selected options;

[1784] The system of claim 1 further comprising:

[1785] (Claim 3)

[1786] 10. The system of claim 1, wherein the order processing means includes means for contacting a delivery service to deliver the ingredients.

[1787] "Example 1"

[1788] (Claim 1)

[1789] a means for a user to take a photo of a meal;

[1790] computer means for receiving the captured photograph;

[1791] an image recognition means for analyzing the received photograph and identifying the contents of the meal;

[1792] means for updating the user's preference data based on the identified meal contents;

[1793] A way for users to input their address, family composition, and monthly food budget;

[1794] a database means for storing the input information;

[1795] A means of collecting sales information from nearby dealers;

[1796] A means for generating an optimal menu based on user preference data, family structure, food budget, nutritional balance, and sales information;

[1797] means for displaying the generated menu and associated ingredient list to a user;

[1798] a means for presenting options for the user to purchase ingredients based on the displayed menu and ingredient list;

[1799] means for processing food orders based on the selections;

[1800] A system including:

[1801] (Claim 2)

[1802] 10. The system of claim 1, wherein the order processing means further comprises means for contacting a delivery service to deliver the ingredients.

[1803] (Claim 3)

[1804] 10. The system of claim 1, wherein the image recognition means includes means for using a cloud-based image recognition API.

[1805] "Application Example 1"

[1806] (Claim 1)

[1807] a means for a user to take a photo of a meal;

[1808] a server means for receiving the photographs;

[1809] an image recognition means for analyzing the received photograph and identifying the contents of the meal;

[1810] means for updating the user's preference data based on the identified meal contents;

[1811] A way for users to input their address, family composition, and monthly food budget;

[1812] means for storing the input information;

[1813] A means of collecting sales information from nearby retailers,

[1814] A means for generating an optimal menu based on user preference data, family structure, food budget, nutritional balance, and special sale information;

[1815] means for displaying the generated menu and associated ingredient list to a user;

[1816] A means for linking with a delivery platform so that ingredients can be quickly obtained based on the ingredient list;

[1817] A system including:

[1818] (Claim 2)

[1819] A means for a user to place a delivery order;

[1820] means for processing the order and delivering the food via a delivery service;

[1821] The system of claim 1 further comprising:

[1822] (Claim 3)

[1823] A method for using a generative AI model to suggest optimal menus based on user input and nearby sales information;

[1824] 10. The system of claim 1, further comprising means for utilizing prompt sentences to input to the generative AI model.

[1825] "Example 2: Combining Emotion Engines"

[1826] (Claim 1)

[1827] a means for a user to take a photo of a meal;

[1828] a server means for receiving the photographs;

[1829] an image recognition means for analyzing the received photograph and identifying the contents of the meal;

[1830] means for updating the user's preference data based on the identified meal contents;

[1831] A way for users to input their address, family composition, and monthly food budget;

[1832] means for storing the input information;

[1833] A means of collecting sales information from nearby retailers,

[1834] A means for generating an optimal menu based on user preference data, family structure, food budget, nutritional balance, and special sale information;

[1835] means for displaying the generated menu and associated ingredient list to a user;

[1836] an emotion engine means for recognizing the emotion of a user using a photograph or input information;

[1837] A means for adjusting a menu based on the user's emotional data;

[1838] A system including:

[1839] (Claim 2)

[1840] a means for a user to select an option for purchasing ingredients based on the displayed ingredient list;

[1841] means for processing the order based on the selected options;

[1842] The system of claim 1 further comprising:

[1843] (Claim 3)

[1844] 10. The system of claim 1, wherein the order processing means includes means for contacting a delivery service to deliver the ingredients.

[1845] "Application example 2 when combining emotion engines"

[1846] (Claim 1)

[1847] a means for a user to take a photo of a meal;

[1848] a server means for receiving the photographs;

[1849] an image recognition means for analyzing the received photograph and identifying the contents of the meal;

[1850] means for updating th...

Claims

1. a means for a user to take a photo of a meal; a server means for receiving the photographs; an image recognition means for analyzing the received photograph and identifying the contents of the meal; means for updating the user's preference data based on the identified meal contents; A way for users to input their address, family composition, and monthly food budget; means for storing the input information; A means of collecting sales information from nearby retailers, A means for generating an optimal menu based on user preference data, family structure, food budget, nutritional balance, and special sale information; means for displaying the generated menu and associated ingredient list to a user; A system including:

2. a means for a user to select an option for purchasing ingredients based on the displayed ingredient list; means for processing the order based on the selected options; The system of claim 1 further comprising:

3. 2. The system of claim 1, wherein the order processing means includes means for contacting a delivery service to deliver the ingredients.

Images