System

Abstract

To provide a system that streamlines a daily menu creation of a user, a food material purchase, and nutritional management, and reduces a burden.SOLUTION: A system includes the means of: registering a preference of a family, allergy information on the family, a health state of the family, contents of a refrigerator, and a repertory of foods a user can make; recognizing an emotion state of the user by an emotion engine; generating a prompt sentence instructing creation of a menu of the food to be suggested to the user on the basis of an emotion state of the user, the preference of the family, the allergy information on the family, the health state of the family, the content of the refrigerator, and the repertory of the food the user can make; means creating the menu of the food to be suggested to the user, using the generated prompt sentence and a generative AI model; and suggesting the created menu to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Planning a daily menu takes time and effort, and managing where to buy ingredients, costs, nutritional balance, etc. based on that menu is complicated. A system that can efficiently manage these things and reduce the burden on users is needed. [Means for solving the problem]

[0005] This invention provides a means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of meals one can make, a means for importing flyers from nearby supermarkets using image recognition, and a means for suggesting a weekly menu and what to buy where.Furthermore, by including a means for reading past purchase receipts to calculate the amount of money required, a means for calculating calories, a means for managing food expenses, and a means for adjusting menus to suit actual conditions after they have been planned, the system streamlines the user's daily menu planning, ingredient purchasing, and nutritional management, reducing their burden. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In the system of the present invention, a user registers his or her family's preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes that the user can make. For example, the user can register "likes fish dishes" and "doesn't like spicy food" as family preferences, "shrimp allergy" as allergy information, "diabetes" as health status, "chicken breast, broccoli, eggs" as refrigerator contents, and "sauteed chicken breast, sauteed garlic broccoli, rolled omelet" as the repertoire of dishes that the user can make.

[0029] "Example 2"

[0030] The system of the present invention uses image recognition to capture flyer information from nearby supermarkets. For example, a user can take a photo of a supermarket flyer with a smartphone camera and extract product names and prices from the image. This allows users to easily obtain the latest sales information from their favorite supermarkets.

[0031] "Example 3"

[0032] The system of this invention will propose a weekly menu and what to buy where. For example, it will automatically generate a weekly menu based on registered information and supermarket flyers, and suggest the necessary ingredients and where to buy them. This will significantly reduce the effort required for users to create a menu and buy ingredients.

[0033] "Example 4"

[0034] The system of the present invention reads receipts from past purchases and calculates the amount of money required for each purchase. For example, after shopping at a supermarket, a user can take a photo of the receipt with a smartphone camera and extract product names and prices from the image. This allows users to easily manage their food expenses.

[0035] "Example 5"

[0036] The system of the present invention allows calorie calculation, food cost management, and menu planning to be modified as needed to suit actual circumstances. For example, if the user changes the proposed menu, such as by deciding to eat out, the menu is modified to reflect that information, and calorie calculation and food cost management are updated. This allows users to flexibly manage their meals in accordance with their daily lifestyle.

[0037] The processing flow of each embodiment will be described below.

[0038] "Example 1"

[0039] Step 1: The user accesses the system and enters their profile information, including family preferences, allergy information, and health conditions.

[0040] Step 2: Next, the user registers their cooking repertoire with the system, including the name of the dish and the ingredients required.

[0041] Step 3: Finally, the user registers the contents of their refrigerator with the system, which allows the system to suggest meals that take into account the ingredients already in their refrigerator.

[0042] "Example 2"

[0043] Step 1: The user takes a photo of a supermarket flyer with their smartphone camera.

[0044] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[0045] Step 3: The extracted information is stored in the system and used to suggest menus and where to buy ingredients.

[0046] "Example 3"

[0047] Step 1: The system automatically generates a weekly menu based on the registered information and supermarket flyer information.

[0048] Step 2: The generated menu and list of ingredients are presented to the user.

[0049] Step 3: The user reviews the proposed menu and makes any necessary modifications or approvals.

[0050] "Example 4"

[0051] Step 1: After shopping at the supermarket, the user takes a photo of the receipt with their smartphone camera.

[0052] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[0053] Step 3: The extracted information is stored in the system and used to manage food expenses.

[0054] "Example 5"

[0055] Step 1: The user enters any changes to the proposed menu into the system.

[0056] Step 2: The system will modify the menu to reflect the information you entered, and update calorie counts and food cost management.

[0057] Step 3: The updated information is notified to the user, who can then check the new menu and meal cost information.

[0058] Example 1

[0059] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0060] Conventional home cooking support systems have had difficulty proposing recipes that take into account family preferences, allergies, health conditions, the contents of the refrigerator, and the user's own cooking repertoire. Another problem was that they were unable to quickly generate appropriate recipes when users requested recipe suggestions. Furthermore, if the generated recipe did not meet the user's requirements, it required the user to re-enter the information, which was time-consuming.

[0061] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health conditions, refrigerator contents, and the user's own cooking repertoire, means for saving the registered information in a database, means for the user to request dish suggestions, means for acquiring user information from the database, means for creating a prompt sentence to be input to the generative AI model based on the acquired information, means for sending the prompt sentence to the generative AI model and generating a recipe, and means for displaying the generated recipe to the user. This makes it possible to quickly generate and display recipes that meet the user's requirements.

[0062] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family likes, and taste tendencies.

[0063] "Allergy information" is information about food allergies that the user or their family members have.

[0064] "Health Status" refers to information about the health of the user and their family, including specific illnesses and health care needs.

[0065] "Refrigerator contents" refers to information about ingredients and foods currently stored in the user's refrigerator.

[0066] "Repertoire that can be cooked by the user" is information on the types of dishes and recipes that the user can cook by himself.

[0067] The "means for registration" refers to a method or device that allows a user to input and save information about family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of dishes that the user can make into the system.

[0068] The "means for saving to a database" refers to a method or device for storing and managing information entered by a user in a database.

[0069] A "means for requesting dish suggestions" is a method or device that allows a user to request dish suggestions from the system.

[0070] The "means for obtaining user information from a database" refers to a method or device that allows the system to search and retrieve user information stored in a database.

[0071] "Means for creating prompt sentences to be input to a generative AI model" refers to a method or device for creating sentences to give appropriate instructions to a generative AI model based on acquired user information.

[0072] "Means for sending a prompt sentence to a generative AI model and generating a recipe" refers to a method or device for sending a created prompt sentence to a generative AI model and generating a recipe that meets the user's requirements.

[0073] "Means for displaying the generated recipe to the user" refers to a method or device for displaying the recipe generated by the generative AI model on the user's terminal.

[0074] This invention is a system that allows users to register their family's preferences, allergy information, health status, refrigerator contents, and cooking repertoire, and then suggests appropriate recipes based on that information. This system is implemented using a server, terminals, and a generative AI model.

[0075] First, users use a dedicated web form or application to enter their family's preferences (e.g., "I like fish dishes" or "I don't like spicy food"), allergy information (e.g., "I'm allergic to shrimp"), health conditions (e.g., "I'm diabetic"), the contents of their refrigerator (e.g., "chicken breast, broccoli, eggs"), and the repertoire of dishes they can make (e.g., "sauteed chicken breast, sauteed garlic broccoli, rolled omelet").

[0076] Next, the terminal converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The server analyzes the received data and stores it in a relational database such as MySQL (registered trademark) or PostgreSQL. For example, it organizes the information by user ID and stores the data corresponding to each field.

[0077] When a user requests a recipe suggestion, the server retrieves user information from a database. For example, it uses SQL queries to retrieve the user's family preferences, allergies, health status, refrigerator contents, and cooking repertoire. Based on this information, the server creates a prompt to be input to the generative AI model.

[0078] Examples of specific prompts include the following:

[0079] "My family likes fish dishes but doesn't like spicy food. I'm allergic to shrimp, and one family member is diabetic. We have chicken breasts, broccoli, and eggs in the refrigerator. I can make sautéed chicken breasts, sautéed garlic broccoli, and omelet. Based on this information, please suggest some suitable recipes."

[0080] The server sends this prompt to a generative AI model (for example, GPT-4 (registered trademark) from OpenAI (registered trademark)) to generate a recipe. The generative AI model analyzes the prompt and generates a recipe that meets the user's requirements. The generated recipe is sent from the server to the device and displayed to the user. For example, the recipe title, ingredients, and instructions are displayed on the application screen.

[0081] The system allows users to quickly find the right recipe based on their family's preferences, health status, and what's in their refrigerator, and by using a generative AI model, it can generate recipes that meet the user's requirements with high accuracy.

[0082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0083] Step 1: Enter your user information

[0084] Users use a dedicated web form or application to enter their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. The information is then stored in text format on the device.

[0085] Step 2: Sending data

[0086] The device converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The input data includes family preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes the user can make. The output data is the JSON format data sent to the server.

[0087] Step 3: Save your data

[0088] The server analyzes the received data and stores it in a relational database such as MySQL or PostgreSQL. The input data is user information in JSON format, and the output data is the user information stored in the database. Specifically, it executes an INSERT statement on the database.

[0089] Step 4: Request a recipe suggestion

[0090] A user requests a recipe suggestion by clicking a "recipe suggestion" button in an application or website. The input data is the user's request, and the output data is a request signal to the server.

[0091] Step 5: Get the data

[0092] The server retrieves user information from the database. The input data is the user ID, and the output data is the user's family preferences, allergy information, health status, contents of the refrigerator, and the repertoire of dishes they can make. Specifically, it executes a SELECT statement on the database.

[0093] Step 6: Create a prompt statement

[0094] The server creates a prompt sentence to be input to the generative AI model based on the acquired information. The input data is the acquired user information, and the output data is the prompt sentence to be sent to the generative AI model. Specifically, the server converts the user information into text format and constructs the prompt sentence.

[0095] Step 7: Generate the recipe

[0096] The server sends a prompt to the generative AI model to generate a recipe. The input data is the prompt, and the output data is the generated recipe. Specifically, the server sends a prompt to the generative AI model (e.g., OpenAI's GPT-4) and receives the generated recipe.

[0097] Step 8: Submit your recipe

[0098] The server converts the generated recipe into a data format such as JSON and sends it to the terminal. The input data is the generated recipe, and the output data is the JSON-formatted recipe sent to the terminal.

[0099] Step 9: View the recipe

[0100] The device displays the received recipe to the user. The input data is a recipe in JSON format, and the output data is the recipe information to be displayed to the user. Specifically, the application screen displays the recipe title, ingredients, instructions, etc.

[0101] (Application example 1)

[0102] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Conventional food delivery services have difficulty suggesting optimal dishes that take into account the user's family preferences, allergies, and health conditions. Furthermore, they do not take into account the contents of the refrigerator or the user's cooking repertoire, making it difficult for users to use ingredients efficiently and maintain a healthy diet.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0105] In this invention, the server includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the recipes the user can make, means for importing flyer information from nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for generating prompts using a generative AI model to suggest optimal dishes, and means for ordering the ingredients needed for the suggested dishes. This enables the server to suggest optimal dishes tailored to the user's individual needs and to use ingredients efficiently.

[0106] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family prefers, as well as taste trends.

[0107] "Allergy information" is information about allergies that the user or their family members have.

[0108] "Health Status" means information about the health of a user or their family members, including specific illnesses or health care restrictions.

[0109] "What's in the refrigerator" is information about the ingredients currently stored in the user's refrigerator.

[0110] "Repertoire you can make" refers to information about the types of dishes and recipes that users can make themselves.

[0111] "Flyer information from nearby stores" is advertisements and sale information for products offered by stores in the user's neighborhood.

[0112] "Image recognition" is a technology that analyzes digital images to extract specific information.

[0113] A "generative AI model" is a model that has been trained using artificial intelligence to perform a specific task.

[0114] A "prompt" is text that is input to a generative AI model to produce a specific output.

[0115] The "optimal dish" is a dish suggested based on the user's family's preferences, allergy information, health condition, what's in the refrigerator, and the recipes they can make themselves.

[0116] "Means for ordering ingredients" is a function for ordering ingredients needed for the proposed dish online.

[0117] The system for implementing this invention is constructed as a smartphone application for the user. The system provides the user with the ability to register information about their family's preferences, allergies, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. This allows the user to receive suggestions for dishes that best suit their needs.

[0118] The server stores the information entered by the user in a database and generates a prompt using a generative AI model. Based on the generated prompt, the AI ​​model suggests the most suitable dish. The user can then order the ingredients needed for the suggested dish online.

[0119] Specifically, the server performs the following process.

[0120] 1. User data registration:

[0121] Through a smartphone application, users can input their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. This information is then sent to a server and stored in a database.

[0122] 2. Generate prompt sentence:

[0123] The server generates a prompt based on the stored user data, such as:

[0124] Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Shrimp allergy, Health condition: Diabetes, Refrigerator contents: Chicken breast, broccoli, eggs, Cooking skills: Sautéed chicken breast, Sautéed broccoli with garlic, Omelette

[0125] 3. Recipe suggestions using generative AI models:

[0126] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3 (registered trademark)) to suggest the best dish. The generative AI model generates a dish that suits the user's needs based on the prompt.

[0127] 4. Order ingredients:

[0128] The ingredients required for the proposed dishes can be ordered online by the user through a smartphone application, and the server provides the user with a list of the proposed dishes and the required ingredients, facilitating the ordering process.

[0129] This system allows users to receive recommendations for optimal meals that take into account their family's preferences and health conditions, allowing them to use ingredients efficiently.It also allows users to easily order the ingredients they need, reducing the hassle of shopping.

[0130] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0131] Step 1:

[0132] Through a smartphone application, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. The input data is sent from the application to a server. The server then stores the received data in a database. Examples of input data include "I like fish dishes," "I'm allergic to shrimp," "I'm diabetic," "chicken breast, broccoli, and eggs," and "sauteed chicken breast, sauteed broccoli with garlic, and rolled omelet."

[0133] Step 2:

[0134] The server generates prompts based on the saved user data. Prompts are text-format input data to be sent to the generative AI model. An example of a generated prompt is "Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Allergy to shrimp, Health condition: Diabetes, Contents of refrigerator: Chicken breast, Broccoli, Eggs, Cooked dishes: Sauteed chicken breast, Sauteed broccoli with garlic, Omelette."

[0135] Step 3:

[0136] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal dish based on the prompt to meet the user's needs. The output from the generative AI model is a list of suggested dishes. For example, it might suggest "teriyaki chicken breast" or "broccoli and egg salad."

[0137] Step 4:

[0138] The server receives the output from the generative AI model and sends a list of suggested dishes to the user's smartphone application, through which the user can review the suggested dishes.

[0139] Step 5:

[0140] The user selects from the proposed dishes and orders the necessary ingredients online. The application generates a list of ingredients required for the selected dish and provides an interface for the user to confirm the order. The server receives the order data and sends the order information to the corresponding store.

[0141] Step 6:

[0142] The store prepares ingredients based on the received order information and delivers them to the user, who can then use the ingredients to create the suggested dish.

[0143] In this way, users can have the best possible recipes suggested based on their family's preferences and health status, making efficient use of ingredients. Furthermore, they can easily order the ingredients they need, saving them the trouble of shopping.

[0144] Example 2

[0145] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] With conventional systems, users had to manually check the latest sales information of nearby stores, which was time-consuming and made it difficult to obtain accurate information.In addition, it was difficult to propose menus that took into account family preferences, allergies, health conditions, and the contents of the refrigerator, making it difficult to create efficient shopping plans.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0148] In this invention, the server includes a means for transmitting captured images to the server, a means for the server to perform image recognition and extract product name and price information, and a means for storing the extracted information in a database. This allows users to automatically analyze store advertising information captured with a smartphone camera and obtain product name and price information. It also facilitates menu suggestions and efficient shopping planning that take into account family preferences, allergies, health conditions, and refrigerator contents.

[0149] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0150] "Allergy information" refers to information about food allergies that each member of the family has.

[0151] "Health status" refers to information about the health of each family member, such as chronic illnesses or nutritional needs.

[0152] "What's in the refrigerator" refers to the types and quantities of food and beverages currently stored in the refrigerator.

[0153] "Repertoire" refers to the types of dishes a user can cook.

[0154] "Store advertising information" refers to product and price information listed in flyers and advertisements issued by nearby stores.

[0155] "Image recognition" refers to the technology of extracting specific information from captured images.

[0156] "Server" refers to a computer system for processing and storing data.

[0157] "Database" refers to a system for storing extracted information in an organized manner.

[0158] "Menu suggestions" refers to proposing a weekly meal plan that takes into account the family's preferences, health status, and the contents of the refrigerator.

[0159] "Shopping planning" refers to planning which stores to purchase the ingredients and products you need.

[0160] "Receipt reading" refers to the technology of reading information about previously purchased products from receipts.

[0161] "Calorie counting" refers to calculating the calories of food.

[0162] "Food cost management" refers to managing the costs of purchasing ingredients and products.

[0163] "Appropriately modify" refers to changing plans or proposals to suit actual circumstances.

[0164] This invention is a system that allows users to take a photo of a nearby store's advertising information using their smartphone camera and extract product names and prices from the image. This allows users to easily obtain the latest sales information. It also enables menu suggestions and efficient shopping plans that take into account family preferences, allergies, health conditions, and the contents of the refrigerator.

[0165] Hardware and software used

[0166] Hardware:

[0167] Smartphone: Used by users to take pictures of advertising information.

[0168] Server: A computer system used to process and store data.

[0169] software:

[0170] Camera app: The camera application installed on your smartphone.

[0171] Image Recognition Software: Software for extracting text information from images, such as Google® Cloud Vision API.

[0172] Database: A system for storing the extracted information, such as MySQL or PostgreSQL.

[0173] Program processing

[0174] The user takes a photo of the advertisement information using their smartphone camera.

[0175] The user launches the camera app on their smartphone and takes a photo of the store's advertising information. For example, if a user wants to take a photo of an "ABC Supermarket" flyer, they open the camera app, adjust the frame so that the entire flyer fits in the frame, and press the shutter button.

[0176] The device sends the captured image to the server.

[0177] The device (smartphone) sends the captured image to the server via an application. Specifically, the application sends the image data to the server as an HTTP POST request. For example, the image data is Base64 encoded and included in the request body.

[0178] The server performs image recognition and extracts product name and price information.

[0179] The server sends the received image data to the Google Cloud Vision API for image recognition. The API analyzes the text in the image and extracts product name and price information. For example, if the API recognizes the text "Apple 100 yen" and "Banana 150 yen," it returns that information to the server in JSON format.

[0180] The server stores the extracted information in a database.

[0181] The server saves the extracted product name and price information in a database. Specifically, it executes an INSERT query in a database such as MySQL or PostgreSQL to save the information. For example, information such as "apples 100 yen" and "bananas 150 yen" is saved in the corresponding tables in the database.

[0182] The device displays the information to the user

[0183] The device (smartphone) displays the product name and price information sent from the server to the user through the application. Specifically, the application receives the response from the server and displays the information on the screen. For example, the information "Apples 100 yen" and "Bananas 150 yen" is displayed on the app screen.

[0184] Examples of concrete examples and prompts

[0185] Examples:

[0186] A user takes a photo of a flyer for "ABC Supermarket." The flyer states "Apples 100 yen" and "Bananas 150 yen." The user takes a photo of the flyer with their smartphone camera and sends the image to a server via the app. The server analyzes the image using the Google Cloud Vision API and extracts the information "Apples 100 yen" and "Bananas 150 yen." This information is stored in a database and displayed in the user's app.

[0187] Example prompt sentence:

[0188] "Please take a photo of a flyer from a nearby supermarket and extract the product name and price information. For example, if the flyer says 'Apples 100 yen' and 'Bananas 150 yen', please extract and display that information."

[0189] In this way, the system allows users to easily obtain the latest sales information.

[0190] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0191] Step 1:

[0192] The user takes a photo of the advertisement information using their smartphone camera.

[0193] Specifically, the user launches the camera app on their smartphone, frames the store's advertising information, and presses the shutter button.

[0194] Input: Image of advertising information

[0195] Output: Captured image data

[0196] Step 2:

[0197] The device sends the captured image to the server.

[0198] Specifically, the smartphone application encodes the image data in Base64 and sends it to the server as an HTTP POST request.

[0199] Input: Photographed image data

[0200] Output: Image data sent to the server

[0201] Step 3:

[0202] The server performs image recognition and extracts product name and price information.

[0203] Specifically, the server sends the received image data to the Google Cloud Vision API, which then analyzes the text in the image to extract product name and price information.

[0204] Input: Image data sent to the server

[0205] Output: Extracted product name and price information (JSON format)

[0206] Step 4:

[0207] The server stores the extracted information in a database.

[0208] Specifically, the server executes an INSERT query to store the extracted product name and price information in a database such as MySQL or PostgreSQL.

[0209] Input: Extracted product name and price information (JSON format)

[0210] Output: Product name and price information stored in the database

[0211] Step 5:

[0212] The device displays the information to the user.

[0213] Specifically, the smartphone application receives a response from the server and displays the product name and price information on the screen.

[0214] Input: Product name and price information stored in the database

[0215] Output: Product name and price information displayed on the smartphone screen

[0216] (Application example 2)

[0217] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] Today's consumers are expected to shop efficiently and plan meals while taking into account their family's preferences, allergies, and health conditions. However, to achieve this, they must manually check supermarket flyers and manually enter product names and prices, which is extremely time-consuming. It is also difficult to track past price fluctuations, and there is a lack of information to make optimal purchases. Furthermore, there is a lack of guides to help users understand how to use the application.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended recipes; means for capturing flyer information from nearby commercial facilities using image recognition; means for suggesting a weekly menu and where to buy what; means for extracting product names and price information from flyer images captured with a smartphone camera; means for saving previously captured flyer information and tracking price fluctuations; means for providing the user with the latest sales information using a generative AI model; and means for generating prompts to guide the user on how to use the application. This enables users to shop efficiently and plan optimal menus. Tracking past price fluctuations also enables smarter shopping, and the generative AI model and prompts allow users to easily understand how to use the application.

[0220] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0221] "Allergy information" refers to information about allergies held by each member of the family.

[0222] "Health status" refers to information about the health of each family member, such as whether they need to avoid certain foods.

[0223] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[0224] "Repertoire" refers to the types of dishes a user can make.

[0225] "Commercial facilities" refers to retail stores such as supermarkets and department stores.

[0226] "Flyer information" refers to the content of flyers issued by commercial facilities, including advertisements and sales information.

[0227] "Image recognition" refers to the technology of extracting specific information from an image.

[0228] A "menu" refers to a meal plan for a specific period of time.

[0229] A "smartphone" is a type of mobile phone that can connect to the Internet and use applications.

[0230] "Camera" refers to a device for taking images or videos.

[0231] "Product Name" refers to the name of a product sold at a commercial establishment.

[0232] "Price Information" refers to information regarding the selling price of a product.

[0233] A "generative AI model" refers to a model that uses artificial intelligence to perform specific tasks.

[0234] A "prompt" is a guide sentence that encourages the user to take a specific action.

[0235] A system for implementing the present invention is configured as follows.

[0236] First, a user takes a photo of a flyer from a nearby commercial establishment using their smartphone camera, and the image is processed by an application on the smartphone, which uses OpenCV and Tesseract OCR to extract product name and price information from the image.

[0237] The extracted information is sent to a server and stored in a database. The server uses the stored data to track past price fluctuations and provide optimal shopping suggestions to users. It also uses generative AI models to provide users with the latest sales information.

[0238] Furthermore, when the user uses the application, the system generates prompts to guide them. For example, the system displays a prompt such as, "Take a photo of a flyer from a nearby supermarket to get the latest sales information."

[0239] The specific processing steps are as follows:

[0240] 1. Hardware: Use your smartphone camera to take a photo of the flyer.

[0241] 2. Software: Python, OpenCV, Tesseract OCR is used to extract text information from images.

[0242] 3. Data processing: Convert the image to grayscale and reduce noise.

[0243] 4. Data Calculation: Extract text from images using Tesseract OCR to obtain product name and price information.

[0244] 5. Data storage: The extracted information is sent to the server and stored in the database.

[0245] 6. Tracking price fluctuations: The server tracks price fluctuations based on historical data.

[0246] 7. Information provision: Using generative AI models, we provide users with the latest sales information.

[0247] 8. Prompt generation: Generate prompts to guide the user on how to use the application.

[0248] The system allows users to shop efficiently and plan optimal meals, tracks historical price fluctuations to help them shop smarter, and uses generative AI models and prompts to help users easily understand how to use the application.

[0249] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0250] Step 1:

[0251] A user uses the smartphone camera to take a picture of a flyer for a nearby business. The input is an image of the flyer, and the output is an image file stored on the smartphone that is used in subsequent processing steps.

[0252] Step 2:

[0253] The device loads the captured image into the application. The input is an image file stored on the smartphone, and the output is image data that can be processed within the application. Specifically, the application opens the image file and loads it into memory.

[0254] Step 3:

[0255] The device uses OpenCV to convert images to grayscale. The input is image data that can be processed within the application, and the output is a grayscale image. Specifically, it calls OpenCV functions to convert color images to grayscale.

[0256] Step 4:

[0257] The device uses Tesseract OCR to extract text information from grayscale images. The input is a grayscale image, and the output is the extracted text information. Specifically, the device calls Tesseract OCR functions to parse the text from the image and obtain product name and price information.

[0258] Step 5:

[0259] The terminal transmits the extracted text information to the server. The input is the extracted text information, and the output is the data transmitted to the server. In a specific operation, the text information is transmitted to the server via network communication.

[0260] Step 6:

[0261] The server stores the received text information in a database. The input is the text information sent to the server, and the output is the data stored in the database. Specifically, the server performs a database insert operation and stores the text information.

[0262] Step 7:

[0263] The server tracks price fluctuations based on the stored data. The input is past text information stored in the database, and the output is the analysis of price fluctuations. Specifically, it executes database queries to retrieve and analyze past price information.

[0264] Step 8:

[0265] The server uses the generative AI model to provide the user with the latest sales information. The input is the analysis results of price fluctuations and the latest text information, and the output is the sales information provided to the user. Specifically, the server runs the generative AI model to generate sales information.

[0266] Step 9:

[0267] The device generates prompts to guide the user on how to use the application. The input is the user's operation status and the application status, and the output is the generated prompt. Specifically, the device analyzes the user's operation history and generates an appropriate prompt. For example, the device might display a prompt saying, "Take a photo of a flyer from a nearby supermarket to get the latest sales information."

[0268] Example 3

[0269] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] In today's busy lifestyles, it is extremely time-consuming to efficiently plan menus and purchase the necessary ingredients while taking into account the preferences and health status of family members. It is also difficult to manage food expenses and calculate calories based on past purchase history. Furthermore, there is a demand for flexible menu changes to suit daily lifestyles. A system that can solve these problems is needed.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0272] In this invention, the server includes: means for registering family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire; means for importing flyer information from nearby stores using image recognition; means for suggesting a weekly menu and what to buy and where; means for receiving receipt images photographed by the user and extracting product names and prices using optical character recognition software; means for comparing the extracted product names and prices with past purchase history to calculate the total purchase price; and means for the user to input changes to the suggested menu, revise the menu based on that information, and update calorie calculations and food cost management. This allows users to efficiently create menus and purchase the necessary ingredients, and easily manage food costs and calorie calculations. It also allows users to flexibly change menus to suit their daily lifestyles.

[0273] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0274] "Allergy information" refers to information about food allergies that each member of the family has.

[0275] "Health status" refers to information about the health of each family member, including restrictions on the intake of certain foods and nutrients.

[0276] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[0277] "Repertoire" refers to the collection of types of dishes or recipes that a user can make.

[0278] "Store flyer information" refers to advertising information including special offers and discount information offered by nearby stores.

[0279] "Image recognition" refers to technology that extracts specific information from images, including optical character recognition (OCR).

[0280] A "menu" refers to a meal plan for a specific period of time (e.g., one week).

[0281] "Receipt image" refers to a photograph of the receipt that a user receives after shopping.

[0282] "Optical character recognition software" refers to software used to extract text information from images.

[0283] "Product name" refers to the name of the purchased product as it appears on the receipt.

[0284] "Price Information" refers to the price of the product listed on the receipt.

[0285] "Purchase history" refers to data that records information about products purchased by a user in the past.

[0286] "Calorie counting" refers to the process of calculating the calories of ingredients and dishes.

[0287] "Food cost management" refers to the process of managing the costs of ingredients and cooking.

[0288] "Change Information" refers to changes that a user makes to a suggested menu.

[0289] This invention is a system that registers the user's family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire, then uses image recognition to import flyers from nearby stores, proposing a weekly menu and what to buy where.The system also reads past purchase receipts, calculates how much each item will cost, calculates calories, manages food expenses, and can adjust the menu accordingly.

[0290] Hardware and software used

[0291] server

[0292] The server is the central location for receiving and processing the data sent by users. Specifically, it uses the following software:

[0293] Google Cloud Vision API: Optical character recognition software for extracting text data from receipt images.

[0294] Python's Pandas library: a library for processing and analyzing data.

[0295] Nutritional Data API: An API for obtaining calorie information for each ingredient.

[0296] Terminal

[0297] The terminal (smartphone or PC) provides an interface for users to input data and send it to the server. Specifically, it has the following functions:

[0298] Camera function: A function for taking photos of receipts.

[0299] Application: An application where the user enters data and sends it to the server.

[0300] user

[0301] Users use the system to:

[0302] Register your family's preferences, allergy information, health conditions, what's in your refrigerator, and cooking repertoire.

[0303] After shopping at the supermarket, you take a photo of the receipt and send it to the server via the app.

[0304] Change information for the proposed menu is entered and sent to the server.

[0305] Specific examples

[0306] For example, suppose a user goes shopping at a supermarket and receives a receipt totaling 3,000 yen. The user takes a photo of the receipt with their smartphone camera and sends it to the server via the app. The server uses the Google Cloud Vision API to extract product names and prices from the receipt, compares them with past purchase history, and calculates the total price.

[0307] If a user changes the menu suggestion by saying, "I've decided to eat out today," the information is sent to the server using a smartphone app. The server then uses this information to revise the menu and update calorie calculations and food cost management.

[0308] Prompt Sentence Examples

[0309] "After shopping at the supermarket, take a photo of your receipt, extract product names and prices from the image, and compare it with your past purchase history to calculate how much you can buy."

[0310] "If the user inputs changes to the proposed menu, modify the menu based on that information and update calorie calculations and food cost management." The flow of the specific processing in the third embodiment will be explained using FIG.

[0311] Step 1:

[0312] Register your family's preferences, allergy information, health status, refrigerator contents, and repertoire

[0313] user

[0314] Using a smartphone or PC application, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make.

[0315] Input: Family preferences, allergy information, health status, refrigerator contents, repertoire

[0316] Output: The registered information is saved in the database.

[0317] Specific operation: The user launches the app, enters information into the input form, and clicks the save button. The app sends the information to the server, which saves it in the database.

[0318] Step 2:

[0319] Importing store flyer information

[0320] Terminal

[0321] The terminal uses image recognition to capture flyer information from nearby stores.

[0322] Input: Store flyer image

[0323] Output: Extracted flyer information (product name, price, special sale information, etc.)

[0324] How it works: A user takes a photo of a store flyer with their smartphone camera, and the app sends the image to a server, which uses image recognition software to extract flyer information and store it in a database.

[0325] Step 3:

[0326] Weekly menu and shopping suggestions

[0327] server

[0328] Based on the registered information and store flyers, the server suggests a weekly menu and what to buy and where to buy it.

[0329] Input: Registered information, store flyer information

[0330] Output: Weekly menu and shopping suggestions

[0331] Specific operation: The server retrieves registered information and flyer information from the database, uses an algorithm to calculate a weekly menu and where to buy, and returns the calculation results to the user.

[0332] Step 4:

[0333] Capture and send a receipt image

[0334] user

[0335] After shopping at a supermarket, users take a photo of the receipt using their smartphone camera and send it to the server via the app.

[0336] Input: Receipt image

[0337] Output: Receipt image sent to the server

[0338] Specific operation: The user launches the app, selects the camera function to take a picture of the receipt, and presses the send button. The app then sends the image to the server.

[0339] Step 5:

[0340] Receipt image analysis and information extraction

[0341] server

[0342] The server analyzes the received receipt image and uses optical character recognition software to extract product name and price information.

[0343] Input: Receipt image

[0344] Output: Extracted product name and price information

[0345] What happens: The server receives the receipt image, uses optical character recognition software to extract the text data, and stores the extracted data in a database.

[0346] Step 6:

[0347] Matching with purchase history and calculating the amount

[0348] server

[0349] The server compares the extracted product name and price information with past purchase history and calculates how much the product will cost.

[0350] Input: Extracted product name and price information, past purchase history

[0351] Output: Calculated amount information

[0352] Specific operation: The server retrieves past purchase history from the database, compares it with the current data, calculates the amount, and returns the calculation result to the user.

[0353] Step 7:

[0354] Enter and submit menu change information

[0355] user

[0356] The user inputs modification information for the proposed menu and sends it to the server.

[0357] Input: Menu change information

[0358] Output: Changes sent to the server

[0359] Specific operation: The user launches the app, opens the menu change screen, enters the changes, and presses the save button. The app then sends the changes to the server.

[0360] Step 8:

[0361] Menu data updates, calorie calculations, and food cost management updates

[0362] server

[0363] The server updates the existing menu data based on the received menu change information, calculates calories, and manages food costs.

[0364] Input: Menu change information

[0365] Output: Updated menu data, calorie calculation results, food cost management results

[0366] Specific operation: The server receives the menu change information and updates the menu data in the database. It uses the Nutritional Data API to calculate calories and update the food cost management by comparing it with past purchase history. It then returns the updated results to the user.

[0367] (Application example 3)

[0368] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] The traditional process of creating menus and purchasing ingredients required time and effort for users. Managing food expenses and calculating calories was also time-consuming, making it difficult to flexibly adapt to daily lifestyles. Managing receipts and calculating costs based on past purchase history was also cumbersome. There is a need for a system that can solve these problems and provide comprehensive support for users' eating habits.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0371] In this invention, the server includes a means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended cooking repertoire; a means for importing flyer information from nearby supermarkets using image recognition; a means for suggesting a weekly menu and what to buy and where; a means for scanning receipts with a smartphone camera and extracting product names and price information; a means for automatically generating menus using a generative AI model based on the user's dietary data; a means for calculating calories and managing food costs based on the generated menu and receipt data; and a means for modifying the menu to suit the user's lifestyle. This allows users to significantly reduce the effort required for creating menus and purchasing ingredients, and makes it easy to manage food costs and calculate calories. It also allows for flexible menu modification to suit daily lifestyles.

[0372] "Family preferences" refers to the types of ingredients, dishes, and seasonings that each family member likes.

[0373] "Allergy information" refers to information about specific ingredients in the case where someone in the family has an allergic reaction to those ingredients.

[0374] "Health Condition" refers to information on dietary restrictions and recommended foods for specific health conditions or illnesses in the family, if any.

[0375] "What's in the refrigerator" refers to the types and quantities of ingredients and seasonings currently stored in the refrigerator.

[0376] "Repertoire" refers to the range of dishes and recipes a user can make.

[0377] "Supermarket flyer information" refers to the content of flyers issued by nearby supermarkets, including advertisements and special sale information.

[0378] "Image recognition" refers to the technology of extracting specific information from digital images.

[0379] A "menu" refers to a meal plan or menu for a specific period of time (for example, a week).

[0380] "Receipt" means a paper or digital receipt listing the details and price of purchased items.

[0381] "Product Name" refers to the name of the purchased product.

[0382] "Price Information" refers to information regarding the price of a purchased item.

[0383] "Dietary Data" refers to information such as a user's dietary preferences, restrictions, and past eating history.

[0384] A "generative AI model" refers to a model that uses artificial intelligence to perform a specific task (in this case, generating a menu).

[0385] "Calorie counting" refers to the process of calculating the total calories in a meal.

[0386] "Food cost management" refers to the process of calculating the costs of ingredients and meals and keeping them within a budget.

[0387] "Lifestyle" refers to a user's daily life patterns and habits.

[0388] "Menu modification" refers to the process of changing an already planned menu.

[0389] The system for carrying out the present invention has a plurality of functions for comprehensively supporting the user's eating habits. Specific embodiments of the system will be described below.

[0390] 1. Register your family's preferences, allergy information, health status, refrigerator contents, and repertoire

[0391] Through a smartphone application, users register their family's preferences, allergies, health conditions, what's in their refrigerator, and the recipes they can make, allowing the system to obtain detailed data about the user's eating habits.

[0392] 2. Importing supermarket flyer information

[0393] The system uses the smartphone camera to capture flyer information from nearby supermarkets using image recognition technology (such as OpenCV), allowing users to obtain the latest sales and product information.

[0394] 3. Automatic menu generation

[0395] Based on the user's dietary data, a generative AI model is used to automatically generate a weekly menu. Specifically, a request is sent to the AI ​​model using the following prompt:

[0396] Example prompt sentence:

[0397] User dietary data: {'preferences': ['vegetarian', 'low-carb'], 'allergies': ['peanuts'], 'budget': 5000}

[0398] Please suggest a menu for the week.

[0399] 4. Reading receipts and extracting price information

[0400] After a user shops at a supermarket, they take a photo of the receipt with their smartphone camera, and the system uses pytesseract to extract product names and prices from the receipt image.

[0401] 5. Calorie counting and food budget management

[0402] Based on the generated menu and receipt data, the system calculates calories and manages food costs, allowing users to easily understand the total calories and costs.

[0403] 6. Menu Modification

[0404] The suggested menu can be adjusted as needed to suit the user's lifestyle. For example, if there is a change in your schedule, such as "I've decided to eat out today," you can enter that information into the system and the menu will be automatically updated.

[0405] Hardware and software used

[0406] Hardware: Smartphone (with camera)

[0407] software:

[0408] OpenCV: Image processing library

[0409] pytesseract: an OCR (optical character recognition) library

[0410] requests: HTTP request library

[0411] Specific examples

[0412] For example, if a user is vegetarian, prefers low-carb meals, and has a budget of 5,000 yen, the system will operate as follows:

[0413] 1. Users register their dietary data through a smartphone application.

[0414] 2. Supermarket flyer information is captured by a camera and analyzed using image recognition technology.

[0415] 3. Send prompts to the generative AI model to automatically generate a weekly menu.

[0416] 4. After shopping at the supermarket, take a photo of the receipt and extract product name and price information.

[0417] 5. Calculate calories and manage food costs based on the generated menu and receipt data.

[0418] 6. Modify the menu as needed to suit the user's lifestyle.

[0419] In this way, users can significantly reduce the effort required for planning meals and purchasing ingredients, and can easily manage food expenses and calculate calories. They can also flexibly modify their meals to suit their daily lifestyles.

[0420] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0421] Step 1:

[0422] Through a smartphone application, users register their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make.

[0423] Input: Family preferences, allergy information, health status, refrigerator contents, repertoire

[0424] Output: Registered dietary data

[0425] Specific operation: The user enters the required information into the application's input form and sends it to the server, which then stores this data in a database.

[0426] Step 2:

[0427] The device uses the smartphone camera to capture flyer information from nearby supermarkets using image recognition technology (such as OpenCV).

[0428] Input: Supermarket flyer image

[0429] Output: Analyzed flyer information (product name, price, special sale information, etc.)

[0430] How it works: A user takes a photo of a flyer with their smartphone camera, and the device analyzes the image using OpenCV to extract product names and prices. The extracted information is then sent to the server.

[0431] Step 3:

[0432] The server automatically generates a weekly menu using a generative AI model based on the user's dietary data.

[0433] Input: User's dietary data, flyer information

[0434] Output: Weekly menu

[0435] Specific operation: The server sends the following prompt sentence to the generation AI model to generate a menu.

[0436] User dietary data: {'preferences': ['vegetarian', 'low-carb'], 'allergies': ['peanuts'], 'budget': 5000}

[0437] Please suggest a menu for the week.

[0438] The generated menu is stored on the server and notified to the user.

[0439] Step 4:

[0440] After a user shops at a supermarket, they take a photo of the receipt with their smartphone camera, and the device uses pytesseract to extract product names and prices from the receipt image.

[0441] Input: Receipt image

[0442] Output: Product name and price information

[0443] How it works: A user takes a photo of a receipt with their smartphone camera, and the device uses pytesseract to extract text from the image, then sends the extracted product name and price information to the server.

[0444] Step 5:

[0445] The server calculates calories and manages food costs based on the generated menu and receipt data.

[0446] Input: Menu, receipt data

[0447] Output: Total calories, total cost

[0448] Specific operation: The server calculates the calories of each dish in the menu and calculates the total cost from the receipt data. The calculation result is notified to the user.

[0449] Step 6:

[0450] The proposed menu can be modified as needed to suit the user's lifestyle.

[0451] Input: User's lifestyle changes

[0452] Output: Modified menu

[0453] Specific operation: The user inputs changes such as "I've decided to eat out today" through the application, and the server automatically updates the menu based on that information. The user is notified of the updated menu.

[0454] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0455] "Example 1"

[0456] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The emotion engine recognizes a user's emotions and adjusts menus based on the emotions. Specifically, when the user is feeling stressed, the emotion engine collects that information and suggests menus that include ingredients that reduce stress. For example, if the emotion engine recognizes that the user is feeling stressed, it suggests menus that include ingredients that are effective in reducing stress, such as chocolate and nuts.

[0457] "Example 2"

[0458] The emotion engine also adjusts food purchase suggestions based on the user's emotions. For example, when a user is feeling happy, the emotion engine collects that information and suggests meals that include the user's favorite ingredients. Specifically, if the emotion engine recognizes that the user is feeling happy due to a birthday or anniversary, it will suggest meals that include cake and celebratory meals.

[0459] "Example 3"

[0460] In addition, the emotion engine will adjust the meal suggestions based on the user's emotions. For example, when the user is feeling sad, the emotion engine will collect that information and suggest comforting meals. Specifically, if the emotion engine recognizes that the user is feeling sad, it will suggest comforting meals such as hot soup or porridge.

[0461] "Example 10"

[0462] The emotion engine also adjusts food purchase suggestions based on the user's emotions. For example, if the user is feeling angry, the emotion engine will collect that information and suggest a meal plan that includes ingredients that will calm the anger. Specifically, if the emotion engine recognizes that the user is feeling angry, it will suggest a meal plan that includes relaxing drinks such as mint tea or chamomile tea.

[0463] "Example 11"

[0464] Furthermore, the emotion engine adjusts the menu suggestions based on the user's emotions. For example, when the user is feeling surprised, the emotion engine will collect that information and suggest menus that include ingredients that provide a fresh surprise. Specifically, if the emotion engine recognizes that the user is feeling surprised, it will suggest menus that include exotic fruits and unusual dishes.

[0465] The processing flow of each embodiment will be described below.

[0466] "Example 1"

[0467] Step 1: The emotion engine recognizes the user's emotion.

[0468] Step 2: The emotion engine determines that the emotion it recognizes is stress.

[0469] Step 3: The emotion engine suggests meals that include ingredients that reduce stress.

[0470] "Example 2"

[0471] Step 1: The emotion engine recognizes the user's emotion.

[0472] Step 2: The emotion engine determines that the recognized emotion is joy.

[0473] Step 3: The emotion engine suggests meals that include ingredients the user likes.

[0474] "Example 3"

[0475] Step 1: The emotion engine recognizes the user's emotion.

[0476] Step 2: The emotion engine determines that the recognized emotion is sadness.

[0477] Step 3: The emotional engine suggests comfort food.

[0478] "Example 10"

[0479] Step 1: The emotion engine recognizes the user's emotion.

[0480] Step 2: The emotion engine determines that the recognized emotion is anger.

[0481] Step 3: The emotion engine suggests meals that include ingredients that calm anger.

[0482] "Example 11"

[0483] Step 1: The emotion engine recognizes the user's emotion.

[0484] Step 2: The emotion engine determines that the emotion recognized is surprise.

[0485] Step 3: The emotion engine suggests menus that include ingredients that offer fresh surprises.

[0486] Example 1

[0487] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0488] While conventional menu suggestion systems can take into account family preferences, allergy information, health status, the contents of the refrigerator, and the user's cooking repertoire, they have the problem of not being able to suggest menus that take into account the user's emotional state.In particular, if the user is feeling stressed or in a particular emotional state, it is not possible to suggest ingredients or dishes that correspond to that emotion, making it difficult to increase user satisfaction.

[0489] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0490] In this invention, the server includes means for registering family preferences, allergy information, health condition, contents of the refrigerator, and the user's cooking repertoire, means for capturing flyer information from nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for recognizing the user's emotions and adjusting the menu based on those emotions, means for generating an optimal menu taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and the user's cooking repertoire, and means for presenting the generated menu to the user. This makes it possible to propose a menu that takes into account the user's emotional state.

[0491] "Family preferences" is information about ingredients and dishes that the user's family particularly likes or dislikes.

[0492] "Allergy information" is information about food allergies that the user or their family members have.

[0493] "Health Status" is information about the health of the user and their family, including specific illnesses and health restrictions.

[0494] "Refrigerator contents" refers to information about ingredients and foods currently stored in the user's refrigerator.

[0495] The "repertoire of dishes that can be made" is information about the types of dishes and recipes that the user can make by himself.

[0496] "Flyer information from nearby stores" is information about sales and special offers offered by stores in the user's area.

[0497] "Image recognition" is a technology that analyzes digital images to extract specific information.

[0498] A "menu" is a meal plan or menu for a specific period of time.

[0499] "Emotion recognition" means analyzing the user's facial expression, tone of voice, etc. to identify their emotional state at that time.

[0500] "Adjusting menus based on emotions" means suggesting appropriate ingredients and dishes taking into account the user's emotional state.

[0501] "Generating the optimal menu" means creating the most suitable meal plan by comprehensively taking into consideration the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and repertoire of dishes they can make.

[0502] "Presenting the created menu to the user" means displaying the created menu to the user.

[0503] This invention is a system that allows users to register their family's preferences, allergy information, health status, refrigerator contents, and cooking repertoire, and then suggests optimal menus based on these.It also has the function of recognizing the user's emotional state and adjusting the menu based on that emotion.

[0504] Hardware and software used

[0505] Hardware:

[0506] Terminal: The device through which the user enters information (smartphone, tablet, PC, etc.)

[0507] Camera: A device for recognizing the user's facial expressions

[0508] Microphone: A device that recognizes the tone of a user's voice.

[0509] Server: A computer system for storing data, analyzing data, and generating menus

[0510] software:

[0511] Database Management System: Software for storing information entered by users

[0512] Image recognition software: Software for capturing flyer information for nearby stores

[0513] Emotion Recognition Engine: Software for analyzing user emotions

[0514] Menu Generation Algorithm: Software for generating optimal menus based on user information and emotional state

[0515] Program processing

[0516] User enters information:

[0517] Using a dedicated application or web interface, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. For example, a user might enter information such as "I like fish dishes," "I'm allergic to shrimp," "I'm diabetic," "chicken breast, broccoli, and eggs," or "sautéed chicken breast, sautéed broccoli with garlic, and rolled eggs."

[0518] The server stores the information in a database:

[0519] The server receives the information entered by the user and stores it in a database, including family preferences, allergies, health conditions, refrigerator contents, and cooking repertoire, so that the information can be referenced later.

[0520] Recognize user emotions:

[0521] The user provides the system with their facial expressions and tone of voice using a camera and microphone. The device collects this data and sends it to the server. For example, if the user is feeling stressed, their facial expression may become tense and their voice tone may become lower.

[0522] The server analyzes the emotion data:

[0523] The server analyzes the received facial expression and tone of voice data to identify the user's emotional state. The emotion engine uses a machine learning model to determine whether the user is feeling stressed. For example, if the user's facial expression is tense, the emotion engine will determine that the user is "stressed."

[0524] The server generates the menu:

[0525] The server generates the optimal menu by taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and available cooking repertoire. For example, if the user is feeling stressed, the server will suggest a menu that includes ingredients that are effective in reducing stress, such as chocolate and nuts.

[0526] The server presents the menu to the user:

[0527] The server then presents the generated menu to the user, who can view the suggested menu through the application or web interface. For example, a menu item might include "sauteed chicken breast," "sauteed broccoli with garlic," and "chocolate dessert."

[0528] Examples and prompts

[0529] Examples:

[0530] The user registers the following information:

[0531] Family preferences: Likes fish dishes, doesn't like spicy food

[0532] Allergy Information: Shrimp allergy

[0533] Health Condition: Diabetes

[0534] What's in the fridge: Chicken breast, broccoli, eggs

[0535] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[0536] If the server determines that the user is feeling stressed, it generates the following menu and presents it to the user:

[0537] Sautéed chicken breast

[0538] Garlic sauteed broccoli

[0539] Chocolate dessert

[0540] Example prompt sentence:

[0541] "When a user is feeling stressed, please suggest a meal that will be effective in reducing stress, taking into account the contents of the refrigerator and the repertoire of dishes that can be made."

[0542] In this way, the server provides a system that suggests menus that take into account the user's emotional state.

[0543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0544] Step 1:

[0545] The user enters the information.

[0546] Using a dedicated application or web interface, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. The input information is saved in text format on the device. For example, a user might input information such as "I like fish dishes," "I'm allergic to shrimp," "I have diabetes," "chicken breast, broccoli, and eggs," or "sauteed chicken breast, sauteed broccoli with garlic, and rolled omelet."

[0547] Input: Family preferences, allergy information, health status, contents of refrigerator, and cooking repertoire

[0548] Output: The entered information is saved on the device.

[0549] Step 2:

[0550] The server stores the information in a database.

[0551] The device sends the information entered by the user to the server, which then stores the information in a database. This information includes family preferences, allergies, health conditions, the contents of the refrigerator, and the recipes available for cooking. This allows the information to be referenced later.

[0552] Input: User information sent from the device

[0553] Output: User information stored in the database

[0554] Step 3:

[0555] Recognize user emotions

[0556] The user provides the system with their facial expressions and tone of voice using a camera and microphone. The device collects this data and sends it to the server. For example, if the user is feeling stressed, their facial expression may become tense and their voice tone may become lower.

[0557] Input: User's facial expression data, voice tone data

[0558] Output: Emotion data sent to the server

[0559] Step 4:

[0560] The server analyzes the emotional data

[0561] The server analyzes the received facial expression and tone of voice data to identify the user's emotional state. The emotion engine uses a machine learning model to determine whether the user is feeling stressed. For example, if the user's facial expression is tense, the emotion engine will determine that the user is "stressed."

[0562] Input: User's facial expression data, voice tone data

[0563] Output: Parsed user's emotional state

[0564] Step 5:

[0565] The server generates the menu

[0566] The server generates the optimal menu by taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and available cooking repertoire. For example, if the user is feeling stressed, the server will suggest a menu that includes ingredients that are effective in reducing stress, such as chocolate and nuts.

[0567] Input: User's emotional state, family preferences, allergy information, health status, refrigerator contents, and cooking repertoire

[0568] Output: Generated menu

[0569] Step 6:

[0570] The server presents the menu to the user

[0571] The server then presents the generated menu to the user, who can view the suggested menu through the application or web interface. For example, a menu item might include "sauteed chicken breast," "sauteed broccoli with garlic," and "chocolate dessert."

[0572] Input: Generated menu

[0573] Output: The menu presented to the user

[0574] (Application example 1)

[0575] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0576] While conventional menu suggestion systems can take into account the user's family preferences, allergy information, health status, and the contents of the refrigerator, they have the problem of not being able to suggest menus that reflect the user's emotional state. It is also difficult to link with delivery services and provide the optimal meal based on the user's emotions.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire, means for capturing information about nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for recognizing emotions, means for adjusting the menu based on emotions, means for providing a delivery service, and means for generating an optimal menu using a generative AI model. This makes it possible to propose an optimal menu that reflects the user's emotional state and provide a delivery service.

[0578] "Family preferences" refers to the types of ingredients and dishes that the user's family prefers.

[0579] "Allergy Information" refers to information about allergies held by the User or their family members.

[0580] "Health Status" refers to information about the health of a user or their family members, such as medical conditions such as diabetes or high blood pressure.

[0581] "What's in the refrigerator" refers to the ingredients and food currently stored in the user's refrigerator.

[0582] "Repertoire you can make" refers to the types of dishes that users can cook themselves.

[0583] "Information about nearby stores" refers to sales and special offers from stores near the user's residence.

[0584] "Image recognition" refers to the technology of analyzing image data and extracting specific information.

[0585] A "menu" refers to a meal plan or menu for a specific period of time.

[0586] "Emotion recognition" refers to the technology of analyzing and recognizing a user's emotions from facial expressions, voice, etc.

[0587] "Adjusting menus based on emotions" refers to modifying or optimizing menus by taking into account the user's emotional state.

[0588] "Delivery Service" refers to a service that delivers meals ordered by a user to a specified location.

[0589] A "generative AI model" refers to a model that uses artificial intelligence to perform specific tasks.

[0590] "Generating the optimal menu" refers to suggesting the most suitable menu based on the user's registered information and emotional state.

[0591] The system for implementing this invention registers the user's family preferences, allergy information, health status, contents of the refrigerator, and the repertoire of dishes they can make, recognizes emotions, suggests optimal menus, and provides delivery services.

[0592] Hardware and software used

[0593] Hardware: Smartphone

[0594] software:

[0595] Emotion recognition API (e.g., Microsoft® Azure® Emotion API)

[0596] Database (e.g. Firebase)

[0597] Food delivery API (e.g. Uber Eats API)

[0598] Generative AI models (e.g., OpenAI GPT-4)

[0599] System configuration and operation

[0600] 1. User information registration:

[0601] Users use a smartphone application to register their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make.

[0602] The registered information is stored in the Firebase database.

[0603] 2. Emotion recognition:

[0604] Users take a picture of their face using their smartphone camera and the emotion is recognized using the Microsoft Azure Emotion API.

[0605] The recognized emotion data is sent to the application in real time.

[0606] 3. Menu suggestions:

[0607] Based on the recognized emotion data and registered user information, the optimal menu is generated using a generative AI model (OpenAI GPT-4).

[0608] For example, if a user is feeling stressed, it will suggest meals that include chocolate and nuts.

[0609] 4. Delivery Service:

[0610] Based on the proposed menu, it uses a food delivery API (Uber Eats API) to search for corresponding dishes and provide delivery options.

[0611] Users can complete their orders within the app.

[0612] Specific examples

[0613] User Information:

[0614] Family preferences: Likes fish dishes, doesn't like spicy food

[0615] Allergy Information: Shrimp allergy

[0616] Health Condition: Diabetes

[0617] What's in the fridge: Chicken breast, broccoli, eggs

[0618] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[0619] Emotional state:

[0620] Feeling stressed

[0621] Prompt Sentence Examples

[0622] User Information:

[0623] Family preferences: Likes fish dishes, doesn't like spicy food

[0624] Allergy Information: Shrimp allergy

[0625] Health Condition: Diabetes

[0626] What's in the fridge: Chicken breast, broccoli, eggs

[0627] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[0628] Emotional state:

[0629] Feeling stressed

[0630] Use this information to suggest a menu that includes ingredients that are effective in reducing stress.

[0631] This system makes it possible to provide optimal menu suggestions and delivery services that reflect the user's emotional state.

[0632] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0633] Step 1:

[0634] Users use a smartphone application to register their family's preferences, allergies, health conditions, the contents of their refrigerator, and the recipes they can make. The information entered is stored in the Firebase database. Specifically, when a user enters information into the application's input form and presses the submit button, the data is sent to and stored in Firebase.

[0635] Step 2:

[0636] The user takes a picture of their face using their smartphone camera, and the emotion is recognized using the emotion recognition API (Microsoft Azure Emotion API). The input is the user's face image, and the output is emotion data. Specifically, when the user activates the camera function in the app and takes a picture of their face, the image is sent to the emotion recognition API, and emotion data is returned.

[0637] Step 3:

[0638] The server uses a generative AI model (OpenAI GPT-4) to generate an optimal menu based on the emotion data received from the emotion recognition API and the user information stored in the Firebase database. The input is emotion data and user information, and the output is the optimal menu. Specifically, the server converts the emotion data and user information into prompt sentences and sends them to the generative AI model to generate the menu.

[0639] Step 4:

[0640] Based on the generated menu, the server uses a food delivery API (Uber Eats API) to search for corresponding dishes and provide delivery options. The input is the generated menu and the output is the delivery options. Specifically, the server sends the generated menu to the food delivery API and obtains the corresponding dishes and delivery options.

[0641] Step 5:

[0642] The user checks the delivery options and completes the order within the app. The input is the delivery option, and the output is a notification that the order has been completed. Specifically, when the user selects a delivery option within the app and presses the order button, the order is sent to the food delivery service and a notification that the order has been completed is displayed to the user.

[0643] Example 2

[0644] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0645] While conventional menu suggestion systems can take into account family preferences, allergies, health conditions, and the contents of the refrigerator, they have difficulty incorporating the latest sales information from nearby stores in real time. Furthermore, they lack the functionality to adjust ingredient purchasing suggestions based on the user's emotions, which prevents them from increasing user satisfaction. Furthermore, they lack the functionality to automatically extract product names and prices from advertising images and store them in a database, which requires time-consuming manual input. A system that can solve these issues is needed.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0647] In this invention, the server includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of meals the user can make, means for capturing advertising information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where to buy it, means for detecting the user's emotions and adjusting ingredient purchasing suggestions based on the emotions, means for extracting product names and price information from advertising images using image recognition software, means for saving the extracted product names and price information in a database, means for detecting the user's emotions using emotion recognition software, and means for generating a menu based on the detected emotion information and suggesting it to the user. This allows the user to obtain the latest sales information from nearby stores in real time and receive ingredient purchasing suggestions based on their emotions.

[0648] "Family preferences" refer to the types of ingredients and dishes that each family member particularly likes.

[0649] "Allergy information" refers to information about food allergies that each member of the family has.

[0650] "Health Status" refers to information about each family member's current health status and specific dietary restrictions.

[0651] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[0652] "Repertoire" refers to the list of types of dishes or menus that a user can cook.

[0653] "Store advertising information" refers to product and price information contained in flyers and advertisements issued by nearby stores.

[0654] "Image recognition" refers to technology that automatically extracts specific information from image data.

[0655] A "menu" refers to a meal plan or menu list for a specific period of time (e.g., a week).

[0656] "Emotion detection" refers to technology that identifies a user's emotional state from their facial expressions and behavior.

[0657] "Ingredient purchase suggestions" refers to suggesting ingredients and menu items to purchase based on the user's preferences and emotional state.

[0658] "Image recognition software" refers to a program that analyzes image data and extracts specific information.

[0659] "Product name" refers to the name of the product as it appears in advertisements and flyers.

[0660] "Price information" refers to the product prices listed in advertisements and flyers.

[0661] A "database" refers to a system that stores information in an organized manner and allows it to be searched and updated as needed.

[0662] "Emotion recognition software" refers to programs designed to detect a user's emotions.

[0663] "Generate menu" refers to creating an appropriate menu based on the user's preferences and emotional state.

[0664] This invention is a system that allows users to obtain the latest sales information from nearby stores in real time and receive emotion-based food purchasing suggestions. Specific embodiments of this system are described below.

[0665] 1. System Configuration

[0666] This system consists of the user's smartphone (terminal), a server, image recognition software, emotion recognition software, and a database.

[0667] 2. Importing flyer information using image recognition

[0668] A user uses the camera of a smartphone to take a picture of a flyer from a nearby store. For example, if a user finds a sale flyer from a nearby store, the user takes a picture of the flyer with the smartphone.

[0669] The device then sends the flyer image to a server via the Internet in an appropriate format (e.g., JPEG, PNG).

[0670] The server analyzes the received images using image recognition software (e.g., Google Cloud Vision API), specifically using the API to extract text within the image and identify product names and pricing information.

[0671] The server stores the extracted product name and price information in a database. For example, information such as "apples - 100 yen" and "bananas - 150 yen" is stored.

[0672] 3. Food purchasing suggestions based on an emotional engine

[0673] The device detects the user's emotions using emotion recognition software (e.g., Microsoft Azure Emotion API). For example, the camera automatically recognizes the user's face and analyzes their emotions while they are using the smartphone.

[0674] The device sends the detected emotional information (e.g., joy, sadness) to the server. This information is sent as data indicating the user's current emotional state.

[0675] The server generates a menu that includes ingredients that the user likes based on the received emotion information. For example, if the user is feeling happy, the server generates a menu that includes cake and celebratory food.

[0676] The server sends the generated menu to the device and displays it to the user, who can then view the suggested menu on their smartphone screen.

[0677] 4. Specific Examples

[0678] Importing flyer information using image recognition

[0679] The user takes a photo of a flyer at a nearby store using the smartphone camera.

[0680] The terminal transmits the captured image to the server.

[0681] The server uses the Google Cloud Vision API to analyze the image and extract product name and price information.

[0682] The server stores the extracted information in a database.

[0683] Food purchasing suggestions based on an emotion engine

[0684] A user uses a smartphone while preparing for a birthday party.

[0685] The device uses the Microsoft Azure Emotion API to detect the user's emotions and recognizes that they are feeling joy.

[0686] The terminal transmits the emotion information to the server.

[0687] The server suggests recipes that include the user's favorite ingredients and generates a menu that includes cake and a celebratory meal.

[0688] The server sends the suggested menu to the terminal and displays it to the user.

[0689] 5. Examples of prompts

[0690] "I took a picture of a flyer from a nearby store. Use image recognition to extract product name and price information."

[0691] “You recognize that users feel joy when they are preparing for a birthday party. Use your emotion engine to suggest menu ideas that include cake and celebratory food.”

[0692] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0693] Step 1:

[0694] A user uses the smartphone camera to take a photo of a flyer for a nearby store. The input is the flyer image taken with the smartphone camera. The output is the flyer image saved on the smartphone. Specifically, the user launches the camera app, frames the flyer, and presses the shutter button.

[0695] Step 2:

[0696] The device sends the flyer image it has taken to a server via the Internet. The input is the flyer image stored on the smartphone. The output is the flyer image sent to the server. Specifically, the device converts the image to JPEG format and uploads it to the server using an HTTP request.

[0697] Step 3:

[0698] The server uses image recognition software (e.g., Google Cloud Vision API) to analyze the received image. The input is the flyer image sent to the server. The output is the extracted product name and price information. Specifically, the server calls the API, extracts the text in the image, and identifies the product name and price information.

[0699] Step 4:

[0700] The server saves the extracted product name and price information in a database. The input is the extracted product name and price information. The output is the product name and price information saved in the database. Specifically, the server establishes a database connection and inserts the information using an SQL query.

[0701] Step 5:

[0702] The device detects the user's emotions using emotion recognition software (e.g., Microsoft Azure Emotion API). The input is the user's facial image or video feed. The output is the detected emotion information. Specifically, the device uses a camera to capture the user's face and calls the API to analyze the emotion.

[0703] Step 6:

[0704] The device sends the detected emotion information to the server. The input is the detected emotion information. The output is the emotion information sent to the server. Specifically, the device converts the emotion information into JSON format and sends it to the server using an HTTP request.

[0705] Step 7:

[0706] The server generates a menu containing the user's favorite ingredients based on the received emotional information. The input is the emotional information sent to the server and the product information stored in the database. The output is the generated menu. Specifically, the server analyzes the emotional information and selects an appropriate menu based on the user's preferences.

[0707] Step 8:

[0708] The server sends the generated menu to the device and displays it to the user. The input is the generated menu. The output is the menu displayed on the device. Specifically, the server converts the menu information into JSON format and sends it to the device using an HTTP request. The user can view the suggested menu on their smartphone screen.

[0709] (Application example 2)

[0710] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0711] Conventional food purchasing support systems lack suggestions based on the user's emotions and individual preferences, making it difficult to increase user satisfaction. Furthermore, there are insufficient methods for efficiently importing supermarket flyer information and extracting product name and price information, making it difficult to quickly obtain the latest sales information. Furthermore, since there are no systems that make food purchasing suggestions based on the user's emotions, there is also the issue of being unable to make suggestions suitable for special occasions or events.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0713] In this invention, the server includes a means for registering family preferences, allergy information, health status, refrigerator contents, and recipes the user can make; a means for importing flyer information from nearby stores using image recognition; a means for suggesting weekly menus and what to buy and where; a means for analyzing the user's emotions and making ingredient purchasing suggestions based on those emotions; and a means for extracting product names and price information from flyer images captured with a smartphone camera. This allows for ingredient purchasing suggestions based on the user's emotions and individual preferences, making suggestions suitable for special occasions and events. It also allows for the rapid acquisition of the latest sales information, enabling efficient ingredient purchasing.

[0714] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0715] "Allergy information" refers to the type and severity of food allergies each family member has.

[0716] "Health status" refers to information about the health of each family member, including restrictions on the intake of certain foods and nutrients.

[0717] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[0718] "Repertoire you can make" refers to the types of dishes and recipes that a user can cook.

[0719] "Flyer information from nearby dealers" refers to the content of advertisements and flyers containing the latest sales information from dealers frequently used by the user.

[0720] "Image recognition" refers to the technology of extracting specific information from images taken with a camera.

[0721] A "weekly menu" refers to a meal plan for one week.

[0722] "Means of suggestion" refers to methods and techniques for suggesting the best ingredients and places to purchase them to users.

[0723] "Analyzing user emotions" refers to technology that determines a user's emotional state from their facial expressions and behavior.

[0724] "Emotion-based food purchasing suggestions" refers to suggesting appropriate ingredients and dishes based on the user's emotional state.

[0725] "Flyer image taken with a smartphone camera" refers to an image of a retailer's flyer taken with a smartphone camera.

[0726] "Means for extracting product name and price information" refers to methods and technologies for extracting product name and price information from flyer images using image recognition technology.

[0727] The system for implementing this invention is configured as follows: First, the user uses a smartphone to register information about their family's preferences, allergies, health conditions, what's in their refrigerator, and the recipes they can make, allowing the system to obtain individual information about the user.

[0728] Next, the user uses their smartphone camera to take a photo of a nearby store flyer, and the system uses image recognition technology to extract product names and prices from the flyer image, using OpenCV and pytesseract.

[0729] Additionally, the system takes a photo of the user's face and performs emotion analysis using Microsoft Azure's Face API, which allows it to determine the user's emotional state and make recommendations for food purchases based on their emotions.

[0730] The system uses this information to create a weekly menu and suggests what to buy and where to buy it, including ingredients and dishes suitable for special occasions and events, taking into account the user's emotional state and individual preferences.

[0731] For example, consider a case where a user photographs a supermarket flyer for a birthday and the system suggests a cake and a celebratory meal. In this case, sentiment analysis determines that the user is happy, so the system suggests special ingredients and a menu.

[0732] An example prompt might look like this:

[0733] "The user takes a photo of a supermarket flyer with their smartphone camera, and the system extracts product names and prices. It then takes a photo of the user's face and performs sentiment analysis. If the user is happy, it will suggest special ingredients and menu items."

[0734] In this way, it is possible to make food purchase suggestions based on the user's emotions and individual preferences, making suggestions suitable for special occasions and events.It also makes it possible to quickly obtain the latest sales information and purchase ingredients efficiently.

[0735] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0736] Step 1:

[0737] Users use their smartphones to register their family's preferences, allergy information, health conditions, what's in the refrigerator, and the recipes they can make.

[0738] Input: Family preferences, allergy information, health status, contents of the refrigerator, and recipes you can make

[0739] Output: Registered user information

[0740] Specific operation: The user starts the application on their smartphone and enters the information into the input form. The entered information is sent to the server and stored in the database.

[0741] Step 2:

[0742] The user uses the smartphone camera to take a picture of a flyer from a nearby store.

[0743] Input: Retailer flyer image

[0744] Output: Photographed flyer image

[0745] Specific operation: The user uses the camera function of the smartphone to take a picture of the store flyer. The taken image is saved in the application.

[0746] Step 3:

[0747] The server uses image recognition technology to extract product name and price information from the flyer image.

[0748] Input: Photographed flyer image

[0749] Output: Extracted product name and price information

[0750] Specific operation: The server uses OpenCV and pytesseract to analyze the flyer image and extract product name and price information as text data. The extracted data is stored in a database.

[0751] Step 4:

[0752] The user takes a picture of their face using the smartphone camera.

[0753] Input: User's face image

[0754] Output: Photographed face image

[0755] Specific operation: The user takes a picture of their face using the camera function of their smartphone. The captured image is saved in the application.

[0756] Step 5:

[0757] The server performs sentiment analysis to determine the emotional state of the user.

[0758] Input: A photographed face image

[0759] Output: User's emotional state

[0760] How it works: The server uses Microsoft Azure's Face API to analyze facial images and obtain the user's emotional state (happiness, sadness, surprise, etc.). The obtained emotional data is stored in a database.

[0761] Step 6:

[0762] Based on the user's emotional state and registered information, the server suggests a weekly menu and what to buy and where to buy it.

[0763] Input: User's emotional state, registered information, extracted product name and price information

[0764] Output: Suggested menu and where to buy

[0765] Specific operation: The server uses a generative AI model to suggest optimal menus and purchasing locations based on the user's emotional state, family preferences, allergy information, health status, refrigerator contents, cooking repertoire, and extracted product names and prices. The suggested information is sent to the user's smartphone.

[0766] Step 7:

[0767] The user checks the suggested menu and purchasing locations and purchases the necessary ingredients.

[0768] Input: Suggested menu and where to buy

[0769] Output: Purchased ingredients

[0770] Specific operation: The user checks the suggested menu and purchasing locations through the smartphone application, and then actually goes to the store to purchase the necessary ingredients.

[0771] Example 3

[0772] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0773] In today's busy lifestyles, it is extremely time-consuming to efficiently plan menus and purchase the necessary ingredients while taking into account the preferences and health status of family members. Managing food expenses and calculating calories is also not easy. Furthermore, there is a demand for flexible menu suggestions that reflect the user's emotions, but this is difficult to achieve with conventional systems. A system that can solve these problems is needed.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0775] In this invention, the server includes means for registering family preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of meals the user can make, means for capturing flyer information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where, means for inputting the user's emotions and adjusting the menu based on those emotions, and means for modifying the menu by reflecting changes made by the user to the suggested menu and updating calorie calculations and food cost management. This enables the user to efficiently create menus and purchase the necessary ingredients, and also receives flexible menu suggestions based on emotions.

[0776] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0777] "Allergy information" refers to information about food allergies held by each member of the family.

[0778] "Health Status" is information about the health of each family member, including restrictions on certain foods and recommended nutrients.

[0779] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[0780] "Repertoire you can make" refers to a list of types of dishes and recipes that the user can cook.

[0781] "Store flyer information" refers to advertisements that include sales information and product information offered by nearby stores.

[0782] "Image recognition" refers to the technology of extracting specific information from image data.

[0783] A "weekly menu" refers to a meal plan for one week.

[0784] "Means of suggestion" refers to the method by which the system presents the user with the optimal menu and purchasing locations.

[0785] "User emotion" refers to the emotional state that a user is feeling.

[0786] "Means for adjusting menu based on emotions" refers to a method for changing the menu based on the user's emotional information.

[0787] "Calorie counting" refers to a method of calculating the calories contained in food.

[0788] "Food cost management" refers to methods for managing the costs of purchasing ingredients and food.

[0789] "Reading a receipt" refers to a method of extracting the information written on a receipt as digital data.

[0790] "Means for calculating purchase amount" refers to a method for calculating the amount of a purchase based on information extracted from a receipt.

[0791] "Means for modifying a menu" refers to a method for updating an existing menu to reflect user changes.

[0792] This invention is a system that allows users to efficiently create menus and purchase the necessary ingredients. This system includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of dishes that the user can make, means for capturing flyer information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where, means for inputting the user's emotions and adjusting the menu based on those emotions, and means for modifying the menu to reflect changes made by the user to the suggested menu, and updating calorie calculations and food cost management.

[0793] Hardware and software used

[0794] Hardware: Servers, smartphones, PCs

[0795] Software: Image recognition software (e.g., image recognition API), emotion engine (e.g., emotion analysis API), database management system (e.g., database software)

[0796] System program processing

[0797] Register user information and supermarket flyer information

[0798] Users log in to the system using their smartphones or PCs and enter their name, age, allergy information, food preferences, etc. In addition, users upload supermarket flyer information to the system in PDF or image format. The server stores this information in a database.

[0799] Automatic menu generation

[0800] The server uses an AI model to automatically generate a weekly menu based on the user's registration information and uploaded flyer information, and then suggests to the user a list of ingredients and where to purchase them based on the generated menu.

[0801] Receipt scanning and food cost management

[0802] After shopping, users take a photo of their receipt with their smartphone camera and upload it to the system. The device then sends the image of the receipt to the server, which uses image recognition software to extract product names and prices from the receipt image, calculates the user's food expenses, and displays them on the management screen.

[0803] Menu modification and management

[0804] The user inputs changes to the menu suggested by the system, such as "I'm going to eat out today." The server modifies the menu based on the user's changes and updates calorie calculations and food cost management.

[0805] Emotion-based menu suggestions

[0806] The user inputs their emotions into the system, such as "I feel sad today." The server uses an emotion engine to analyze the user's emotions and, based on the analysis results, suggests a menu that matches the user's emotions. For example, if the user is feeling sad, it suggests a hot soup; if the user is feeling angry, it suggests a relaxing drink; and if the user is feeling surprised, it suggests an exotic dish.

[0807] Specific examples

[0808] Example 1: Automatic menu generation and ingredient purchasing suggestions

[0809] Users log in to the system and register supermarket flyer information. The server automatically generates a weekly menu based on the registered information and suggests the necessary ingredients and where to purchase them.

[0810] Example prompt sentence:

[0811] "Please suggest a menu for this week. The supermarket flyer information is as follows..."

[0812] Example 2: Receipt scanning and food cost management

[0813] After a user has finished shopping at a supermarket, they take a photo of the receipt with their smartphone camera. The server extracts the product name and price information from the image and calculates the total price.

[0814] Example prompt sentence:

[0815] "Please scan the information on this receipt to calculate the purchase amount..."

[0816] Example 3: Emotion-based menu suggestions

[0817] Users input emotions into the system: for example, if a user types "I'm sad today," the server uses the emotion engine to suggest comforting meals such as hot soup or porridge.

[0818] Example prompt sentence:

[0819] "I'm feeling sad today. Please suggest a comfort meal."

[0820] This system allows users to significantly reduce the effort required for creating menus and purchasing ingredients, and further enriches their daily lives by receiving meal suggestions that correspond to their emotions. The flow of the specification process in Example 3 will be described with reference to FIG.

[0821] Step 1:

[0822] Users log in to the system using a smartphone or PC and enter their name, age, allergy information, food preferences, etc. This information is then stored in a database.

[0823] Input: User's personal information (name, age, allergy information, food preferences)

[0824] Output: User information stored in the database

[0825] Step 2:

[0826] Users upload supermarket flyer information to the system in PDF or image format, and the terminal sends the uploaded flyer information to the server.

[0827] Input: Supermarket flyer information (PDF or image format)

[0828] Output: Flyer information sent to the server

[0829] Step 3:

[0830] The server uses image recognition software to extract product names and prices from the flyers, and stores the extracted information in a database.

[0831] Input: Flyer information (PDF or image format)

[0832] Output: Product name and price information stored in the database

[0833] Step 4:

[0834] The server uses an AI model to automatically generate a weekly menu based on the user's registration information and the extracted flyer information, and the generated menu is then suggested to the user.

[0835] Input: User registration information, extracted flyer information

[0836] Output: Automatically generated weekly menu

[0837] Step 5:

[0838] After shopping, users take a photo of their receipt with their smartphone camera and upload it to the system. The device then sends the captured receipt image to the server.

[0839] Input: Receipt image

[0840] Output: Receipt image sent to the server

[0841] Step 6:

[0842] The server uses image recognition software to extract product names and prices from receipt images, which are then stored in a database and used to calculate the user's food costs.

[0843] Input: Receipt image

[0844] Output: Product name and price information stored in the database, calculated food costs

[0845] Step 7:

[0846] The user inputs changes to the menu suggested by the system, such as "I'm going to eat out today." The server modifies the menu based on the user's changes and updates calorie calculations and food cost management.

[0847] Input: User change information

[0848] Output: revised menu, updated calorie count and food budget

[0849] Step 8:

[0850] The user inputs their emotions into the system, such as "I feel sad today." The server uses an emotion engine to analyze the user's emotions and, based on the analysis results, suggests a menu that matches the user's emotions.

[0851] Input: User's emotional information

[0852] Output: Emotion-based meal suggestions

[0853] In this way, the system provides flexible menu suggestions and food purchasing support that are tailored to the user's lifestyle and emotions.

[0854] (Application example 3)

[0855] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0856] In today's busy lifestyles, it is extremely difficult to efficiently plan meals and purchase the necessary ingredients while taking into account the preferences and health status of family members. There is also a need for meal suggestions that reflect the user's emotions and efficient ingredient purchasing based on past purchase history. Furthermore, there is a need for comprehensive dietary management, including food cost management and calorie calculation.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended recipes; means for capturing advertising information for nearby stores using image recognition; means for proposing a weekly menu and recommendations on what to buy and where; means for recognizing the user's emotions and adjusting the menu based on those emotions; means for generating a menu based on past purchase history; and means for automatically ordering ingredients based on the generated menu. This allows users to efficiently create menus and purchase the necessary ingredients. It also enables meal suggestions based on the user's emotions and efficient ingredient purchasing based on past purchase history. Furthermore, comprehensive dietary management, including food cost management and calorie calculation, is possible.

[0858] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[0859] "Allergy information" refers to information about food allergies that each member of the family has.

[0860] "Health status" refers to information about the health of each family member, such as specific illnesses or health limitations.

[0861] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[0862] "Repertoire you can make" refers to the types of dishes and recipes that a user can cook.

[0863] "Advertising information from nearby stores" refers to advertisements and sale information provided by stores near the user's residence.

[0864] "Image recognition" refers to the technology of extracting specific information from image data.

[0865] A "weekly menu" refers to a meal plan for one week.

[0866] "Means of suggesting what to buy and where" refers to a function that suggests the best store to purchase the ingredients you need.

[0867] "Means for recognizing user emotions" refers to technology that analyzes and recognizes the user's emotional state.

[0868] "Means for adjusting menus based on emotions" refers to a function that changes menus according to the user's recognized emotions.

[0869] "Means for generating menus based on past purchase history" refers to a function that creates menus by referring to the user's past purchase history.

[0870] "Means for automatically ordering ingredients based on the created menu" refers to a function for automatically ordering ingredients required for the created menu.

[0871] The system for implementing this invention uses the following hardware and software: a smartphone, smart glasses, and a camera. The software uses OpenCV, Tesseract OCR, EmotionRecognizer, RecipeGenerator, and FoodDeliveryAPI.

[0872] The server provides a way for users to register their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the recipes they can make. Users input this information via their smartphone or smart glasses, allowing the system to generate menus tailored to the user's individual needs.

[0873] The server then provides a means to retrieve advertisement information from nearby stores using image recognition. When a user takes a photo of a store advertisement using a smartphone or smart glasses camera, the advertisement information is analyzed using OpenCV and Tesseract OCR to extract the price and sale information of the desired ingredients.

[0874] Furthermore, the server provides a means to recognize the user's emotions and adjust the menu based on those emotions. When the user takes a picture of their face using the camera on their smartphone or smart glasses, the EmotionRecognizer analyzes the image and recognizes the user's emotions. Based on the recognized emotions, the RecipeGenerator generates an appropriate menu.

[0875] The server also provides a means to generate menus based on past purchase history. When a user takes a photo of a receipt from a past purchase with a smartphone or smart glasses camera, Tesseract OCR analyzes the image and extracts the purchase history. Based on this, the Recipe Generator generates an efficient menu.

[0876] Finally, the server provides a means to automatically order ingredients based on the generated menu. The ingredients required for the generated menu can be automatically ordered using the FoodDelivery API.

[0877] As a concrete example, when a user takes a photo of a receipt with their smartphone and uploads an emotional image, the system operates as follows: It extracts information such as "Tomatoes 100 yen" and "Milk 150 yen" from the receipt image and recognizes "sadness" from the emotional image. For users who feel sad, it suggests a menu including hot soup or porridge and automatically orders the necessary ingredients.

[0878] An example of a prompt sentence to input to the generative AI model is as follows:

[0879] "Generate meal suggestions for when the user is feeling sad. Past purchase history is as follows: Tomatoes 100 yen, Milk 150 yen. The suggested meal should include hot soup or porridge."

[0880] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0881] Step 1:

[0882] Users use their smartphones or smart glasses to input their family's preferences, allergy information, health conditions, what's in the refrigerator, and the recipes they can make. This information is sent to the server and stored in a database. The input data is saved in text format and used for subsequent processing.

[0883] Step 2:

[0884] Users use the camera on their smartphone or smart glasses to take a photo of an advertisement for a nearby store. The device then sends the image to the server. The server then analyzes the image using OpenCV and Tesseract OCR and extracts the advertisement information as text data. The extracted data includes product names, prices, and sale information.

[0885] Step 3:

[0886] The user takes a picture of their face using the camera on their smartphone or smart glasses. The device sends the captured facial image to the server. The server then analyzes the facial image using EmotionRecognizer and recognizes the user's emotions. The recognized emotions are saved in text format and used for subsequent processing.

[0887] Step 4:

[0888] The server uses RecipeGenerator to generate a weekly menu based on the user's emotions, family preferences, allergy information, health status, refrigerator contents, available cooking repertoire, and advertising information. The input data is passed to RecipeGenerator in text format, and a list of menus is generated as output.

[0889] Step 5:

[0890] Users take a photo of a receipt from a previous purchase using the camera on their smartphone or smart glasses. The device then sends the captured receipt image to the server. The server then analyzes the receipt image using Tesseract OCR and extracts the purchase history as text data. The extracted data includes product names, prices, purchase dates, and more.

[0891] Step 6:

[0892] The server uses RecipeGenerator to generate efficient menus based on past purchase history. Input data is passed to RecipeGenerator in text format, and a list of menus is generated as output.

[0893] Step 7:

[0894] The server automatically orders the necessary ingredients based on the generated menu using the FoodDelivery API. The input data is a list of menu items, and the output is order confirmation information, which is sent to the user's smartphone or smart glasses.

[0895] Step 8:

[0896] The user receives the order confirmation via their smartphone or smart glasses and can confirm the order details. If necessary, they can modify the order details. The modified order details are then sent back to the server, where the final order is confirmed.

[0897] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0898] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0899] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0900] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0901] [Second embodiment]

[0902] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0903] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0904] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0905] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0906] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0907] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0908] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0909] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0910] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0911] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0912] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0913] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0914] "Example 1"

[0915] In the system of the present invention, a user registers his or her family's preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes that the user can make. For example, the user can register "likes fish dishes" and "doesn't like spicy food" as family preferences, "shrimp allergy" as allergy information, "diabetes" as health status, "chicken breast, broccoli, eggs" as refrigerator contents, and "sauteed chicken breast, sauteed garlic broccoli, rolled omelet" as the repertoire of dishes that the user can make.

[0916] "Example 2"

[0917] The system of the present invention uses image recognition to capture flyer information from nearby supermarkets. For example, a user can take a photo of a supermarket flyer with a smartphone camera and extract product names and prices from the image. This allows users to easily obtain the latest sales information from their favorite supermarkets.

[0918] "Example 3"

[0919] The system of this invention will propose a weekly menu and what to buy where. For example, it will automatically generate a weekly menu based on registered information and supermarket flyers, and suggest the necessary ingredients and where to buy them. This will significantly reduce the effort required for users to create a menu and buy ingredients.

[0920] "Example 4"

[0921] The system of the present invention reads receipts from past purchases and calculates the amount of money required for each purchase. For example, after shopping at a supermarket, a user can take a photo of the receipt with a smartphone camera and extract product names and prices from the image. This allows users to easily manage their food expenses.

[0922] "Example 5"

[0923] The system of the present invention allows calorie calculation, food cost management, and menu planning to be modified as needed to suit actual circumstances. For example, if the user changes the proposed menu, such as by deciding to eat out, the menu is modified to reflect that information, and calorie calculation and food cost management are updated. This allows users to flexibly manage their meals in accordance with their daily lifestyle.

[0924] The processing flow of each embodiment will be described below.

[0925] "Example 1"

[0926] Step 1: The user accesses the system and enters their profile information, including family preferences, allergy information, and health conditions.

[0927] Step 2: Next, the user registers their cooking repertoire with the system, including the name of the dish and the ingredients required.

[0928] Step 3: Finally, the user registers the contents of their refrigerator with the system, which allows the system to suggest meals that take into account the ingredients already in their refrigerator.

[0929] "Example 2"

[0930] Step 1: The user takes a photo of a supermarket flyer with their smartphone camera.

[0931] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[0932] Step 3: The extracted information is stored in the system and used to suggest menus and where to buy ingredients.

[0933] "Example 3"

[0934] Step 1: The system automatically generates a weekly menu based on the registered information and supermarket flyer information.

[0935] Step 2: The generated menu and list of ingredients are presented to the user.

[0936] Step 3: The user reviews the proposed menu and makes any necessary modifications or approvals.

[0937] "Example 4"

[0938] Step 1: After shopping at the supermarket, the user takes a photo of the receipt with their smartphone camera.

[0939] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[0940] Step 3: The extracted information is stored in the system and used to manage food expenses.

[0941] "Example 5"

[0942] Step 1: The user enters any changes to the proposed menu into the system.

[0943] Step 2: The system will modify the menu to reflect the information you entered, and update calorie counts and food cost management.

[0944] Step 3: The updated information is notified to the user, who can then check the new menu and meal cost information.

[0945] Example 1

[0946] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0947] Conventional home cooking support systems have had difficulty proposing recipes that take into account family preferences, allergies, health conditions, the contents of the refrigerator, and the user's own cooking repertoire. Another problem was that they were unable to quickly generate appropriate recipes when users requested recipe suggestions. Furthermore, if the generated recipe did not meet the user's requirements, it required the user to re-enter the information, which was time-consuming.

[0948] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health conditions, refrigerator contents, and the user's own cooking repertoire, means for saving the registered information in a database, means for the user to request dish suggestions, means for acquiring user information from the database, means for creating a prompt sentence to be input to the generative AI model based on the acquired information, means for sending the prompt sentence to the generative AI model and generating a recipe, and means for displaying the generated recipe to the user. This makes it possible to quickly generate and display recipes that meet the user's requirements.

[0949] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family likes, and taste tendencies.

[0950] "Allergy information" is information about food allergies that the user or their family members have.

[0951] "Health Status" refers to information about the health of the user and their family, including specific illnesses and health care needs.

[0952] "Refrigerator contents" refers to information about ingredients and foods currently stored in the user's refrigerator.

[0953] "Repertoire that can be cooked by the user" is information on the types of dishes and recipes that the user can cook by himself.

[0954] The "means for registration" refers to a method or device that allows a user to input and save information about family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of dishes that the user can make into the system.

[0955] The "means for saving to a database" refers to a method or device for storing and managing information entered by a user in a database.

[0956] A "means for requesting dish suggestions" is a method or device that allows a user to request dish suggestions from the system.

[0957] The "means for obtaining user information from a database" refers to a method or device that allows the system to search and retrieve user information stored in a database.

[0958] "Means for creating prompt sentences to be input to a generative AI model" refers to a method or device for creating sentences to give appropriate instructions to a generative AI model based on acquired user information.

[0959] "Means for sending a prompt sentence to a generative AI model and generating a recipe" refers to a method or device for sending a created prompt sentence to a generative AI model and generating a recipe that meets the user's requirements.

[0960] "Means for displaying the generated recipe to the user" refers to a method or device for displaying the recipe generated by the generative AI model on the user's terminal.

[0961] This invention is a system that allows users to register their family's preferences, allergy information, health status, refrigerator contents, and cooking repertoire, and then suggests appropriate recipes based on that information. This system is implemented using a server, terminals, and a generative AI model.

[0962] First, users use a dedicated web form or application to enter their family's preferences (e.g., "I like fish dishes" or "I don't like spicy food"), allergy information (e.g., "I'm allergic to shrimp"), health conditions (e.g., "I'm diabetic"), the contents of their refrigerator (e.g., "chicken breast, broccoli, eggs"), and the repertoire of dishes they can make (e.g., "sauteed chicken breast, sauteed garlic broccoli, rolled omelet").

[0963] The terminal then converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The server then analyzes the received data and stores it in a relational database such as MySQL or PostgreSQL. For example, it organizes the information by user ID and stores the data corresponding to each field.

[0964] When a user requests a recipe suggestion, the server retrieves user information from a database. For example, it uses SQL queries to retrieve the user's family preferences, allergies, health status, refrigerator contents, and cooking repertoire. Based on this information, the server creates a prompt to be input to the generative AI model.

[0965] Examples of specific prompts include the following:

[0966] "My family likes fish dishes but doesn't like spicy food. I'm allergic to shrimp, and one family member is diabetic. We have chicken breasts, broccoli, and eggs in the refrigerator. I can make sautéed chicken breasts, sautéed garlic broccoli, and omelet. Based on this information, please suggest some suitable recipes."

[0967] The server sends this prompt to a generative AI model (for example, OpenAI's GPT-4) to generate a recipe. The generative AI model analyzes the prompt and generates a recipe that meets the user's requirements. The generated recipe is sent from the server to the device and displayed to the user. For example, the recipe title, ingredients, and instructions are displayed on the application screen.

[0968] The system allows users to quickly find the right recipe based on their family's preferences, health status, and what's in their refrigerator, and by using a generative AI model, it can generate recipes that meet the user's requirements with high accuracy.

[0969] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0970] Step 1: Enter your user information

[0971] Users use a dedicated web form or application to enter their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. The information is then stored in text format on the device.

[0972] Step 2: Sending data

[0973] The device converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The input data includes family preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes the user can make. The output data is the JSON format data sent to the server.

[0974] Step 3: Save your data

[0975] The server analyzes the received data and stores it in a relational database such as MySQL or PostgreSQL. The input data is user information in JSON format, and the output data is the user information stored in the database. Specifically, it executes an INSERT statement on the database.

[0976] Step 4: Request a recipe suggestion

[0977] A user requests a recipe suggestion by clicking a "recipe suggestion" button in an application or website. The input data is the user's request, and the output data is a request signal to the server.

[0978] Step 5: Get the data

[0979] The server retrieves user information from the database. The input data is the user ID, and the output data is the user's family preferences, allergy information, health status, contents of the refrigerator, and the repertoire of dishes they can make. Specifically, it executes a SELECT statement on the database.

[0980] Step 6: Create a prompt statement

[0981] The server creates a prompt sentence to be input to the generative AI model based on the acquired information. The input data is the acquired user information, and the output data is the prompt sentence to be sent to the generative AI model. Specifically, the server converts the user information into text format and constructs the prompt sentence.

[0982] Step 7: Generate the recipe

[0983] The server sends a prompt to the generative AI model to generate a recipe. The input data is the prompt, and the output data is the generated recipe. Specifically, the server sends a prompt to the generative AI model (e.g., OpenAI's GPT-4) and receives the generated recipe.

[0984] Step 8: Submit your recipe

[0985] The server converts the generated recipe into a data format such as JSON and sends it to the terminal. The input data is the generated recipe, and the output data is the JSON-formatted recipe sent to the terminal.

[0986] Step 9: View the recipe

[0987] The device displays the received recipe to the user. The input data is a recipe in JSON format, and the output data is the recipe information to be displayed to the user. Specifically, the application screen displays the recipe title, ingredients, instructions, etc.

[0988] (Application example 1)

[0989] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0990] Conventional food delivery services have difficulty suggesting optimal dishes that take into account the user's family preferences, allergies, and health conditions. Furthermore, they do not take into account the contents of the refrigerator or the user's cooking repertoire, making it difficult for users to use ingredients efficiently and maintain a healthy diet.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0992] In this invention, the server includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the recipes the user can make, means for importing flyer information from nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for generating prompts using a generative AI model to suggest optimal dishes, and means for ordering the ingredients needed for the suggested dishes. This enables the server to suggest optimal dishes tailored to the user's individual needs and to use ingredients efficiently.

[0993] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family prefers, as well as taste trends.

[0994] "Allergy information" is information about allergies that the user or their family members have.

[0995] "Health Status" means information about the health of a user or their family members, including specific illnesses or health care restrictions.

[0996] "What's in the refrigerator" is information about the ingredients currently stored in the user's refrigerator.

[0997] "Repertoire you can make" refers to information about the types of dishes and recipes that users can make themselves.

[0998] "Flyer information from nearby stores" is advertisements and sale information for products offered by stores in the user's neighborhood.

[0999] "Image recognition" is a technology that analyzes digital images to extract specific information.

[1000] A "generative AI model" is a model that has been trained using artificial intelligence to perform a specific task.

[1001] A "prompt" is text that is input to a generative AI model to produce a specific output.

[1002] The "optimal dish" is a dish suggested based on the user's family's preferences, allergy information, health condition, what's in the refrigerator, and the recipes they can make themselves.

[1003] "Means for ordering ingredients" is a function for ordering ingredients needed for the proposed dish online.

[1004] The system for implementing this invention is constructed as a smartphone application for the user. The system provides the user with the ability to register information about their family's preferences, allergies, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. This allows the user to receive suggestions for dishes that best suit their needs.

[1005] The server stores the information entered by the user in a database and generates a prompt using a generative AI model. Based on the generated prompt, the AI ​​model suggests the most suitable dish. The user can then order the ingredients needed for the suggested dish online.

[1006] Specifically, the server performs the following process.

[1007] 1. User data registration:

[1008] Through a smartphone application, users can input their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. This information is then sent to a server and stored in a database.

[1009] 2. Generate prompt sentence:

[1010] The server generates a prompt based on the stored user data, such as:

[1011] Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Shrimp allergy, Health condition: Diabetes, Refrigerator contents: Chicken breast, broccoli, eggs, Cooking skills: Sautéed chicken breast, Sautéed broccoli with garlic, Omelette

[1012] 3. Recipe suggestions using generative AI models:

[1013] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3) to suggest the best dish. The generative AI model generates a dish that matches the user's needs based on the prompt.

[1014] 4. Order ingredients:

[1015] The ingredients required for the proposed dishes can be ordered online by the user through a smartphone application, and the server provides the user with a list of the proposed dishes and the required ingredients, facilitating the ordering process.

[1016] This system allows users to receive recommendations for optimal meals that take into account their family's preferences and health conditions, allowing them to use ingredients efficiently.It also allows users to easily order the ingredients they need, reducing the hassle of shopping.

[1017] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1018] Step 1:

[1019] Through a smartphone application, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. The input data is sent from the application to a server. The server then stores the received data in a database. Examples of input data include "I like fish dishes," "I'm allergic to shrimp," "I'm diabetic," "chicken breast, broccoli, and eggs," and "sauteed chicken breast, sauteed broccoli with garlic, and rolled omelet."

[1020] Step 2:

[1021] The server generates prompts based on the saved user data. Prompts are text-format input data to be sent to the generative AI model. An example of a generated prompt is "Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Allergy to shrimp, Health condition: Diabetes, Contents of refrigerator: Chicken breast, Broccoli, Eggs, Cooked dishes: Sauteed chicken breast, Sauteed broccoli with garlic, Omelette."

[1022] Step 3:

[1023] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal dish based on the prompt to meet the user's needs. The output from the generative AI model is a list of suggested dishes. For example, it might suggest "teriyaki chicken breast" or "broccoli and egg salad."

[1024] Step 4:

[1025] The server receives the output from the generative AI model and sends a list of suggested dishes to the user's smartphone application, through which the user can review the suggested dishes.

[1026] Step 5:

[1027] The user selects from the proposed dishes and orders the necessary ingredients online. The application generates a list of ingredients required for the selected dish and provides an interface for the user to confirm the order. The server receives the order data and sends the order information to the corresponding store.

[1028] Step 6:

[1029] The store prepares ingredients based on the received order information and delivers them to the user, who can then use the ingredients to create the suggested dish.

[1030] In this way, users can have the best possible recipes suggested based on their family's preferences and health status, making efficient use of ingredients. Furthermore, they can easily order the ingredients they need, saving them the trouble of shopping.

[1031] Example 2

[1032] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1033] With conventional systems, users had to manually check the latest sales information of nearby stores, which was time-consuming and made it difficult to obtain accurate information.In addition, it was difficult to propose menus that took into account family preferences, allergies, health conditions, and the contents of the refrigerator, making it difficult to create efficient shopping plans.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1035] In this invention, the server includes a means for transmitting captured images to the server, a means for the server to perform image recognition and extract product name and price information, and a means for storing the extracted information in a database. This allows users to automatically analyze store advertising information captured with a smartphone camera and obtain product name and price information. It also facilitates menu suggestions and efficient shopping planning that take into account family preferences, allergies, health conditions, and refrigerator contents.

[1036] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1037] "Allergy information" refers to information about food allergies that each member of the family has.

[1038] "Health status" refers to information about the health of each family member, such as chronic illnesses or nutritional needs.

[1039] "What's in the refrigerator" refers to the types and quantities of food and beverages currently stored in the refrigerator.

[1040] "Repertoire" refers to the types of dishes a user can cook.

[1041] "Store advertising information" refers to product and price information listed in flyers and advertisements issued by nearby stores.

[1042] "Image recognition" refers to the technology of extracting specific information from captured images.

[1043] "Server" refers to a computer system for processing and storing data.

[1044] "Database" refers to a system for storing extracted information in an organized manner.

[1045] "Menu suggestions" refers to proposing a weekly meal plan that takes into account the family's preferences, health status, and the contents of the refrigerator.

[1046] "Shopping planning" refers to planning which stores to purchase the ingredients and products you need.

[1047] "Receipt reading" refers to the technology of reading information about previously purchased products from receipts.

[1048] "Calorie counting" refers to calculating the calories of food.

[1049] "Food cost management" refers to managing the costs of purchasing ingredients and products.

[1050] "Appropriately modify" refers to changing plans or proposals to suit actual circumstances.

[1051] This invention is a system that allows users to take a photo of a nearby store's advertising information using their smartphone camera and extract product names and prices from the image. This allows users to easily obtain the latest sales information. It also enables menu suggestions and efficient shopping plans that take into account family preferences, allergies, health conditions, and the contents of the refrigerator.

[1052] Hardware and software used

[1053] Hardware:

[1054] Smartphone: Used by users to take pictures of advertising information.

[1055] Server: A computer system used to process and store data.

[1056] software:

[1057] Camera app: The camera application installed on your smartphone.

[1058] Image recognition software: Software for extracting text information from images, such as Google Cloud Vision API.

[1059] Database: A system for storing the extracted information, such as MySQL or PostgreSQL.

[1060] Program processing

[1061] The user takes a photo of the advertisement using their smartphone camera.

[1062] The user launches the camera app on their smartphone and takes a photo of the store's advertising information. For example, if a user wants to take a photo of an "ABC Supermarket" flyer, they open the camera app, adjust the frame so that the entire flyer fits in the frame, and press the shutter button.

[1063] The device sends the captured image to the server.

[1064] The device (smartphone) sends the captured image to the server via an application. Specifically, the application sends the image data to the server as an HTTP POST request. For example, the image data is Base64 encoded and included in the request body.

[1065] The server performs image recognition and extracts product name and price information.

[1066] The server sends the received image data to the Google Cloud Vision API for image recognition. The API analyzes the text in the image and extracts product name and price information. For example, if the API recognizes the text "Apple 100 yen" and "Banana 150 yen," it returns that information to the server in JSON format.

[1067] The server stores the extracted information in a database.

[1068] The server saves the extracted product name and price information in a database. Specifically, it executes an INSERT query in a database such as MySQL or PostgreSQL to save the information. For example, information such as "apples 100 yen" and "bananas 150 yen" is saved in the corresponding tables in the database.

[1069] The device displays the information to the user.

[1070] The device (smartphone) displays the product name and price information sent from the server to the user through the application. Specifically, the application receives the response from the server and displays the information on the screen. For example, the information "Apples 100 yen" and "Bananas 150 yen" is displayed on the app screen.

[1071] Examples of concrete examples and prompts

[1072] Examples:

[1073] A user takes a photo of a flyer for "ABC Supermarket." The flyer states "Apples 100 yen" and "Bananas 150 yen." The user takes a photo of the flyer with their smartphone camera and sends the image to a server via the app. The server analyzes the image using the Google Cloud Vision API and extracts the information "Apples 100 yen" and "Bananas 150 yen." This information is stored in a database and displayed in the user's app.

[1074] Example prompt sentence:

[1075] "Please take a photo of a flyer from a nearby supermarket and extract the product name and price information. For example, if the flyer says 'Apples 100 yen' and 'Bananas 150 yen', please extract and display that information."

[1076] In this way, the system allows users to easily obtain the latest sales information.

[1077] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1078] Step 1:

[1079] The user takes a photo of the advertisement information using their smartphone camera.

[1080] Specifically, the user launches the camera app on their smartphone, frames the store's advertising information, and presses the shutter button.

[1081] Input: Image of advertising information

[1082] Output: Captured image data

[1083] Step 2:

[1084] The device sends the captured image to the server.

[1085] Specifically, the smartphone application encodes the image data in Base64 and sends it to the server as an HTTP POST request.

[1086] Input: Photographed image data

[1087] Output: Image data sent to the server

[1088] Step 3:

[1089] The server performs image recognition and extracts product name and price information.

[1090] Specifically, the server sends the received image data to the Google Cloud Vision API, which then analyzes the text in the image to extract product name and price information.

[1091] Input: Image data sent to the server

[1092] Output: Extracted product name and price information (JSON format)

[1093] Step 4:

[1094] The server stores the extracted information in a database.

[1095] Specifically, the server executes an INSERT query to store the extracted product name and price information in a database such as MySQL or PostgreSQL.

[1096] Input: Extracted product name and price information (JSON format)

[1097] Output: Product name and price information stored in the database

[1098] Step 5:

[1099] The device displays the information to the user.

[1100] Specifically, the smartphone application receives a response from the server and displays the product name and price information on the screen.

[1101] Input: Product name and price information stored in the database

[1102] Output: Product name and price information displayed on the smartphone screen

[1103] (Application example 2)

[1104] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1105] Today's consumers are expected to shop efficiently and plan meals while taking into account their family's preferences, allergies, and health conditions. However, to achieve this, they must manually check supermarket flyers and manually enter product names and prices, which is extremely time-consuming. It is also difficult to track past price fluctuations, and there is a lack of information to make optimal purchases. Furthermore, there is a lack of guides to help users understand how to use the application.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended recipes; means for capturing flyer information from nearby commercial facilities using image recognition; means for suggesting a weekly menu and where to buy what; means for extracting product names and price information from flyer images captured with a smartphone camera; means for saving previously captured flyer information and tracking price fluctuations; means for providing the user with the latest sales information using a generative AI model; and means for generating prompts to guide the user on how to use the application. This enables users to shop efficiently and plan optimal menus. Tracking past price fluctuations also enables smarter shopping, and the generative AI model and prompts allow users to easily understand how to use the application.

[1107] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1108] "Allergy information" refers to information about allergies held by each member of the family.

[1109] "Health status" refers to information about the health of each family member, such as whether they need to avoid certain foods.

[1110] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[1111] "Repertoire" refers to the types of dishes a user can make.

[1112] "Commercial facilities" refers to retail stores such as supermarkets and department stores.

[1113] "Flyer information" refers to the content of flyers issued by commercial facilities, including advertisements and sales information.

[1114] "Image recognition" refers to the technology of extracting specific information from an image.

[1115] A "menu" refers to a meal plan for a specific period of time.

[1116] A "smartphone" is a type of mobile phone that can connect to the Internet and use applications.

[1117] "Camera" refers to a device for taking images or videos.

[1118] "Product Name" refers to the name of a product sold at a commercial establishment.

[1119] "Price Information" refers to information regarding the selling price of a product.

[1120] A "generative AI model" refers to a model that uses artificial intelligence to perform specific tasks.

[1121] A "prompt" is a guide sentence that encourages the user to take a specific action.

[1122] A system for implementing the present invention is configured as follows.

[1123] First, a user takes a photo of a flyer from a nearby commercial establishment using their smartphone camera, and the image is processed by an application on the smartphone, which uses OpenCV and Tesseract OCR to extract product name and price information from the image.

[1124] The extracted information is sent to a server and stored in a database. The server uses the stored data to track past price fluctuations and provide optimal shopping suggestions to users. It also uses generative AI models to provide users with the latest sales information.

[1125] Furthermore, when the user uses the application, the system generates prompts to guide them. For example, the system displays a prompt such as, "Take a photo of a flyer from a nearby supermarket to get the latest sales information."

[1126] The specific processing steps are as follows:

[1127] 1. Hardware: Use your smartphone camera to take a photo of the flyer.

[1128] 2. Software: Python, OpenCV, Tesseract OCR is used to extract text information from images.

[1129] 3. Data processing: Convert the image to grayscale and reduce noise.

[1130] 4. Data Calculation: Extract text from images using Tesseract OCR to obtain product name and price information.

[1131] 5. Data storage: The extracted information is sent to the server and stored in the database.

[1132] 6. Tracking price fluctuations: The server tracks price fluctuations based on historical data.

[1133] 7. Information provision: Using generative AI models, we provide users with the latest sales information.

[1134] 8. Prompt generation: Generate prompts to guide the user on how to use the application.

[1135] The system allows users to shop efficiently and plan optimal meals, tracks historical price fluctuations to help them shop smarter, and uses generative AI models and prompts to help users easily understand how to use the application.

[1136] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1137] Step 1:

[1138] A user uses the smartphone camera to take a picture of a flyer for a nearby business. The input is an image of the flyer, and the output is an image file stored on the smartphone that is used in subsequent processing steps.

[1139] Step 2:

[1140] The device loads the captured image into the application. The input is an image file stored on the smartphone, and the output is image data that can be processed within the application. Specifically, the application opens the image file and loads it into memory.

[1141] Step 3:

[1142] The device uses OpenCV to convert images to grayscale. The input is image data that can be processed within the application, and the output is a grayscale image. Specifically, it calls OpenCV functions to convert color images to grayscale.

[1143] Step 4:

[1144] The device uses Tesseract OCR to extract text information from grayscale images. The input is a grayscale image, and the output is the extracted text information. Specifically, the device calls Tesseract OCR functions to parse the text from the image and obtain product name and price information.

[1145] Step 5:

[1146] The terminal transmits the extracted text information to the server. The input is the extracted text information, and the output is the data transmitted to the server. In a specific operation, the text information is transmitted to the server via network communication.

[1147] Step 6:

[1148] The server stores the received text information in a database. The input is the text information sent to the server, and the output is the data stored in the database. Specifically, the server performs a database insert operation and stores the text information.

[1149] Step 7:

[1150] The server tracks price fluctuations based on the stored data. The input is past text information stored in the database, and the output is the analysis of price fluctuations. Specifically, it executes database queries to retrieve and analyze past price information.

[1151] Step 8:

[1152] The server uses the generative AI model to provide the user with the latest sales information. The input is the analysis results of price fluctuations and the latest text information, and the output is the sales information provided to the user. Specifically, the server runs the generative AI model to generate sales information.

[1153] Step 9:

[1154] The device generates prompts to guide the user on how to use the application. The input is the user's operation status and the application status, and the output is the generated prompt. Specifically, the device analyzes the user's operation history and generates an appropriate prompt. For example, the device might display a prompt saying, "Take a photo of a flyer from a nearby supermarket to get the latest sales information."

[1155] Example 3

[1156] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1157] In today's busy lifestyles, it is extremely time-consuming to efficiently plan menus and purchase the necessary ingredients while taking into account the preferences and health status of family members. It is also difficult to manage food expenses and calculate calories based on past purchase history. Furthermore, there is a demand for flexible menu changes to suit daily lifestyles. A system that can solve these problems is needed.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1159] In this invention, the server includes: means for registering family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire; means for importing flyer information from nearby stores using image recognition; means for suggesting a weekly menu and what to buy and where; means for receiving receipt images photographed by the user and extracting product names and prices using optical character recognition software; means for comparing the extracted product names and prices with past purchase history to calculate the total purchase price; and means for the user to input changes to the suggested menu, revise the menu based on that information, and update calorie calculations and food cost management. This allows users to efficiently create menus and purchase the necessary ingredients, and easily manage food costs and calorie calculations. It also allows users to flexibly change menus to suit their daily lifestyles.

[1160] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1161] "Allergy information" refers to information about food allergies that each member of the family has.

[1162] "Health status" refers to information about the health of each family member, including restrictions on the intake of certain foods and nutrients.

[1163] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[1164] "Repertoire" refers to the collection of types of dishes or recipes that a user can make.

[1165] "Store flyer information" refers to advertising information including special offers and discount information offered by nearby stores.

[1166] "Image recognition" refers to technology that extracts specific information from images, including optical character recognition (OCR).

[1167] A "menu" refers to a meal plan for a specific period of time (e.g., one week).

[1168] "Receipt image" refers to a photograph of the receipt that a user receives after shopping.

[1169] "Optical character recognition software" refers to software used to extract text information from images.

[1170] "Product name" refers to the name of the purchased product as it appears on the receipt.

[1171] "Price Information" refers to the price of the product listed on the receipt.

[1172] "Purchase history" refers to data that records information about products purchased by a user in the past.

[1173] "Calorie counting" refers to the process of calculating the calories of ingredients and dishes.

[1174] "Food cost management" refers to the process of managing the costs of ingredients and cooking.

[1175] "Change Information" refers to changes that a user makes to a suggested menu.

[1176] This invention is a system that registers the user's family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire, then uses image recognition to import flyers from nearby stores, proposing a weekly menu and what to buy where.The system also reads past purchase receipts, calculates how much each item will cost, calculates calories, manages food expenses, and can adjust the menu accordingly.

[1177] Hardware and software used

[1178] server

[1179] The server is the central location for receiving and processing the data sent by users. Specifically, it uses the following software:

[1180] Google Cloud Vision API: Optical character recognition software for extracting text data from receipt images.

[1181] Python's Pandas library: a library for processing and analyzing data.

[1182] Nutritional Data API: An API for obtaining calorie information for each ingredient.

[1183] Terminal

[1184] The terminal (smartphone or PC) provides an interface for users to input data and send it to the server. Specifically, it has the following functions:

[1185] Camera function: A function for taking photos of receipts.

[1186] Application: An application where the user enters data and sends it to the server.

[1187] user

[1188] Users use the system to:

[1189] Register your family's preferences, allergy information, health conditions, what's in your refrigerator, and cooking repertoire.

[1190] After shopping at the supermarket, you take a photo of the receipt and send it to the server via the app.

[1191] Change information for the proposed menu is entered and sent to the server.

[1192] Specific examples

[1193] For example, suppose a user goes shopping at a supermarket and receives a receipt totaling 3,000 yen. The user takes a photo of the receipt with their smartphone camera and sends it to the server via the app. The server uses the Google Cloud Vision API to extract product names and prices from the receipt, compares them with past purchase history, and calculates the total price.

[1194] If a user changes the menu suggestion by saying, "I've decided to eat out today," the information is sent to the server using a smartphone app. The server then uses this information to revise the menu and update calorie calculations and food cost management.

[1195] Prompt Sentence Examples

[1196] "After shopping at the supermarket, take a photo of your receipt, extract product names and prices from the image, and compare it with your past purchase history to calculate how much you can buy."

[1197] "If the user inputs changes to the proposed menu, modify the menu based on that information and update calorie calculations and food cost management." The flow of the specific processing in the third embodiment will be explained using FIG.

[1198] Step 1:

[1199] Register your family's preferences, allergy information, health status, refrigerator contents, and repertoire

[1200] user

[1201] Using a smartphone or PC application, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make.

[1202] Input: Family preferences, allergy information, health status, refrigerator contents, repertoire

[1203] Output: The registered information is saved in the database.

[1204] Specific operation: The user launches the app, enters information into the input form, and clicks the save button. The app sends the information to the server, which saves it in the database.

[1205] Step 2:

[1206] Importing store flyer information

[1207] Terminal

[1208] The terminal uses image recognition to capture flyer information from nearby stores.

[1209] Input: Store flyer image

[1210] Output: Extracted flyer information (product name, price, special sale information, etc.)

[1211] How it works: A user takes a photo of a store flyer with their smartphone camera, and the app sends the image to a server, which uses image recognition software to extract flyer information and store it in a database.

[1212] Step 3:

[1213] Weekly menu and shopping suggestions

[1214] server

[1215] Based on the registered information and store flyers, the server suggests a weekly menu and what to buy and where to buy it.

[1216] Input: Registered information, store flyer information

[1217] Output: Weekly menu and shopping suggestions

[1218] Specific operation: The server retrieves registered information and flyer information from the database, uses an algorithm to calculate a weekly menu and where to buy, and returns the calculation results to the user.

[1219] Step 4:

[1220] Capture and send a receipt image

[1221] user

[1222] After shopping at a supermarket, users take a photo of the receipt using their smartphone camera and send it to the server via the app.

[1223] Input: Receipt image

[1224] Output: Receipt image sent to the server

[1225] Specific operation: The user launches the app, selects the camera function to take a picture of the receipt, and presses the send button. The app then sends the image to the server.

[1226] Step 5:

[1227] Receipt image analysis and information extraction

[1228] server

[1229] The server analyzes the received receipt image and uses optical character recognition software to extract product name and price information.

[1230] Input: Receipt image

[1231] Output: Extracted product name and price information

[1232] What happens: The server receives the receipt image, uses optical character recognition software to extract the text data, and stores the extracted data in a database.

[1233] Step 6:

[1234] Matching with purchase history and calculating the amount

[1235] server

[1236] The server compares the extracted product name and price information with past purchase history and calculates how much the product will cost.

[1237] Input: Extracted product name and price information, past purchase history

[1238] Output: Calculated amount information

[1239] Specific operation: The server retrieves past purchase history from the database, compares it with the current data, calculates the amount, and returns the calculation result to the user.

[1240] Step 7:

[1241] Enter and submit menu change information

[1242] user

[1243] The user inputs modification information for the proposed menu and sends it to the server.

[1244] Input: Menu change information

[1245] Output: Changes sent to the server

[1246] Specific operation: The user launches the app, opens the menu change screen, enters the changes, and presses the save button. The app then sends the changes to the server.

[1247] Step 8:

[1248] Menu data updates, calorie calculations, and food cost management updates

[1249] server

[1250] The server updates the existing menu data based on the received menu change information, calculates calories, and manages food costs.

[1251] Input: Menu change information

[1252] Output: Updated menu data, calorie calculation results, food cost management results

[1253] Specific operation: The server receives the menu change information and updates the menu data in the database. It uses the Nutritional Data API to calculate calories and update the food cost management by comparing it with past purchase history. It then returns the updated results to the user.

[1254] (Application example 3)

[1255] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1256] The traditional process of creating menus and purchasing ingredients required time and effort for users. Managing food expenses and calculating calories was also time-consuming, making it difficult to flexibly adapt to daily lifestyles. Managing receipts and calculating costs based on past purchase history was also cumbersome. There is a need for a system that can solve these problems and provide comprehensive support for users' eating habits.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1258] In this invention, the server includes a means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended cooking repertoire; a means for importing flyer information from nearby supermarkets using image recognition; a means for suggesting a weekly menu and what to buy and where; a means for scanning receipts with a smartphone camera and extracting product names and price information; a means for automatically generating menus using a generative AI model based on the user's dietary data; a means for calculating calories and managing food costs based on the generated menu and receipt data; and a means for modifying the menu to suit the user's lifestyle. This allows users to significantly reduce the effort required for creating menus and purchasing ingredients, and makes it easy to manage food costs and calculate calories. It also allows for flexible menu modification to suit daily lifestyles.

[1259] "Family preferences" refers to the types of ingredients, dishes, and seasonings that each family member likes.

[1260] "Allergy information" refers to information about specific ingredients in the case where someone in the family has an allergic reaction to those ingredients.

[1261] "Health Condition" refers to information on dietary restrictions and recommended foods for specific health conditions or illnesses in the family, if any.

[1262] "What's in the refrigerator" refers to the types and quantities of ingredients and seasonings currently stored in the refrigerator.

[1263] "Repertoire" refers to the range of dishes and recipes a user can make.

[1264] "Supermarket flyer information" refers to the content of flyers issued by nearby supermarkets, including advertisements and special sale information.

[1265] "Image recognition" refers to the technology of extracting specific information from digital images.

[1266] A "menu" refers to a meal plan or menu for a specific period of time (for example, a week).

[1267] "Receipt" means a paper or digital receipt listing the details and price of purchased items.

[1268] "Product Name" refers to the name of the purchased product.

[1269] "Price Information" refers to information regarding the price of a purchased item.

[1270] "Dietary Data" refers to information such as a user's dietary preferences, restrictions, and past eating history.

[1271] A "generative AI model" refers to a model that uses artificial intelligence to perform a specific task (in this case, generating a menu).

[1272] "Calorie counting" refers to the process of calculating the total calories in a meal.

[1273] "Food cost management" refers to the process of calculating the costs of ingredients and meals and keeping them within a budget.

[1274] "Lifestyle" refers to a user's daily life patterns and habits.

[1275] "Menu modification" refers to the process of changing an already planned menu.

[1276] The system for carrying out the present invention has a plurality of functions for comprehensively supporting the user's eating habits. Specific embodiments of the system will be described below.

[1277] 1. Register your family's preferences, allergy information, health status, refrigerator contents, and repertoire

[1278] Through a smartphone application, users register their family's preferences, allergies, health conditions, what's in their refrigerator, and the recipes they can make, allowing the system to obtain detailed data about the user's eating habits.

[1279] 2. Importing supermarket flyer information

[1280] The system uses the smartphone camera to capture flyer information from nearby supermarkets using image recognition technology (such as OpenCV), allowing users to obtain the latest sales and product information.

[1281] 3. Automatic menu generation

[1282] Based on the user's dietary data, a generative AI model is used to automatically generate a weekly menu. Specifically, a request is sent to the AI ​​model using the following prompt:

[1283] Example prompt sentence:

[1284] User dietary data: {'preferences': ['vegetarian', 'low-carb'], 'allergies': ['peanuts'], 'budget': 5000}

[1285] Please suggest a menu for the week.

[1286] 4. Reading receipts and extracting price information

[1287] After a user shops at a supermarket, they take a photo of the receipt with their smartphone camera, and the system uses pytesseract to extract product names and prices from the receipt image.

[1288] 5. Calorie counting and food budget management

[1289] Based on the generated menu and receipt data, the system calculates calories and manages food costs, allowing users to easily understand the total calories and costs.

[1290] 6. Menu Modification

[1291] The suggested menu can be adjusted as needed to suit the user's lifestyle. For example, if there is a change in your schedule, such as "I've decided to eat out today," you can enter that information into the system and the menu will be automatically updated.

[1292] Hardware and software used

[1293] Hardware: Smartphone (with camera)

[1294] software:

[1295] OpenCV: Image processing library

[1296] pytesseract: an OCR (optical character recognition) library

[1297] requests: HTTP request library

[1298] Specific examples

[1299] For example, if a user is vegetarian, prefers low-carb meals, and has a budget of 5,000 yen, the system will operate as follows:

[1300] 1. Users register their dietary data through a smartphone application.

[1301] 2. Supermarket flyer information is captured by a camera and analyzed using image recognition technology.

[1302] 3. Send prompts to the generative AI model to automatically generate a weekly menu.

[1303] 4. After shopping at the supermarket, take a photo of the receipt and extract product name and price information.

[1304] 5. Calculate calories and manage food costs based on the generated menu and receipt data.

[1305] 6. Modify the menu as needed to suit the user's lifestyle.

[1306] In this way, users can significantly reduce the effort required for planning meals and purchasing ingredients, and can easily manage food expenses and calculate calories. They can also flexibly modify their meals to suit their daily lifestyles.

[1307] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1308] Step 1:

[1309] Through a smartphone application, users register their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make.

[1310] Input: Family preferences, allergy information, health status, refrigerator contents, repertoire

[1311] Output: Registered dietary data

[1312] Specific operation: The user enters the required information into the application's input form and sends it to the server, which then stores this data in a database.

[1313] Step 2:

[1314] The device uses the smartphone camera to capture flyer information from nearby supermarkets using image recognition technology (such as OpenCV).

[1315] Input: Supermarket flyer image

[1316] Output: Analyzed flyer information (product name, price, special sale information, etc.)

[1317] How it works: A user takes a photo of a flyer with their smartphone camera, and the device analyzes the image using OpenCV to extract product names and prices. The extracted information is then sent to the server.

[1318] Step 3:

[1319] The server automatically generates a weekly menu using a generative AI model based on the user's dietary data.

[1320] Input: User's dietary data, flyer information

[1321] Output: Weekly menu

[1322] Specific operation: The server sends the following prompt sentence to the generation AI model to generate a menu.

[1323] User dietary data: {'preferences': ['vegetarian', 'low-carb'], 'allergies': ['peanuts'], 'budget': 5000}

[1324] Please suggest a menu for the week.

[1325] The generated menu is stored on the server and notified to the user.

[1326] Step 4:

[1327] After a user shops at a supermarket, they take a photo of the receipt with their smartphone camera, and the device uses pytesseract to extract product names and prices from the receipt image.

[1328] Input: Receipt image

[1329] Output: Product name and price information

[1330] How it works: A user takes a photo of a receipt with their smartphone camera, and the device uses pytesseract to extract text from the image, then sends the extracted product name and price information to the server.

[1331] Step 5:

[1332] The server calculates calories and manages food costs based on the generated menu and receipt data.

[1333] Input: Menu, receipt data

[1334] Output: Total calories, total cost

[1335] Specific operation: The server calculates the calories of each dish in the menu and calculates the total cost from the receipt data. The calculation result is notified to the user.

[1336] Step 6:

[1337] The proposed menu can be modified as needed to suit the user's lifestyle.

[1338] Input: User's lifestyle changes

[1339] Output: Modified menu

[1340] Specific operation: The user inputs changes such as "I've decided to eat out today" through the application, and the server automatically updates the menu based on that information. The user is notified of the updated menu.

[1341] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1342] "Example 1"

[1343] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The emotion engine recognizes a user's emotions and adjusts menus based on the emotions. Specifically, when the user is feeling stressed, the emotion engine collects that information and suggests menus that include ingredients that reduce stress. For example, if the emotion engine recognizes that the user is feeling stressed, it suggests menus that include ingredients that are effective in reducing stress, such as chocolate and nuts.

[1344] "Example 2"

[1345] The emotion engine also adjusts food purchase suggestions based on the user's emotions. For example, when a user is feeling happy, the emotion engine collects that information and suggests meals that include the user's favorite ingredients. Specifically, if the emotion engine recognizes that the user is feeling happy due to a birthday or anniversary, it will suggest meals that include cake and celebratory meals.

[1346] "Example 3"

[1347] In addition, the emotion engine will adjust the meal suggestions based on the user's emotions. For example, when the user is feeling sad, the emotion engine will collect that information and suggest comforting meals. Specifically, if the emotion engine recognizes that the user is feeling sad, it will suggest comforting meals such as hot soup or porridge.

[1348] "Example 10"

[1349] The emotion engine also adjusts food purchase suggestions based on the user's emotions. For example, if the user is feeling angry, the emotion engine will collect that information and suggest a meal plan that includes ingredients that will calm the anger. Specifically, if the emotion engine recognizes that the user is feeling angry, it will suggest a meal plan that includes relaxing drinks such as mint tea or chamomile tea.

[1350] "Example 11"

[1351] Furthermore, the emotion engine adjusts the menu suggestions based on the user's emotions. For example, when the user is feeling surprised, the emotion engine will collect that information and suggest menus that include ingredients that provide a fresh surprise. Specifically, if the emotion engine recognizes that the user is feeling surprised, it will suggest menus that include exotic fruits and unusual dishes.

[1352] The processing flow of each embodiment will be described below.

[1353] "Example 1"

[1354] Step 1: The emotion engine recognizes the user's emotion.

[1355] Step 2: The emotion engine determines that the emotion it recognizes is stress.

[1356] Step 3: The emotion engine suggests meals that include ingredients that reduce stress.

[1357] "Example 2"

[1358] Step 1: The emotion engine recognizes the user's emotion.

[1359] Step 2: The emotion engine determines that the recognized emotion is joy.

[1360] Step 3: The emotion engine suggests meals that include ingredients the user likes.

[1361] "Example 3"

[1362] Step 1: The emotion engine recognizes the user's emotion.

[1363] Step 2: The emotion engine determines that the recognized emotion is sadness.

[1364] Step 3: The emotional engine suggests comfort food.

[1365] "Example 10"

[1366] Step 1: The emotion engine recognizes the user's emotion.

[1367] Step 2: The emotion engine determines that the recognized emotion is anger.

[1368] Step 3: The emotion engine suggests meals that include ingredients that calm anger.

[1369] "Example 11"

[1370] Step 1: The emotion engine recognizes the user's emotion.

[1371] Step 2: The emotion engine determines that the emotion recognized is surprise.

[1372] Step 3: The emotion engine suggests menus that include ingredients that offer fresh surprises.

[1373] Example 1

[1374] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1375] While conventional menu suggestion systems can take into account family preferences, allergy information, health status, the contents of the refrigerator, and the user's cooking repertoire, they have the problem of not being able to suggest menus that take into account the user's emotional state.In particular, if the user is feeling stressed or in a particular emotional state, it is not possible to suggest ingredients or dishes that correspond to that emotion, making it difficult to increase user satisfaction.

[1376] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1377] In this invention, the server includes means for registering family preferences, allergy information, health condition, contents of the refrigerator, and the user's cooking repertoire, means for capturing flyer information from nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for recognizing the user's emotions and adjusting the menu based on those emotions, means for generating an optimal menu taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and the user's cooking repertoire, and means for presenting the generated menu to the user. This makes it possible to propose a menu that takes into account the user's emotional state.

[1378] "Family preferences" is information about ingredients and dishes that the user's family particularly likes or dislikes.

[1379] "Allergy information" is information about food allergies that the user or their family members have.

[1380] "Health Status" is information about the health of the user and their family, including specific illnesses and health restrictions.

[1381] "Refrigerator contents" refers to information about ingredients and foods currently stored in the user's refrigerator.

[1382] The "repertoire of dishes that can be made" is information about the types of dishes and recipes that the user can make by himself.

[1383] "Flyer information from nearby stores" is information about sales and special offers offered by stores in the user's area.

[1384] "Image recognition" is a technology that analyzes digital images to extract specific information.

[1385] A "menu" is a meal plan or menu for a specific period of time.

[1386] "Emotion recognition" means analyzing the user's facial expression, tone of voice, etc. to identify their emotional state at that time.

[1387] "Adjusting menus based on emotions" means suggesting appropriate ingredients and dishes taking into account the user's emotional state.

[1388] "Generating the optimal menu" means creating the most suitable meal plan by comprehensively taking into consideration the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and repertoire of dishes they can make.

[1389] "Presenting the created menu to the user" means displaying the created menu to the user.

[1390] This invention is a system that allows users to register their family's preferences, allergy information, health status, refrigerator contents, and cooking repertoire, and then suggests optimal menus based on these.It also has the function of recognizing the user's emotional state and adjusting the menu based on that emotion.

[1391] Hardware and software used

[1392] Hardware:

[1393] Terminal: The device through which the user enters information (smartphone, tablet, PC, etc.)

[1394] Camera: A device for recognizing the user's facial expressions

[1395] Microphone: A device that recognizes the tone of a user's voice.

[1396] Server: A computer system for storing data, analyzing data, and generating menus

[1397] software:

[1398] Database Management System: Software for storing information entered by users

[1399] Image recognition software: Software for capturing flyer information for nearby stores

[1400] Emotion Recognition Engine: Software for analyzing user emotions

[1401] Menu Generation Algorithm: Software for generating optimal menus based on user information and emotional state

[1402] Program processing

[1403] User enters information:

[1404] Using a dedicated application or web interface, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. For example, a user might enter information such as "I like fish dishes," "I'm allergic to shrimp," "I'm diabetic," "chicken breast, broccoli, and eggs," or "sautéed chicken breast, sautéed broccoli with garlic, and rolled eggs."

[1405] The server stores the information in a database:

[1406] The server receives the information entered by the user and stores it in a database, including family preferences, allergies, health conditions, refrigerator contents, and cooking repertoire, so that the information can be referenced later.

[1407] Recognize user emotions:

[1408] The user provides the system with their facial expressions and tone of voice using a camera and microphone. The device collects this data and sends it to the server. For example, if the user is feeling stressed, their facial expression may become tense and their voice tone may become lower.

[1409] The server analyzes the emotion data:

[1410] The server analyzes the received facial expression and tone of voice data to identify the user's emotional state. The emotion engine uses a machine learning model to determine whether the user is feeling stressed. For example, if the user's facial expression is tense, the emotion engine will determine that the user is "stressed."

[1411] The server generates the menu:

[1412] The server generates the optimal menu by taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and available cooking repertoire. For example, if the user is feeling stressed, the server will suggest a menu that includes ingredients that are effective in reducing stress, such as chocolate and nuts.

[1413] The server presents the menu to the user:

[1414] The server then presents the generated menu to the user, who can view the suggested menu through the application or web interface. For example, a menu item might include "sauteed chicken breast," "sauteed broccoli with garlic," and "chocolate dessert."

[1415] Examples and prompts

[1416] Examples:

[1417] The user registers the following information:

[1418] Family preferences: Likes fish dishes, doesn't like spicy food

[1419] Allergy Information: Shrimp allergy

[1420] Health Condition: Diabetes

[1421] What's in the fridge: Chicken breast, broccoli, eggs

[1422] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[1423] If the server determines that the user is feeling stressed, it generates the following menu and presents it to the user:

[1424] Sautéed chicken breast

[1425] Garlic sauteed broccoli

[1426] Chocolate dessert

[1427] Example prompt sentence:

[1428] "When a user is feeling stressed, please suggest a meal that will be effective in reducing stress, taking into account the contents of the refrigerator and the repertoire of dishes that can be made."

[1429] In this way, the server provides a system that suggests menus that take into account the user's emotional state.

[1430] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1431] Step 1:

[1432] The user enters information

[1433] Using a dedicated application or web interface, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. The input information is saved in text format on the device. For example, a user might input information such as "I like fish dishes," "I'm allergic to shrimp," "I have diabetes," "chicken breast, broccoli, and eggs," or "sauteed chicken breast, sauteed broccoli with garlic, and rolled omelet."

[1434] Input: Family preferences, allergy information, health status, contents of refrigerator, and cooking repertoire

[1435] Output: The entered information is saved on the device.

[1436] Step 2:

[1437] The server stores the information in a database

[1438] The device sends the information entered by the user to the server, which then stores the information in a database. This information includes family preferences, allergies, health conditions, the contents of the refrigerator, and the recipes available for cooking. This allows the information to be referenced later.

[1439] Input: User information sent from the device

[1440] Output: User information stored in the database

[1441] Step 3:

[1442] Recognize user emotions

[1443] The user provides the system with their facial expressions and tone of voice using a camera and microphone. The device collects this data and sends it to the server. For example, if the user is feeling stressed, their facial expression may become tense and their voice tone may become lower.

[1444] Input: User's facial expression data, voice tone data

[1445] Output: Emotion data sent to the server

[1446] Step 4:

[1447] The server analyzes the emotional data

[1448] The server analyzes the received facial expression and tone of voice data to identify the user's emotional state. The emotion engine uses a machine learning model to determine whether the user is feeling stressed. For example, if the user's facial expression is tense, the emotion engine will determine that the user is "stressed."

[1449] Input: User's facial expression data, voice tone data

[1450] Output: Parsed user's emotional state

[1451] Step 5:

[1452] The server generates the menu

[1453] The server generates the optimal menu by taking into account the user's emotional state, family preferences, allergy information, health condition, contents of the refrigerator, and available cooking repertoire. For example, if the user is feeling stressed, the server will suggest a menu that includes ingredients that are effective in reducing stress, such as chocolate and nuts.

[1454] Input: User's emotional state, family preferences, allergy information, health status, refrigerator contents, and cooking repertoire

[1455] Output: Generated menu

[1456] Step 6:

[1457] The server presents the menu to the user

[1458] The server then presents the generated menu to the user, who can view the suggested menu through the application or web interface. For example, a menu item might include "sauteed chicken breast," "sauteed broccoli with garlic," and "chocolate dessert."

[1459] Input: Generated menu

[1460] Output: The menu presented to the user

[1461] (Application example 1)

[1462] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1463] While conventional menu suggestion systems can take into account the user's family preferences, allergy information, health status, and the contents of the refrigerator, they have the problem of not being able to suggest menus that reflect the user's emotional state. It is also difficult to link with delivery services and provide the optimal meal based on the user's emotions.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health status, refrigerator contents, and the user's cooking repertoire, means for capturing information about nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for recognizing emotions, means for adjusting the menu based on emotions, means for providing a delivery service, and means for generating an optimal menu using a generative AI model. This makes it possible to propose an optimal menu that reflects the user's emotional state and provide a delivery service.

[1465] "Family preferences" refers to the types of ingredients and dishes that the user's family prefers.

[1466] "Allergy Information" refers to information about allergies held by the User or their family members.

[1467] "Health Status" refers to information about the health of a user or their family members, such as medical conditions such as diabetes or high blood pressure.

[1468] "What's in the refrigerator" refers to the ingredients and food currently stored in the user's refrigerator.

[1469] "Repertoire you can make" refers to the types of dishes that users can cook themselves.

[1470] "Information about nearby stores" refers to sales and special offers from stores near the user's residence.

[1471] "Image recognition" refers to the technology of analyzing image data and extracting specific information.

[1472] A "menu" refers to a meal plan or menu for a specific period of time.

[1473] "Emotion recognition" refers to the technology of analyzing and recognizing a user's emotions from facial expressions, voice, etc.

[1474] "Adjusting menus based on emotions" refers to modifying or optimizing menus by taking into account the user's emotional state.

[1475] "Delivery Service" refers to a service that delivers meals ordered by a user to a specified location.

[1476] A "generative AI model" refers to a model that uses artificial intelligence to perform specific tasks.

[1477] "Generating the optimal menu" refers to suggesting the most suitable menu based on the user's registered information and emotional state.

[1478] The system for implementing this invention registers the user's family preferences, allergy information, health status, contents of the refrigerator, and the repertoire of dishes they can make, recognizes emotions, suggests optimal menus, and provides delivery services.

[1479] Hardware and software used

[1480] Hardware: Smartphone

[1481] software:

[1482] Emotion recognition API (e.g. Microsoft Azure Emotion API)

[1483] Database (e.g. Firebase)

[1484] Food delivery API (e.g. Uber Eats API)

[1485] Generative AI models (e.g., OpenAI GPT-4)

[1486] System configuration and operation

[1487] 1. User information registration:

[1488] Users use a smartphone application to register their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make.

[1489] The registered information is stored in the Firebase database.

[1490] 2. Emotion recognition:

[1491] Users take a picture of their face using their smartphone camera and the emotion is recognized using the Microsoft Azure Emotion API.

[1492] The recognized emotion data is sent to the application in real time.

[1493] 3. Menu suggestions:

[1494] Based on the recognized emotion data and registered user information, the optimal menu is generated using a generative AI model (OpenAI GPT-4).

[1495] For example, if a user is feeling stressed, it will suggest meals that include chocolate and nuts.

[1496] 4. Delivery Service:

[1497] Based on the proposed menu, it uses a food delivery API (Uber Eats API) to search for corresponding dishes and provide delivery options.

[1498] Users can complete their orders within the app.

[1499] Specific examples

[1500] User Information:

[1501] Family preferences: Likes fish dishes, doesn't like spicy food

[1502] Allergy Information: Shrimp allergy

[1503] Health Condition: Diabetes

[1504] What's in the fridge: Chicken breast, broccoli, eggs

[1505] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[1506] Emotional state:

[1507] Feeling stressed

[1508] Prompt Sentence Examples

[1509] User Information:

[1510] Family preferences: Likes fish dishes, doesn't like spicy food

[1511] Allergy Information: Shrimp allergy

[1512] Health Condition: Diabetes

[1513] What's in the fridge: Chicken breast, broccoli, eggs

[1514] Repertoire of dishes you can make: Sautéed chicken breast, sautéed garlic broccoli, omelet

[1515] Emotional state:

[1516] Feeling stressed

[1517] Use this information to suggest a menu that includes ingredients that are effective in reducing stress.

[1518] This system makes it possible to provide optimal menu suggestions and delivery services that reflect the user's emotional state.

[1519] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1520] Step 1:

[1521] Users use a smartphone application to register their family's preferences, allergies, health conditions, the contents of their refrigerator, and the recipes they can make. The information entered is stored in the Firebase database. Specifically, when a user enters information into the application's input form and presses the submit button, the data is sent to and stored in Firebase.

[1522] Step 2:

[1523] The user takes a picture of their face using their smartphone camera, and the emotion is recognized using the emotion recognition API (Microsoft Azure Emotion API). The input is the user's face image, and the output is emotion data. Specifically, when the user activates the camera function in the app and takes a picture of their face, the image is sent to the emotion recognition API, and emotion data is returned.

[1524] Step 3:

[1525] The server uses a generative AI model (OpenAI GPT-4) to generate an optimal menu based on the emotion data received from the emotion recognition API and the user information stored in the Firebase database. The input is emotion data and user information, and the output is the optimal menu. Specifically, the server converts the emotion data and user information into prompt sentences and sends them to the generative AI model to generate the menu.

[1526] Step 4:

[1527] Based on the generated menu, the server uses a food delivery API (Uber Eats API) to search for corresponding dishes and provide delivery options. The input is the generated menu and the output is the delivery options. Specifically, the server sends the generated menu to the food delivery API and obtains the corresponding dishes and delivery options.

[1528] Step 5:

[1529] The user checks the delivery options and completes the order within the app. The input is the delivery option, and the output is a notification that the order has been completed. Specifically, when the user selects a delivery option within the app and presses the order button, the order is sent to the food delivery service and a notification that the order has been completed is displayed to the user.

[1530] Example 2

[1531] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1532] While conventional menu suggestion systems can take into account family preferences, allergies, health conditions, and the contents of the refrigerator, they have difficulty incorporating the latest sales information from nearby stores in real time. Furthermore, they lack the functionality to adjust ingredient purchasing suggestions based on the user's emotions, which prevents them from increasing user satisfaction. Furthermore, they lack the functionality to automatically extract product names and prices from advertising images and store them in a database, which requires time-consuming manual input. A system that can solve these issues is needed.

[1533] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1534] In this invention, the server includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of meals the user can make, means for capturing advertising information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where to buy it, means for detecting the user's emotions and adjusting ingredient purchasing suggestions based on the emotions, means for extracting product names and price information from advertising images using image recognition software, means for saving the extracted product names and price information in a database, means for detecting the user's emotions using emotion recognition software, and means for generating a menu based on the detected emotion information and suggesting it to the user. This allows the user to obtain the latest sales information from nearby stores in real time and receive ingredient purchasing suggestions based on their emotions.

[1535] "Family preferences" refer to the types of ingredients and dishes that each family member particularly likes.

[1536] "Allergy information" refers to information about food allergies that each member of the family has.

[1537] "Health Status" refers to information about each family member's current health status and specific dietary restrictions.

[1538] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[1539] "Repertoire" refers to the list of types of dishes or menus that a user can cook.

[1540] "Store advertising information" refers to product and price information contained in flyers and advertisements issued by nearby stores.

[1541] "Image recognition" refers to technology that automatically extracts specific information from image data.

[1542] A "menu" refers to a meal plan or menu list for a specific period of time (e.g., a week).

[1543] "Emotion detection" refers to technology that identifies a user's emotional state from their facial expressions and behavior.

[1544] "Ingredient purchase suggestions" refers to suggesting ingredients and menu items to purchase based on the user's preferences and emotional state.

[1545] "Image recognition software" refers to a program that analyzes image data and extracts specific information.

[1546] "Product name" refers to the name of the product as it appears in advertisements and flyers.

[1547] "Price information" refers to the product prices listed in advertisements and flyers.

[1548] A "database" refers to a system that stores information in an organized manner and allows it to be searched and updated as needed.

[1549] "Emotion recognition software" refers to programs designed to detect a user's emotions.

[1550] "Generate menu" refers to creating an appropriate menu based on the user's preferences and emotional state.

[1551] This invention is a system that allows users to obtain the latest sales information from nearby stores in real time and receive emotion-based food purchasing suggestions. Specific embodiments of this system are described below.

[1552] 1. System Configuration

[1553] This system consists of the user's smartphone (terminal), a server, image recognition software, emotion recognition software, and a database.

[1554] 2. Importing flyer information using image recognition

[1555] A user uses the camera of a smartphone to take a picture of a flyer from a nearby store. For example, if a user finds a sale flyer from a nearby store, the user takes a picture of the flyer with the smartphone.

[1556] The device then sends the flyer image to a server via the Internet in an appropriate format (e.g., JPEG, PNG).

[1557] The server analyzes the received images using image recognition software (e.g., Google Cloud Vision API), specifically using the API to extract text within the image and identify product names and pricing information.

[1558] The server stores the extracted product name and price information in a database. For example, information such as "apples - 100 yen" and "bananas - 150 yen" is stored.

[1559] 3. Food purchasing suggestions based on an emotional engine

[1560] The device detects the user's emotions using emotion recognition software (e.g., Microsoft Azure Emotion API). For example, the camera automatically recognizes the user's face and analyzes their emotions while they are using the smartphone.

[1561] The device sends the detected emotional information (e.g., joy, sadness) to the server. This information is sent as data indicating the user's current emotional state.

[1562] The server generates a menu that includes ingredients that the user likes based on the received emotion information. For example, if the user is feeling happy, the server generates a menu that includes cake and celebratory food.

[1563] The server sends the generated menu to the device and displays it to the user, who can then view the suggested menu on their smartphone screen.

[1564] 4. Specific Examples

[1565] Importing flyer information using image recognition

[1566] The user takes a photo of a flyer at a nearby store using the smartphone camera.

[1567] The terminal transmits the captured image to the server.

[1568] The server uses the Google Cloud Vision API to analyze the image and extract product name and price information.

[1569] The server stores the extracted information in a database.

[1570] Food purchasing suggestions based on an emotion engine

[1571] A user uses a smartphone while preparing for a birthday party.

[1572] The device uses the Microsoft Azure Emotion API to detect the user's emotions and recognizes that they are feeling joy.

[1573] The terminal transmits the emotion information to the server.

[1574] The server suggests recipes that include the user's favorite ingredients and generates a menu that includes cake and a celebratory meal.

[1575] The server sends the suggested menu to the terminal and displays it to the user.

[1576] 5. Examples of prompts

[1577] "I took a picture of a flyer from a nearby store. Use image recognition to extract product name and price information."

[1578] “You recognize that users feel joy when they are preparing for a birthday party. Use your emotion engine to suggest menu ideas that include cake and celebratory food.”

[1579] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1580] Step 1:

[1581] A user uses the smartphone camera to take a photo of a flyer for a nearby store. The input is the flyer image taken with the smartphone camera. The output is the flyer image saved on the smartphone. Specifically, the user launches the camera app, frames the flyer, and presses the shutter button.

[1582] Step 2:

[1583] The device sends the flyer image it has taken to a server via the Internet. The input is the flyer image stored on the smartphone. The output is the flyer image sent to the server. Specifically, the device converts the image to JPEG format and uploads it to the server using an HTTP request.

[1584] Step 3:

[1585] The server uses image recognition software (e.g., Google Cloud Vision API) to analyze the received image. The input is the flyer image sent to the server. The output is the extracted product name and price information. Specifically, the server calls the API, extracts the text in the image, and identifies the product name and price information.

[1586] Step 4:

[1587] The server saves the extracted product name and price information in a database. The input is the extracted product name and price information. The output is the product name and price information saved in the database. Specifically, the server establishes a database connection and inserts the information using an SQL query.

[1588] Step 5:

[1589] The device detects the user's emotions using emotion recognition software (e.g., Microsoft Azure Emotion API). The input is the user's facial image or video feed. The output is the detected emotion information. Specifically, the device uses a camera to capture the user's face and calls the API to analyze the emotion.

[1590] Step 6:

[1591] The device sends the detected emotion information to the server. The input is the detected emotion information. The output is the emotion information sent to the server. Specifically, the device converts the emotion information into JSON format and sends it to the server using an HTTP request.

[1592] Step 7:

[1593] The server generates a menu containing the user's favorite ingredients based on the received emotional information. The input is the emotional information sent to the server and the product information stored in the database. The output is the generated menu. Specifically, the server analyzes the emotional information and selects an appropriate menu based on the user's preferences.

[1594] Step 8:

[1595] The server sends the generated menu to the device and displays it to the user. The input is the generated menu. The output is the menu displayed on the device. Specifically, the server converts the menu information into JSON format and sends it to the device using an HTTP request. The user can view the suggested menu on their smartphone screen.

[1596] (Application example 2)

[1597] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1598] Conventional food purchasing support systems lack suggestions based on the user's emotions and individual preferences, making it difficult to increase user satisfaction. Furthermore, there are insufficient methods for efficiently importing supermarket flyer information and extracting product name and price information, making it difficult to quickly obtain the latest sales information. Furthermore, since there are no systems that make food purchasing suggestions based on the user's emotions, there is also the issue of being unable to make suggestions suitable for special occasions or events.

[1599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1600] In this invention, the server includes a means for registering family preferences, allergy information, health status, refrigerator contents, and recipes the user can make; a means for importing flyer information from nearby stores using image recognition; a means for suggesting weekly menus and what to buy and where; a means for analyzing the user's emotions and making ingredient purchasing suggestions based on those emotions; and a means for extracting product names and price information from flyer images captured with a smartphone camera. This allows for ingredient purchasing suggestions based on the user's emotions and individual preferences, making suggestions suitable for special occasions and events. It also allows for the rapid acquisition of the latest sales information, enabling efficient ingredient purchasing.

[1601] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1602] "Allergy information" refers to the type and severity of food allergies each family member has.

[1603] "Health status" refers to information about the health of each family member, including restrictions on the intake of certain foods and nutrients.

[1604] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[1605] "Repertoire you can make" refers to the types of dishes and recipes that a user can cook.

[1606] "Flyer information from nearby dealers" refers to the content of advertisements and flyers containing the latest sales information from dealers frequently used by the user.

[1607] "Image recognition" refers to the technology of extracting specific information from images taken with a camera.

[1608] A "weekly menu" refers to a meal plan for one week.

[1609] "Means of suggestion" refers to methods and techniques for suggesting the best ingredients and places to purchase them to users.

[1610] "Analyzing user emotions" refers to technology that determines a user's emotional state from their facial expressions and behavior.

[1611] "Emotion-based food purchasing suggestions" refers to suggesting appropriate ingredients and dishes based on the user's emotional state.

[1612] "Flyer image taken with a smartphone camera" refers to an image of a retailer's flyer taken with a smartphone camera.

[1613] "Means for extracting product name and price information" refers to methods and technologies for extracting product name and price information from flyer images using image recognition technology.

[1614] The system for implementing this invention is configured as follows: First, the user uses a smartphone to register information about their family's preferences, allergies, health conditions, what's in their refrigerator, and the recipes they can make, allowing the system to obtain individual information about the user.

[1615] Next, the user uses their smartphone camera to take a photo of a nearby store flyer, and the system uses image recognition technology to extract product names and prices from the flyer image, using OpenCV and pytesseract.

[1616] Additionally, the system takes a photo of the user's face and performs emotion analysis using Microsoft Azure's Face API, which allows it to determine the user's emotional state and make recommendations for food purchases based on their emotions.

[1617] The system uses this information to create a weekly menu and suggests what to buy and where to buy it, including ingredients and dishes suitable for special occasions and events, taking into account the user's emotional state and individual preferences.

[1618] For example, consider a case where a user photographs a supermarket flyer for a birthday and the system suggests a cake and a celebratory meal. In this case, sentiment analysis determines that the user is happy, so the system suggests special ingredients and a menu.

[1619] An example prompt might look like this:

[1620] "The user takes a photo of a supermarket flyer with their smartphone camera, and the system extracts product names and prices. It then takes a photo of the user's face and performs sentiment analysis. If the user is happy, it will suggest special ingredients and menu items."

[1621] In this way, it is possible to make food purchase suggestions based on the user's emotions and individual preferences, making suggestions suitable for special occasions and events.It also makes it possible to quickly obtain the latest sales information and purchase ingredients efficiently.

[1622] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1623] Step 1:

[1624] Users use their smartphones to register their family's preferences, allergy information, health conditions, what's in the refrigerator, and the recipes they can make.

[1625] Input: Family preferences, allergy information, health status, contents of the refrigerator, and recipes you can make

[1626] Output: Registered user information

[1627] Specific operation: The user starts the application on their smartphone and enters the information into the input form. The entered information is sent to the server and stored in the database.

[1628] Step 2:

[1629] The user uses the smartphone camera to take a picture of a flyer from a nearby store.

[1630] Input: Retailer flyer image

[1631] Output: Photographed flyer image

[1632] Specific operation: The user uses the camera function of the smartphone to take a picture of the store flyer. The taken image is saved in the application.

[1633] Step 3:

[1634] The server uses image recognition technology to extract product name and price information from the flyer image.

[1635] Input: Photographed flyer image

[1636] Output: Extracted product name and price information

[1637] Specific operation: The server uses OpenCV and pytesseract to analyze the flyer image and extract product name and price information as text data. The extracted data is stored in a database.

[1638] Step 4:

[1639] The user takes a picture of their face using the smartphone camera.

[1640] Input: User's face image

[1641] Output: Photographed face image

[1642] Specific operation: The user takes a picture of their face using the camera function of their smartphone. The captured image is saved in the application.

[1643] Step 5:

[1644] The server performs sentiment analysis to determine the emotional state of the user.

[1645] Input: A photographed face image

[1646] Output: User's emotional state

[1647] How it works: The server uses Microsoft Azure's Face API to analyze facial images and obtain the user's emotional state (happiness, sadness, surprise, etc.). The obtained emotional data is stored in a database.

[1648] Step 6:

[1649] Based on the user's emotional state and registered information, the server suggests a weekly menu and what to buy and where to buy it.

[1650] Input: User's emotional state, registered information, extracted product name and price information

[1651] Output: Suggested menu and where to buy

[1652] Specific operation: The server uses a generative AI model to suggest optimal menus and purchasing locations based on the user's emotional state, family preferences, allergy information, health status, refrigerator contents, cooking repertoire, and extracted product names and prices. The suggested information is sent to the user's smartphone.

[1653] Step 7:

[1654] The user checks the suggested menu and purchasing locations and purchases the necessary ingredients.

[1655] Input: Suggested menu and where to buy

[1656] Output: Purchased ingredients

[1657] Specific operation: The user checks the suggested menu and purchasing locations through the smartphone application, and then actually goes to the store to purchase the necessary ingredients.

[1658] Example 3

[1659] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1660] In today's busy lifestyles, it is extremely time-consuming to efficiently plan menus and purchase the necessary ingredients while taking into account the preferences and health status of family members. Managing food expenses and calculating calories is also not easy. Furthermore, there is a demand for flexible menu suggestions that reflect the user's emotions, but this is difficult to achieve with conventional systems. A system that can solve these problems is needed.

[1661] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1662] In this invention, the server includes means for registering family preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of meals the user can make, means for capturing flyer information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where, means for inputting the user's emotions and adjusting the menu based on those emotions, and means for modifying the menu by reflecting changes made by the user to the suggested menu and updating calorie calculations and food cost management. This enables the user to efficiently create menus and purchase the necessary ingredients, and also receives flexible menu suggestions based on emotions.

[1663] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1664] "Allergy information" refers to information about food allergies held by each member of the family.

[1665] "Health Status" is information about the health of each family member, including restrictions on certain foods and recommended nutrients.

[1666] "What's in the fridge" refers to a list of ingredients and foods currently stored in the fridge.

[1667] "Repertoire you can make" refers to a list of types of dishes and recipes that the user can cook.

[1668] "Store flyer information" refers to advertisements that include sales information and product information offered by nearby stores.

[1669] "Image recognition" refers to the technology of extracting specific information from image data.

[1670] A "weekly menu" refers to a meal plan for one week.

[1671] "Means of suggestion" refers to the method by which the system presents the user with the optimal menu and purchasing locations.

[1672] "User emotion" refers to the emotional state that a user is feeling.

[1673] "Means for adjusting menu based on emotions" refers to a method for changing the menu based on the user's emotional information.

[1674] "Calorie counting" refers to a method of calculating the calories contained in food.

[1675] "Food cost management" refers to methods for managing the costs of purchasing ingredients and food.

[1676] "Reading a receipt" refers to a method of extracting the information written on a receipt as digital data.

[1677] "Means for calculating purchase amount" refers to a method for calculating the amount of a purchase based on information extracted from a receipt.

[1678] "Means for modifying a menu" refers to a method for updating an existing menu to reflect user changes.

[1679] This invention is a system that allows users to efficiently create menus and purchase the necessary ingredients. This system includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of dishes that the user can make, means for capturing flyer information from nearby stores using image recognition, means for suggesting a weekly menu and what to buy and where, means for inputting the user's emotions and adjusting the menu based on those emotions, and means for modifying the menu to reflect changes made by the user to the suggested menu, and updating calorie calculations and food cost management.

[1680] Hardware and software used

[1681] Hardware: Servers, smartphones, PCs

[1682] Software: Image recognition software (e.g., image recognition API), emotion engine (e.g., emotion analysis API), database management system (e.g., database software)

[1683] System program processing

[1684] Register user information and supermarket flyer information

[1685] Users log in to the system using their smartphones or PCs and enter their name, age, allergy information, food preferences, etc. In addition, users upload supermarket flyer information to the system in PDF or image format. The server stores this information in a database.

[1686] Automatic menu generation

[1687] The server uses an AI model to automatically generate a weekly menu based on the user's registration information and uploaded flyer information, and then suggests to the user a list of ingredients and where to purchase them based on the generated menu.

[1688] Receipt scanning and food cost management

[1689] After shopping, users take a photo of their receipt with their smartphone camera and upload it to the system. The device then sends the image of the receipt to the server, which uses image recognition software to extract product names and prices from the receipt image, calculates the user's food expenses, and displays them on the management screen.

[1690] Menu modification and management

[1691] The user inputs changes to the menu suggested by the system, such as "I'm going to eat out today." The server modifies the menu based on the user's changes and updates calorie calculations and food cost management.

[1692] Emotion-based menu suggestions

[1693] The user inputs their emotions into the system, such as "I feel sad today." The server uses an emotion engine to analyze the user's emotions and, based on the analysis results, suggests a menu that matches the user's emotions. For example, if the user is feeling sad, it suggests a hot soup; if the user is feeling angry, it suggests a relaxing drink; and if the user is feeling surprised, it suggests an exotic dish.

[1694] Specific examples

[1695] Example 1: Automatic menu generation and ingredient purchasing suggestions

[1696] Users log in to the system and register supermarket flyer information. The server automatically generates a weekly menu based on the registered information and suggests the necessary ingredients and where to purchase them.

[1697] Example prompt sentence:

[1698] "Please suggest a menu for this week. The supermarket flyer information is as follows..."

[1699] Example 2: Receipt scanning and food cost management

[1700] After a user has finished shopping at a supermarket, they take a photo of the receipt with their smartphone camera. The server extracts the product name and price information from the image and calculates the total price.

[1701] Example prompt sentence:

[1702] "Please scan the information on this receipt to calculate the purchase amount..."

[1703] Example 3: Emotion-based menu suggestions

[1704] Users input emotions into the system: for example, if a user types "I'm sad today," the server uses the emotion engine to suggest comforting meals such as hot soup or porridge.

[1705] Example prompt sentence:

[1706] "I'm feeling sad today. Please suggest a comfort meal."

[1707] This system allows users to significantly reduce the effort required for creating menus and purchasing ingredients, and further enriches their daily lives by receiving meal suggestions that correspond to their emotions. The flow of the specification process in Example 3 will be described with reference to FIG.

[1708] Step 1:

[1709] Users log in to the system using a smartphone or PC and enter their name, age, allergy information, food preferences, etc. This information is then stored in a database.

[1710] Input: User's personal information (name, age, allergy information, food preferences)

[1711] Output: User information stored in the database

[1712] Step 2:

[1713] Users upload supermarket flyer information to the system in PDF or image format, and the terminal sends the uploaded flyer information to the server.

[1714] Input: Supermarket flyer information (PDF or image format)

[1715] Output: Flyer information sent to the server

[1716] Step 3:

[1717] The server uses image recognition software to extract product names and prices from the flyers, and stores the extracted information in a database.

[1718] Input: Flyer information (PDF or image format)

[1719] Output: Product name and price information stored in the database

[1720] Step 4:

[1721] The server uses an AI model to automatically generate a weekly menu based on the user's registration information and the extracted flyer information, and the generated menu is then suggested to the user.

[1722] Input: User registration information, extracted flyer information

[1723] Output: Automatically generated weekly menu

[1724] Step 5:

[1725] After shopping, users take a photo of their receipt with their smartphone camera and upload it to the system. The device then sends the captured receipt image to the server.

[1726] Input: Receipt image

[1727] Output: Receipt image sent to the server

[1728] Step 6:

[1729] The server uses image recognition software to extract product names and prices from receipt images, which are then stored in a database and used to calculate the user's food costs.

[1730] Input: Receipt image

[1731] Output: Product name and price information stored in the database, calculated food costs

[1732] Step 7:

[1733] The user inputs changes to the menu suggested by the system, such as "I'm going to eat out today." The server modifies the menu based on the user's changes and updates calorie calculations and food cost management.

[1734] Input: User change information

[1735] Output: revised menu, updated calorie count and food budget

[1736] Step 8:

[1737] The user inputs their emotions into the system, such as "I feel sad today." The server uses an emotion engine to analyze the user's emotions and, based on the analysis results, suggests a menu that matches the user's emotions.

[1738] Input: User's emotional information

[1739] Output: Emotion-based meal suggestions

[1740] In this way, the system provides flexible menu suggestions and food purchasing support that are tailored to the user's lifestyle and emotions.

[1741] (Application example 3)

[1742] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1743] In today's busy lifestyles, it is extremely difficult to efficiently plan meals and purchase the necessary ingredients while taking into account the preferences and health status of family members. There is also a need for meal suggestions that reflect the user's emotions and efficient ingredient purchasing based on past purchase history. Furthermore, there is a need for comprehensive dietary management, including food cost management and calorie calculation.

[1744] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health status, refrigerator contents, and user-recommended recipes; means for capturing advertising information for nearby stores using image recognition; means for proposing a weekly menu and recommendations on what to buy and where; means for recognizing the user's emotions and adjusting the menu based on those emotions; means for generating a menu based on past purchase history; and means for automatically ordering ingredients based on the generated menu. This allows users to efficiently create menus and purchase the necessary ingredients. It also enables meal suggestions based on the user's emotions and efficient ingredient purchasing based on past purchase history. Furthermore, comprehensive dietary management, including food cost management and calorie calculation, is possible.

[1745] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1746] "Allergy information" refers to information about food allergies that each member of the family has.

[1747] "Health status" refers to information about the health of each family member, such as specific illnesses or health limitations.

[1748] "What's in the refrigerator" refers to the types and quantities of ingredients and food currently stored in the refrigerator.

[1749] "Repertoire you can make" refers to the types of dishes and recipes that a user can cook.

[1750] "Advertising information from nearby stores" refers to advertisements and sale information provided by stores near the user's residence.

[1751] "Image recognition" refers to the technology of extracting specific information from image data.

[1752] A "weekly menu" refers to a meal plan for one week.

[1753] "Means of suggesting what to buy and where" refers to a function that suggests the best store to purchase the ingredients you need.

[1754] "Means for recognizing user emotions" refers to technology that analyzes and recognizes the user's emotional state.

[1755] "Means for adjusting menus based on emotions" refers to a function that changes menus according to the user's recognized emotions.

[1756] "Means for generating menus based on past purchase history" refers to a function that creates menus by referring to the user's past purchase history.

[1757] "Means for automatically ordering ingredients based on the created menu" refers to a function for automatically ordering ingredients required for the created menu.

[1758] The system for implementing this invention uses the following hardware and software: a smartphone, smart glasses, and a camera. The software uses OpenCV, Tesseract OCR, EmotionRecognizer, RecipeGenerator, and FoodDeliveryAPI.

[1759] The server provides a way for users to register their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the recipes they can make. Users input this information via their smartphone or smart glasses, allowing the system to generate menus tailored to the user's individual needs.

[1760] The server then provides a means to retrieve advertisement information from nearby stores using image recognition. When a user takes a photo of a store advertisement using a smartphone or smart glasses camera, the advertisement information is analyzed using OpenCV and Tesseract OCR to extract the price and sale information of the desired ingredients.

[1761] Furthermore, the server provides a means to recognize the user's emotions and adjust the menu based on those emotions. When the user takes a picture of their face using the camera on their smartphone or smart glasses, the EmotionRecognizer analyzes the image and recognizes the user's emotions. Based on the recognized emotions, the RecipeGenerator generates an appropriate menu.

[1762] The server also provides a means to generate menus based on past purchase history. When a user takes a photo of a receipt from a past purchase with a smartphone or smart glasses camera, Tesseract OCR analyzes the image and extracts the purchase history. Based on this, the Recipe Generator generates an efficient menu.

[1763] Finally, the server provides a means to automatically order ingredients based on the generated menu. The ingredients required for the generated menu can be automatically ordered using the FoodDelivery API.

[1764] As a concrete example, when a user takes a photo of a receipt with their smartphone and uploads an emotional image, the system operates as follows: It extracts information such as "Tomatoes 100 yen" and "Milk 150 yen" from the receipt image and recognizes "sadness" from the emotional image. For users who feel sad, it suggests a menu including hot soup or porridge and automatically orders the necessary ingredients.

[1765] An example of a prompt sentence to input to the generative AI model is as follows:

[1766] "Generate meal suggestions for when the user is feeling sad. Past purchase history is as follows: Tomatoes 100 yen, Milk 150 yen. The suggested meal should include hot soup or porridge."

[1767] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1768] Step 1:

[1769] Users use their smartphones or smart glasses to input their family's preferences, allergy information, health conditions, what's in the refrigerator, and the recipes they can make. This information is sent to the server and stored in a database. The input data is saved in text format and used for subsequent processing.

[1770] Step 2:

[1771] Users use the camera on their smartphone or smart glasses to take a photo of an advertisement for a nearby store. The device then sends the image to the server. The server then analyzes the image using OpenCV and Tesseract OCR and extracts the advertisement information as text data. The extracted data includes product names, prices, and sale information.

[1772] Step 3:

[1773] The user takes a picture of their face using the camera on their smartphone or smart glasses. The device sends the captured facial image to the server. The server then analyzes the facial image using EmotionRecognizer and recognizes the user's emotions. The recognized emotions are saved in text format and used for subsequent processing.

[1774] Step 4:

[1775] The server uses RecipeGenerator to generate a weekly menu based on the user's emotions, family preferences, allergy information, health status, refrigerator contents, available cooking repertoire, and advertising information. The input data is passed to RecipeGenerator in text format, and a list of menus is generated as output.

[1776] Step 5:

[1777] Users take a photo of a receipt from a previous purchase using the camera on their smartphone or smart glasses. The device then sends the captured receipt image to the server. The server then analyzes the receipt image using Tesseract OCR and extracts the purchase history as text data. The extracted data includes product names, prices, purchase dates, and more.

[1778] Step 6:

[1779] The server uses RecipeGenerator to generate efficient menus based on past purchase history. Input data is passed to RecipeGenerator in text format, and a list of menus is generated as output.

[1780] Step 7:

[1781] The server automatically orders the necessary ingredients based on the generated menu using the FoodDelivery API. The input data is a list of menu items, and the output is order confirmation information, which is sent to the user's smartphone or smart glasses.

[1782] Step 8:

[1783] The user receives the order confirmation via their smartphone or smart glasses and can confirm the order details. If necessary, they can modify the order details. The modified order details are then sent back to the server, where the final order is confirmed.

[1784] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1785] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1786] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1787] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1788] [Third embodiment]

[1789] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1790] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1791] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1792] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1793] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1794] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1795] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1796] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1797] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1798] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1799] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1800] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1801] "Example 1"

[1802] In the system of the present invention, a user registers his or her family's preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes that the user can make. For example, the user can register "likes fish dishes" and "doesn't like spicy food" as family preferences, "shrimp allergy" as allergy information, "diabetes" as health status, "chicken breast, broccoli, eggs" as refrigerator contents, and "sauteed chicken breast, sauteed garlic broccoli, rolled omelet" as the repertoire of dishes that the user can make.

[1803] "Example 2"

[1804] The system of the present invention uses image recognition to capture flyer information from nearby supermarkets. For example, a user can take a photo of a supermarket flyer with a smartphone camera and extract product names and prices from the image. This allows users to easily obtain the latest sales information from their favorite supermarkets.

[1805] "Example 3"

[1806] The system of this invention will propose a weekly menu and what to buy where. For example, it will automatically generate a weekly menu based on registered information and supermarket flyers, and suggest the necessary ingredients and where to buy them. This will significantly reduce the effort required for users to create a menu and buy ingredients.

[1807] "Example 4"

[1808] The system of the present invention reads receipts from past purchases and calculates the amount of money required for each purchase. For example, after shopping at a supermarket, a user can take a photo of the receipt with a smartphone camera and extract product names and prices from the image. This allows users to easily manage their food expenses.

[1809] "Example 5"

[1810] The system of the present invention allows calorie calculation, food cost management, and menu planning to be modified as needed to suit actual circumstances. For example, if the user changes the proposed menu, such as by deciding to eat out, the menu is modified to reflect that information, and calorie calculation and food cost management are updated. This allows users to flexibly manage their meals in accordance with their daily lifestyle.

[1811] The processing flow of each embodiment will be described below.

[1812] "Example 1"

[1813] Step 1: The user accesses the system and enters their profile information, including family preferences, allergy information, and health conditions.

[1814] Step 2: Next, the user registers their cooking repertoire with the system, including the name of the dish and the ingredients required.

[1815] Step 3: Finally, the user registers the contents of their refrigerator with the system, which allows the system to suggest meals that take into account the ingredients already in their refrigerator.

[1816] "Example 2"

[1817] Step 1: The user takes a photo of a supermarket flyer with their smartphone camera.

[1818] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[1819] Step 3: The extracted information is stored in the system and used to suggest menus and where to buy ingredients.

[1820] "Example 3"

[1821] Step 1: The system automatically generates a weekly menu based on the registered information and supermarket flyer information.

[1822] Step 2: The generated menu and list of ingredients are presented to the user.

[1823] Step 3: The user reviews the proposed menu and makes any necessary modifications or approvals.

[1824] "Example 4"

[1825] Step 1: After shopping at the supermarket, the user takes a photo of the receipt with their smartphone camera.

[1826] Step 2: The captured image is sent to the system, where product name and price information is extracted using image recognition technology.

[1827] Step 3: The extracted information is stored in the system and used to manage food expenses.

[1828] "Example 5"

[1829] Step 1: The user enters any changes to the proposed menu into the system.

[1830] Step 2: The system will modify the menu to reflect the information you entered, and update calorie counts and food cost management.

[1831] Step 3: The updated information is notified to the user, who can then check the new menu and meal cost information.

[1832] Example 1

[1833] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1834] Conventional home cooking support systems have had difficulty proposing recipes that take into account family preferences, allergies, health conditions, the contents of the refrigerator, and the user's own cooking repertoire. Another problem was that they were unable to quickly generate appropriate recipes when users requested recipe suggestions. Furthermore, if the generated recipe did not meet the user's requirements, it required the user to re-enter the information, which was time-consuming.

[1835] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for registering family preferences, allergy information, health conditions, refrigerator contents, and the user's own cooking repertoire, means for saving the registered information in a database, means for the user to request dish suggestions, means for acquiring user information from the database, means for creating a prompt sentence to be input to the generative AI model based on the acquired information, means for sending the prompt sentence to the generative AI model and generating a recipe, and means for displaying the generated recipe to the user. This makes it possible to quickly generate and display recipes that meet the user's requirements.

[1836] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family likes, and taste tendencies.

[1837] "Allergy information" is information about food allergies that the user or their family members have.

[1838] "Health Status" refers to information about the health of the user and their family, including specific illnesses and health care needs.

[1839] "Refrigerator contents" refers to information about ingredients and foods currently stored in the user's refrigerator.

[1840] "Repertoire that can be cooked by the user" is information on the types of dishes and recipes that the user can cook by himself.

[1841] The "means for registration" refers to a method or device that allows a user to input and save information about family preferences, allergy information, health conditions, the contents of the refrigerator, and the repertoire of dishes that the user can make into the system.

[1842] The "means for saving to a database" refers to a method or device for storing and managing information entered by a user in a database.

[1843] A "means for requesting dish suggestions" is a method or device that allows a user to request dish suggestions from the system.

[1844] The "means for obtaining user information from a database" refers to a method or device that allows the system to search and retrieve user information stored in a database.

[1845] "Means for creating prompt sentences to be input to a generative AI model" refers to a method or device for creating sentences to give appropriate instructions to a generative AI model based on acquired user information.

[1846] "Means for sending a prompt sentence to a generative AI model and generating a recipe" refers to a method or device for sending a created prompt sentence to a generative AI model and generating a recipe that meets the user's requirements.

[1847] "Means for displaying the generated recipe to the user" refers to a method or device for displaying the recipe generated by the generative AI model on the user's terminal.

[1848] This invention is a system that allows users to register their family's preferences, allergy information, health status, refrigerator contents, and cooking repertoire, and then suggests appropriate recipes based on that information. This system is implemented using a server, terminals, and a generative AI model.

[1849] First, users use a dedicated web form or application to enter their family's preferences (e.g., "I like fish dishes" or "I don't like spicy food"), allergy information (e.g., "I'm allergic to shrimp"), health conditions (e.g., "I'm diabetic"), the contents of their refrigerator (e.g., "chicken breast, broccoli, eggs"), and the repertoire of dishes they can make (e.g., "sauteed chicken breast, sauteed garlic broccoli, rolled omelet").

[1850] The terminal then converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The server then analyzes the received data and stores it in a relational database such as MySQL or PostgreSQL. For example, it organizes the information by user ID and stores the data corresponding to each field.

[1851] When a user requests a recipe suggestion, the server retrieves user information from a database. For example, it uses SQL queries to retrieve the user's family preferences, allergies, health status, refrigerator contents, and cooking repertoire. Based on this information, the server creates a prompt to be input to the generative AI model.

[1852] Examples of specific prompts include the following:

[1853] "My family likes fish dishes but doesn't like spicy food. I'm allergic to shrimp, and one family member is diabetic. We have chicken breasts, broccoli, and eggs in the refrigerator. I can make sautéed chicken breasts, sautéed garlic broccoli, and omelet. Based on this information, please suggest some suitable recipes."

[1854] The server sends this prompt to a generative AI model (for example, OpenAI's GPT-4) to generate a recipe. The generative AI model analyzes the prompt and generates a recipe that meets the user's requirements. The generated recipe is sent from the server to the device and displayed to the user. For example, the recipe title, ingredients, and instructions are displayed on the application screen.

[1855] The system allows users to quickly find the right recipe based on their family's preferences, health status, and what's in their refrigerator, and by using a generative AI model, it can generate recipes that meet the user's requirements with high accuracy.

[1856] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1857] Step 1: Enter your user information

[1858] Users use a dedicated web form or application to enter their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. The information is then stored in text format on the device.

[1859] Step 2: Sending data

[1860] The device converts the input information into a data format such as JSON and sends it to the server using the HTTPS protocol. The input data includes family preferences, allergy information, health status, the contents of the refrigerator, and the repertoire of dishes the user can make. The output data is the JSON format data sent to the server.

[1861] Step 3: Save your data

[1862] The server analyzes the received data and stores it in a relational database such as MySQL or PostgreSQL. The input data is user information in JSON format, and the output data is the user information stored in the database. Specifically, it executes an INSERT statement on the database.

[1863] Step 4: Request a recipe suggestion

[1864] A user requests a recipe suggestion by clicking a "recipe suggestion" button in an application or website. The input data is the user's request, and the output data is a request signal to the server.

[1865] Step 5: Get the data

[1866] The server retrieves user information from the database. The input data is the user ID, and the output data is the user's family preferences, allergy information, health status, contents of the refrigerator, and the repertoire of dishes they can make. Specifically, it executes a SELECT statement on the database.

[1867] Step 6: Create a prompt statement

[1868] The server creates a prompt sentence to be input to the generative AI model based on the acquired information. The input data is the acquired user information, and the output data is the prompt sentence to be sent to the generative AI model. Specifically, the server converts the user information into text format and constructs the prompt sentence.

[1869] Step 7: Generate the recipe

[1870] The server sends a prompt to the generative AI model to generate a recipe. The input data is the prompt, and the output data is the generated recipe. Specifically, the server sends a prompt to the generative AI model (e.g., OpenAI's GPT-4) and receives the generated recipe.

[1871] Step 8: Submit your recipe

[1872] The server converts the generated recipe into a data format such as JSON and sends it to the terminal. The input data is the generated recipe, and the output data is the JSON-formatted recipe sent to the terminal.

[1873] Step 9: View the recipe

[1874] The device displays the received recipe to the user. The input data is a recipe in JSON format, and the output data is the recipe information to be displayed to the user. Specifically, the application screen displays the recipe title, ingredients, instructions, etc.

[1875] (Application example 1)

[1876] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1877] Conventional food delivery services have difficulty suggesting optimal dishes that take into account the user's family preferences, allergies, and health conditions. Furthermore, they do not take into account the contents of the refrigerator or the user's cooking repertoire, making it difficult for users to use ingredients efficiently and maintain a healthy diet.

[1878] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1879] In this invention, the server includes means for registering family preferences, allergy information, health conditions, the contents of the refrigerator, and the recipes the user can make, means for importing flyer information from nearby stores using image recognition, means for proposing a weekly menu and what to buy and where, means for generating prompts using a generative AI model to suggest optimal dishes, and means for ordering the ingredients needed for the suggested dishes. This enables the server to suggest optimal dishes tailored to the user's individual needs and to use ingredients efficiently.

[1880] "Family preferences" refers to information such as the ingredients and types of dishes that the user's family prefers, as well as taste trends.

[1881] "Allergy information" is information about allergies that the user or their family members have.

[1882] "Health Status" means information about the health of a user or their family members, including specific illnesses or health care restrictions.

[1883] "What's in the refrigerator" is information about the ingredients currently stored in the user's refrigerator.

[1884] "Repertoire you can make" refers to information about the types of dishes and recipes that users can make themselves.

[1885] "Flyer information from nearby stores" is advertisements and sale information for products offered by stores in the user's neighborhood.

[1886] "Image recognition" is a technology that analyzes digital images to extract specific information.

[1887] A "generative AI model" is a model that has been trained using artificial intelligence to perform a specific task.

[1888] A "prompt" is text that is input to a generative AI model to produce a specific output.

[1889] The "optimal dish" is a dish suggested based on the user's family's preferences, allergy information, health condition, what's in the refrigerator, and the recipes they can make themselves.

[1890] "Means for ordering ingredients" is a function for ordering ingredients needed for the proposed dish online.

[1891] The system for implementing this invention is constructed as a smartphone application for the user. The system provides the user with the ability to register information about their family's preferences, allergies, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. This allows the user to receive suggestions for dishes that best suit their needs.

[1892] The server stores the information entered by the user in a database and generates a prompt using a generative AI model. Based on the generated prompt, the AI ​​model suggests the most suitable dish. The user can then order the ingredients needed for the suggested dish online.

[1893] Specifically, the server performs the following process.

[1894] 1. User data registration:

[1895] Through a smartphone application, users can input their family's preferences, allergy information, health conditions, what's in their refrigerator, and the recipes they can make. This information is then sent to a server and stored in a database.

[1896] 2. Generate prompt sentence:

[1897] The server generates a prompt based on the stored user data, such as:

[1898] Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Shrimp allergy, Health condition: Diabetes, Refrigerator contents: Chicken breast, broccoli, eggs, Cooking skills: Sautéed chicken breast, Sautéed broccoli with garlic, Omelette

[1899] 3. Recipe suggestions using generative AI models:

[1900] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3) to suggest the best dish. The generative AI model generates a dish that matches the user's needs based on the prompt.

[1901] 4. Order ingredients:

[1902] The ingredients required for the proposed dishes can be ordered online by the user through a smartphone application, and the server provides the user with a list of the proposed dishes and the required ingredients, facilitating the ordering process.

[1903] This system allows users to receive recommendations for optimal meals that take into account their family's preferences and health conditions, allowing them to use ingredients efficiently.It also allows users to easily order the ingredients they need, reducing the hassle of shopping.

[1904] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1905] Step 1:

[1906] Through a smartphone application, users input their family's preferences, allergy information, health conditions, the contents of their refrigerator, and the repertoire of dishes they can make. The input data is sent from the application to a server. The server then stores the received data in a database. Examples of input data include "I like fish dishes," "I'm allergic to shrimp," "I'm diabetic," "chicken breast, broccoli, and eggs," and "sauteed chicken breast, sauteed broccoli with garlic, and rolled omelet."

[1907] Step 2:

[1908] The server generates prompts based on the saved user data. Prompts are text-format input data to be sent to the generative AI model. An example of a generated prompt is "Family preferences: Likes fish dishes, Doesn't like spicy food, Allergy information: Allergy to shrimp, Health condition: Diabetes, Contents of refrigerator: Chicken breast, Broccoli, Eggs, Cooked dishes: Sauteed chicken breast, Sauteed broccoli with garlic, Omelette."

[1909] Step 3:

[1910] The server sends the generated prompt to a generative AI model (e.g., OpenAI's GPT-3). The generative AI model generates the optimal dish based on the prompt to meet the user's needs. The output from the generative AI model is a list of suggested dishes. For example, it might suggest "teriyaki chicken breast" or "broccoli and egg salad."

[1911] Step 4:

[1912] The server receives the output from the generative AI model and sends a list of suggested dishes to the user's smartphone application, through which the user can review the suggested dishes.

[1913] Step 5:

[1914] The user selects from the proposed dishes and orders the necessary ingredients online. The application generates a list of ingredients required for the selected dish and provides an interface for the user to confirm the order. The server receives the order data and sends the order information to the corresponding store.

[1915] Step 6:

[1916] The store prepares ingredients based on the received order information and delivers them to the user, who can then use the ingredients to create the suggested dish.

[1917] In this way, users can have the best possible recipes suggested based on their family's preferences and health status, making efficient use of ingredients. Furthermore, they can easily order the ingredients they need, saving them the trouble of shopping.

[1918] Example 2

[1919] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1920] With conventional systems, users had to manually check the latest sales information of nearby stores, which was time-consuming and made it difficult to obtain accurate information.In addition, it was difficult to propose menus that took into account family preferences, allergies, health conditions, and the contents of the refrigerator, making it difficult to create efficient shopping plans.

[1921] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1922] In this invention, the server includes a means for transmitting captured images to the server, a means for the server to perform image recognition and extract product name and price information, and a means for storing the extracted information in a database. This allows users to automatically analyze store advertising information captured with a smartphone camera and obtain product name and price information. It also facilitates menu suggestions and efficient shopping planning that take into account family preferences, allergies, health conditions, and refrigerator contents.

[1923] "Family preferences" refers to the types of ingredients and dishes preferred by each family member.

[1924] "Allergy information" refers to information about food allergies that each member of the family has.

[1925] "Health status" refers to information about the health of each family member, such as chronic illnesses or nutritional needs.

[1926] "What's in the refrigerator" refers to the types and quantities of food and beverages currently stored in the refrigerator.

[1927] "Repertoire" refers to the types of dishes a user can cook.

[1928] "Store advertising information" refers to product and price information listed in flyers and advertisements issued by nearby stores.

[1929] "Image recognition" refers to the technology of extracting specific information from captured images.

[1930] "Server" refers to a computer system for processing and storing data.

[1931] "Database" refers to a system for storing extracted information in an organized manner.

[1932] "Menu suggestions" refers to proposing a weekly meal plan that takes into account the family's preferences, health status, and the contents of the refrigerator.

[1933] "Shopping planning" refers to planning which stores to purchase the ingredients and products you need.

[1934] "Receipt reading" refers to the technology of reading information about previously purchased products from receipts.

[1935] "Calorie counting" refers to calculating the calories of food.

[1936] "Food cost management" refers to managing the costs of purchasing ingredients and products.

[1937] "Appropriately modify" refers to changing plans or proposals to suit actual circumstances.

[1938] This invention is a system that allows users to...

Claims

1. A means for registering the preferences of family members, information about allergies of family members, health conditions of family members, contents of a refrigerator, and a repertoire of dishes that the user can make; means for recognizing an emotional state of a user by an emotion engine; A means for generating a prompt sentence for instructing the user to generate a menu of dishes to be suggested to the user based on the emotional state of the user, the preferences of the family, allergy information of the family, the health condition of the family, the contents of the refrigerator, and a repertoire of dishes that the user can make; A means for generating a menu of dishes to be suggested to a user using the generated prompt sentence and the generative AI model; A means for presenting the generated menu to a user; A system including:

2. The device further includes a means for reading receipts of past purchases of ingredients and calculating how much the proposed menu would cost to purchase. The system of claim 1 .

3. A method for using image recognition software to extract product names and price information from images of advertisements for nearby supermarkets taken with a smartphone camera; A means for storing the extracted product name and price information in a database; The method further includes calculating how much the suggested menu would cost based on the stored product name and price information. The system of claim 1 .

Images